Free crawler and discovery tools

Validate, generate and inspect robots.txt, llms.txt, structured data and crawlability signals.