How to Create a Robots.txt File

A robots.txt file tells search engine crawlers which parts of a website they may crawl. It is not a security feature, but it is useful for crawl management, duplicate pages, private-looking utility folders, and sitemap discovery.

For a static site, the file should sit at the root of the domain, such as https://example.com/robots.txt. Crawlers look for that exact location before crawling the rest of the site.

Basic robots.txt example

What to block

Block folders and files that should not appear in search results, such as private drafts, build notes, scripts, internal checklists, and development artifacts. Do not rely on robots.txt for secrets because blocked URLs may still be visible if linked elsewhere.

Testing checklist

• Open the live robots.txt URL in your browser.
• Check every disallow path for spelling and leading slashes.
• Confirm your sitemap URL is correct.
• Use Google Search Console URL inspection for important pages.

Common mistake

The biggest mistake is blocking important content accidentally. If you add Disallow: /tools/, your tool pages may become harder for crawlers to discover. Keep rules narrow and test after every deployment.