robots.txt Generator
Generate a robots.txt with allow or disallow rules, crawl delay and a sitemap reference.
Generate a robots.txt with sensible crawl rules and your sitemap reference.
What this does
- ✅ Search engines can index this site.
- Blocked: /admin, /api
- Sitemap advertised to crawlers.
User-agent: *
Disallow: /admin
Disallow: /api
Sitemap: https://example.com/sitemap.xmlAbout the robots.txt generator
robots.txt controls crawling, not indexing. A page blocked here can still appear in search results if other sites link to it, because the crawler never reads the noindex tag it was blocked from fetching.
If you need a page kept out of the index, allow crawling and use a noindex meta tag instead. Use robots.txt to save crawl budget on things like admin paths and internal search results.
How to use it
- 1Choose a policy. Block everything only on staging.
- 2List paths to disallow, one per line.
- 3Add your sitemap URL so crawlers find it immediately.
- 4Copy the file and place it at the root of your domain.
Questions
Does Disallow stop a page appearing in Google?
No. It stops crawling, not indexing. A blocked page can still be listed based on external links. Use a noindex meta tag for reliable exclusion, and leave the page crawlable so the tag is seen.
Where does robots.txt go?
At the root of the domain, for example example.com/robots.txt. Anywhere else and it is ignored.
Should I set a crawl delay?
Rarely. Google ignores it entirely. It only helps if a specific aggressive crawler is straining your server.