Robots.txt Explained: What It Blocks, and What It Cannot

Where it lives and how it is read

The file must be at the root of the host, such as https://example.com/robots.txt. It applies only to that exact host and protocol, so blog.example.com needs its own file. Crawlers fetch it before crawling and cache it, Google for up to about a day, so changes are not instant.

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

• User-agent says which crawler the group of rules is for. * means all of them.
• Disallow blocks any path that starts with the value. /cart blocks /cart, /cart/ and /cartoon too.
• Allow makes an exception inside a blocked path. When rules conflict, Google follows the most specific, meaning the longest, matching rule.
• * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF.
• Sitemap points crawlers to your sitemap, with a full URL.

Blocking crawling is not blocking indexing

Disallow stops a crawler from reading a page. It does not stop the page from appearing in search results. If other sites link to a blocked URL, Google can still index the address and show it, usually with no description, because it was never allowed to read the content.

To keep a page out of search, do the opposite of what most people try: allow it to be crawled, and add a noindex robots meta tag or header. Google has to fetch the page to see the noindex. Block it in robots.txt and it never will.

<meta name="robots" content="noindex">

Robots.txt is not security

The file is public, and following it is voluntary. Well-behaved search engines obey it; scrapers and attackers do not, and some read it precisely to find paths you would rather hide. Anything private needs a login, not a Disallow line.

Mistakes that hurt rankings

• Shipping a staging file with Disallow: / to production, which asks every search engine to stop crawling the whole site.
• Blocking CSS and JavaScript files. Google renders pages like a browser, and without those files it may see a broken page.
• Blocking pages you want removed, which keeps the noindex from ever being seen.
• Returning an error for robots.txt itself. If the file returns a server error, Google may hold off crawling the site until it can read it. A missing file, returning 404, is fine and means everything is allowed.

A safe default

Most sites need very little: allow everything, block a few genuinely useless areas such as internal search results or the checkout, and list the sitemap. When in doubt, block less. A site that is too open loses almost nothing; a site that blocks too much can disappear from search.