What it is
robots.txt is a text file at the root of your domain — always at /robots.txt, never anywhere else — that tells automated crawlers which paths they may request. It's a convention, not a security mechanism: well-behaved crawlers respect it, and anything malicious ignores it entirely.
User-agent: *Disallow: /admin/Disallow: /api/Allow: / Sitemap: https://navitallabs.com/sitemap.xml- User-agent — which crawler the following rules apply to; * means all of them
- Disallow — a path prefix the crawler should not request
- Allow — an exception carved out of a broader Disallow
- Sitemap — an absolute URL to your sitemap; the most useful line in the file
Blocking a crawl is not preventing indexing
This distinction accounts for most robots.txt confusion. Disallow says "don't fetch this page." It doesn't say "don't list it." A page that other sites link to can still appear in results — with no description, because the crawler was never allowed to read it.
To keep a page out of results, you must let the crawler fetch it and find a noindex instruction. Blocking the fetch means the instruction is never seen — the two directives work against each other.
| Goal | Use |
|---|---|
| Save crawl budget on infinite filter URLs | Disallow in robots.txt |
| Keep a page out of search results | Allow the crawl, add noindex |
| Keep content private | Authentication — neither of the above |
| Point crawlers at your pages | The Sitemap line |
The mistake that removes a site from search
User-agent: *Disallow: /Two lines that block the entire site. This is the standard content of a staging site's robots.txt, and it reaches production by being deployed along with everything else. Traffic then declines over days as pages drop out, which is slow enough that the cause is rarely the first suspect.
AI crawlers
Model providers and AI search products run their own crawlers, each with its own user-agent, and most respect robots.txt. So you can allow or block them separately from search engines.
It's a genuine trade-off with no universal answer. Blocking them protects content from being used as training or answer material. Allowing them is a precondition for being cited in AI answers at all. For most small businesses that want to be discoverable, allowing is the more consistent choice — blocking a crawler and then hoping to appear in its answers is wishing for both sides of the same decision.
# Allow search and AI crawlers everywhere except private paths.User-agent: *Disallow: /admin/ # Example of blocking one specific crawler, if you choose to.# User-agent: SomeCrawler# Disallow: /Common mistakes
- Shipping a staging Disallow: / to production
- Using Disallow to try to hide a page that's already indexed
- Combining Disallow with noindex, so the noindex is never read
- Listing sensitive paths, which advertises them
- Blocking CSS and JavaScript, which prevents the page being rendered and judged properly
Key takeaways
- robots.txt controls crawling, not indexing, and it isn't security
- To remove a page from results, allow the crawl and use noindex
- Disallow: / is two characters away from removing your site from search
- AI crawlers can be handled separately — decide deliberately, then be consistent
Try it yourself
Open yoursite.com/robots.txt right now. Confirm it isn't blocking everything, and that the Sitemap line is present and correct. If there's no robots.txt at all, that's fine — everything is allowed by default — but you're missing the sitemap pointer.