How AI Crawlers Work

Treating "AI crawlers" as one thing leads to blocking decisions that don't do what people expect. There are three distinct behaviours, and they have different consequences for you.

Providers publish separate user-agent names for these, precisely so the decisions can be made separately. Check the current names in each provider's documentation rather than copying a robots.txt from a blog post — the list changes.

• They usually don't render JavaScript. Search engines will, on a second pass. A retrieval fetch typically takes the raw HTML and moves on — so client-rendered content is often simply absent
• They extract passages, not pages. The unit that matters is a paragraph that survives being lifted out of context
• They are far more sensitive to contradiction. Two conflicting facts about your pricing means neither is safe to state, and a competitor's clearer page gets used
• They may not send a referrer, so the traffic is often invisible in analytics even when you influenced the answer
• A live fetch happens once, immediately, with no second chance — a slow or erroring page is just missing from that answer

llms.txt is a proposed convention: a Markdown file at your site root that points AI systems at a clean, structured summary of your most useful content, so a model doesn't have to reconstruct it from navigation and marketing pages.

# Navital Learning > Practical lessons on building a business: business, design,> development, AI, search, and marketing. Written by the people> who build the products at Navital Labs. ## Lessons- [What Is an MVP?](https://navitallabs.com/learning/lessons/what-is-an-mvp):  The smallest thing that completely finishes one job for one person.- [What Is SEO?](https://navitallabs.com/learning/lessons/what-is-seo):  Making a site easy to crawl, understand, and trust. ## Reference- [Business Dictionary](https://navitallabs.com/learning/dictionary):  23 terms with definitions, examples, and why each matters.

The reason to add one anyway is that writing it forces a useful exercise: stating, in one file, what your site actually offers and which twenty pages matter. Most sites find that harder to answer than they expected, and the answer improves the site.

1. 01Put the content in the HTMLIf it needs JavaScript to appear, assume a retrieval fetch won't see it.
2. 02Make each key claim complete in one paragraphA fact assembled across three paragraphs cannot be lifted without being reconstructed, and reconstruction is where errors enter.
3. 03State facts once, consistently, everywherePricing, founding date, product names. Fix or remove stale pages that contradict the current one.
4. 04Date the page, visibly and in structured dataRetrieval systems prefer content they can tell is current.
5. 05Decide your crawler policy deliberately, then be consistentBlocking retrieval and hoping to be cited is wanting both sides of one decision.