How AI Crawlers Work
AI systems fetch the web for training, for live retrieval, and for answering a question you asked a moment ago. Those are three different crawlers with three different consequences.
Intermediate8 min readDaksh BathlaDEFINITIONSearch & AI Discovery
llms.txt
A proposed convention: a Markdown file at a site's root pointing AI systems at a clean summary of its most useful content. It is not a standard and grants no permissions.
EXAMPLE
A file listing your twenty most useful pages with one-line descriptions, so a model doesn't have to reconstruct what your site offers from navigation and marketing copy.
WHY IT MATTERS
Adoption is partial and voluntary, so treat it as cheap and low-risk rather than decisive. The real benefit is that writing it forces you to state which pages actually matter.