LLMs.txt and GEO: Why the File Is Ignored by AI Search Today
LLMs.txt is a proposed plain-text file placed at a site's root that lists the pages you want language models to read first. As of mid-2026, it has no confirmed impact on how AI search engines rank, cite, or include your content. Crawlers fetch it. None of the major answer engines have publicly stated they weight it during answer generation. Google has said that standard search signals, not a dedicated LLM file, drive inclusion in AI Overviews [S1]. The honest position for a content team is this: llms.txt is a forward-looking bet with real (non-ranking) uses, but it is not a lever you can pull for AI search visibility this quarter.
That gap between the promise some vendors attach to the file and what the data actually shows is the central story of this post. Below is what llms.txt is, what the evidence says about its impact on AI search, where it does deliver value, and how to make a clear-eyed decision about whether it belongs in your stack.
What is llms.txt?
LLMs.txt is a Markdown-formatted file served at yourdomain.com/llms.txt that acts as a curated map of your most valuable content for language models. Think of it as a sitemap written for reading comprehension rather than crawl discovery. A typical file opens with an H1 site name, a short summary in a blockquote, then grouped links to documentation, FAQs, guides, and reference material, each with a one-line description.
The format was proposed in late 2024 by Jeremy Howard of Answer.AI as a way to help models find clean, high-signal pages without wading through navigation, ads, and boilerplate [S2]. Two variants exist. The base llms.txt lists links. An optional llms-full.txt inlines the actual page content into one document so a model can ingest everything in a single fetch. The intent is reasonable and narrowly defined: give language models a clean starting point. The question is whether that starting point translates into better AI search outcomes, and the evidence says it does not.
What the server log data actually shows
Several teams have analyzed server logs across large site networks to see whether publishing an llms.txt file correlates with any measurable change in AI search behavior. The pattern that emerges is consistent: the file gets fetched, but it produces no observable ranking or citation advantage tied to its presence [S3].
Requests for /llms.txt do appear in logs from AI crawler user agents, including GPTBot, ClaudeBot, and PerplexityBot. Fetch volume varies by site authority and category, but the key observation is that these fetches look identical to how a crawler handles any publicly accessible text file at a predictable path. The crawler discovers the file, fetches it, and moves on. There is no logged evidence of preferential treatment for pages listed in the file versus pages not listed in it.
Promptwatch conducted a detailed analysis of llms.txt adoption and crawler behavior across a network of monitored sites and found that while crawl requests to the file are real and growing, there is no measurable difference in citation rates or answer presence between sites with and without the file [S3]. The crawlers read it. The answer engines do not appear to act on it during generation.
This is an important distinction. A vendor or consultant who shows you growing crawler requests to your llms.txt file is showing you a real metric, but they are showing you an input (fetches), not an outcome (citations). The connection between the two has not been established.
Why no major engine has adopted it as a ranking signal
Three structural reasons explain why the major answer engines have not committed to llms.txt as part of their ranking and citation pipeline.
First, the trust problem. An llms.txt file is a document you write about yourself. From the engine's perspective, it is a claim, not a verified signal. If answer engines weighted it, every site would have an incentive to fill the file with self-serving descriptions and prioritize pages designed to sell rather than inform. This is the same reason search engines long ago stopped trusting meta keywords: the signal is too easy to game when the publisher controls it entirely. Engines prefer signals that are expensive to fake, like off-site mentions, backlink patterns, and cross-source corroboration.
Second, the infrastructure reality. ChatGPT search, Perplexity, and Google's AI surfaces already run sophisticated retrieval pipelines built on existing search indexes, live crawls, and proprietary ranking models. These systems were designed to handle the messiness of the open web. They do not need a publisher-authored shortcut, and integrating one would mean building support, testing for abuse, and maintaining a parallel input channel for a standard that a small minority of sites have adopted. Google has stated explicitly that AI Overviews draw from the same web index and signals that power traditional search results, not from supplementary files like llms.txt [S1].
Third, the adoption chicken-and-egg. For an engine to invest in supporting llms.txt, the file would need to be widespread enough to provide useful signal at scale. But site owners are unlikely to invest in maintaining the file until engines confirm they use it. This standoff keeps the standard in a holding pattern where adoption grows slowly among developer-facing sites but does not reach the critical mass that would justify engine-side support.
None of this means the standard is dead. Web standards have taken years to reach adoption before. Schema markup went through a long period of limited engine support before becoming genuinely useful. The difference is that schema had the backing of Google, Bing, Yahoo, and Yandex from the start. LLMs.txt does not have equivalent backing from any major AI answer engine.
Where llms.txt delivers real value today
The file does exactly what Jeremy Howard designed it to do: it helps language models consume a site's best content efficiently in controlled contexts. Three use cases provide clear, present-day returns.
Developer and API documentation. Coding assistants, IDE integrations, and doc-aware agents actively look for structured reference indexes. If your product has a developer audience, an llms.txt file that points to your SDK docs, API reference, and getting-started guide makes those tools work better when pointed at your site. This is where early adoption clusters because the consumers (coding agents) are known and the benefit is direct.
Internal and partner RAG systems. When you know the consumer respects the file, the behavior is deterministic rather than probabilistic. A customer support bot powered by retrieval-augmented generation that reads your llms.txt will index the right pages in the right order. You control both the file and the consumer, so the investment has a clear payoff.
Large content libraries. If you run hundreds of articles, a hand-curated list of your thirty best pages tells any cooperating system what to prioritize. This is particularly valuable for open-source projects, educational platforms, and reference sites where the volume of content makes automatic prioritization unreliable.
In all three cases, the value comes from the file being read by a system you influence, not from the file being weighted by an independent answer engine. That is a real, useful distinction. It means the file has a place in your toolbox. It just does not belong in your GEO strategy.
What actually drives AI search visibility
If llms.txt does not improve AI search rankings, what does? The answer is consistent across every study and analysis of generative engine behavior published to date. Visibility in AI answers comes from four things, and none of them are shortcuts.
Answer-first content. Pages that state a direct, complete answer in the opening sentences get cited far more often than pages that build up to the answer over several paragraphs. Answer engines extract and cite the first clean, self-contained statement they find. If your answer lives in paragraph four, the model may never reach it before filling its context with a competitor's cleaner page.
Clean, semantic structure. Headings that are real questions (not clever slogans), sections that can stand alone when extracted, and consistent formatting that gives the model clear boundaries to lift from. Schema markup adds explicit signals about your entities, content type, and authorship, which reduces the chance a model misreads what a page is about.
Cited, verifiable claims. Every statistic and factual assertion should carry a named, linkable source. Models reward pages that show their work because it reduces the risk of propagating a hallucination. A page full of unsourced assertions reads as low trust to a model deciding which source to cite.
Off-site corroboration. Being mentioned, reviewed, and quoted across reputable third-party sources raises the chance the model treats your claims as established fact. Models reconcile what you say about yourself against what others say about you. When those signals align, trust compounds. When they contradict, the model hedges or skips you entirely.
Dokeo checks those properties as a content quality gate: 19 content checks across SEO, AEO and GEO (3 engines), applied to every draft before it ships. Whether or not you publish an llms.txt file, the pages it points to need to pass these fundamentals, or no amount of metadata will make them quotable.
How to make the decision for your team
The decision tree is short. If you have a developer audience, significant documentation, or RAG systems that read from your site, publish llms.txt now and automate its maintenance. The setup cost is measured in minutes. The maintenance cost is near zero if you generate it from your CMS or build pipeline. The upside in agent and RAG contexts is real and immediate.
If you do not have those use cases, publish the file anyway as a hedge, but do not allocate meaningful time to it. Do not let a vendor convince you that the file is moving your AI search visibility. The evidence does not support that claim. Redirect that time toward the four fundamentals listed above, which have confirmed, measurable returns in AI answer citations.
One maintenance warning applies to both camps: a stale llms.txt is worse than none. If the file points to dead URLs or describes products you have retired, and a future engine begins honoring it, you will have hard-coded bad signals into your own site. Keep it current or remove it. There is no safe middle ground.
Focus on what works: structure, citations, and answer-first content.
Sources
[S1] Search Engine Land, "Google: llms.txt is not used for AI Overviews," https://searchengineland.com/google-llms-txt-ai-overviews
[S2] Answer.AI, "The /llms.txt proposal," https://llmstxt.org
[S3] Promptwatch, "What our crawler logs show about llms.txt," https://promptwatch.io/blog/llms-txt-logs
Get started with a free Dokeo scan to see how your content scores across SEO, AEO, and GEO. Read more on the Dokeo blog.