Skip to content
Climb Google Maps: Google Business Profile optimization.

GEO and AEOGuide

llms.txt: What It Is, What Goes in It, and Which AI Crawlers Actually Read It. We Checked 33 Big Sites

llms.txt is a short Markdown file at the root of a site that tells AI tools what the site is and where its key pages are. Google says you don't need it for AI Overviews, and no major AI crawler documents reading it. We checked 33 well-known sites: 16 have one, and Google, Bing and every big news publisher we tried do not.

Rows of servers with blue status lights in a data center

Key takeaways

  • llms.txt is a Markdown file served at /llms.txt that gives AI tools a short summary of a site and links to its most useful pages; Jeremy Howard proposed it on Sept. 3, 2024 at llmstxt.org.
  • Google's AI features documentation says you don't need new machine readable files or AI text files to appear in AI Overviews or AI Mode, and the crawler pages of OpenAI, Anthropic and Perplexity don't say their crawlers read llms.txt.
  • In our Nov. 13, 2026 check of 33 well-known sites, 16 served an llms.txt, 15 did not and 2 blocked our request; Google, Bing, Microsoft, Ahrefs, Moz and every large news publisher we tried had none.
  • The AI companies that publish llms.txt put it on their developer docs (OpenAI, Anthropic, Perplexity), where coding assistants use it, not on their main websites.
  • Google's Lighthouse now has an llms.txt check in its agentic browsing audits: a missing file is marked not applicable, and only a server error fails.

Short answer: llms.txt is a plain Markdown file at yoursite.com/llms.txt that tells AI tools, in a few lines, what your site is and which pages matter most. It is a proposal, not a standard any search engine has adopted. Google says you don't need it to appear in AI Overviews or AI Mode, and none of the big AI companies says its crawler reads it. It is still cheap to make, and it does help one group: AI coding and research agents that are pointed at your site.

To see who actually uses it, we fetched /llms.txt from 33 well-known sites on Nov. 13, 2026. The results, the format, and a script that checks your own file are below.

What is llms.txt?

Jeremy Howard proposed llms.txt on Sept. 3, 2024, on llmstxt.org. The idea: a web page is built for people, full of menus, scripts and ads, and a language model reading it has limited room. A short Markdown file at the root of the site gives the model a clean summary and a list of links to the pages worth reading.

It is not robots.txt. robots.txt tells crawlers what they may fetch. llms.txt tells a model what is worth reading. It allows and blocks nothing.

What goes in an llms.txt file

The proposal defines the format in this order:

  1. An H1 with the site or project name. This is the only required part.
  2. A blockquote (a line starting with >) with a one or two sentence summary.
  3. Optional paragraphs or lists with anything else a model should know, without headings.
  4. H2 sections of links, each item a Markdown link with an optional note after a colon: - [Page name](https://example.com/page): what it covers.
  5. An optional ## Optional section for secondary links a tool can skip when it is short on room.

The proposal also suggests serving a Markdown copy of each page at the same address with .md added, for example /pricing.md. Many sites skip that part.

Here is the start of ours, live at thesearchpost.com/llms.txt:

# The Search Post
 
> Daily news, tips and guides on SEO, AI search (GEO and AEO), Google Business Profile and paid ads, explained for people who run websites.
 
- [About us and how we report](https://thesearchpost.com/about)
- [Latest stories](https://thesearchpost.com/latest)
- [Every story in full (Markdown)](https://thesearchpost.com/llms-full.txt)
 
## Sections
 
- [News](https://thesearchpost.com/news): What changed today in Google, AI search and paid ads, and what it means for your site.
- [Tips and guides](https://thesearchpost.com/guides): Step-by-step tips and how-tos for SEO, AI search, Google Business Profile and paid ads.

The full file is 12,913 bytes with 55 links. We also publish /llms-full.txt, every story in one Markdown file (349,574 bytes today). That name is a common convention, not part of the proposal. We send both with an X-Robots-Tag: noindex header, so Google never shows the full-text copy in search results instead of the real pages.

Does llms.txt work? Which AI crawlers read it

This is the part most guides skip. Here is what each company's own documentation says, as of Nov. 13, 2026:

  • Google: its AI features page says "You don't need to create new machine readable files, AI text files, or markup to appear in these features," meaning AI Overviews and AI Mode. They use the normal Search index.
  • OpenAI: its crawler page describes OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot and how robots.txt controls them. It says nothing about any of them reading llms.txt, though the page itself links to OpenAI's own docs llms.txt.
  • Anthropic: its crawler help article covers ClaudeBot, Claude-User and Claude-SearchBot and does not mention llms.txt.
  • Perplexity: its crawler page covers PerplexityBot and Perplexity-User. Like OpenAI's, it links to its own llms.txt as a docs index, but says nothing about its crawlers using other sites' files.

So no major AI search product documents llms.txt as something that affects whether or how you are cited. Claims that it raises AI visibility are not backed by any of these companies. Treat them with care.

The one Google product that does look at it is Lighthouse. Its agentic browsing audits now include an llms.txt check. A missing file is marked not applicable, because Google calls the file "optional at the moment." Only a server error when fetching it fails the audit. Google's note says that without the file, agents "may spend more time crawling the site." That is about AI agents browsing a site, not about Search rankings.

Who has an llms.txt? Our check of 33 sites

On Nov. 13, 2026, at about 1:40 p.m. Eastern (18:40 UTC), we requested https://<site>/llms.txt from 33 well-known sites, following redirects. We counted a file only when it returned HTTP 200 with text, not an HTML page.

16 had one:

SiteSize (bytes)Links
docs.anthropic.com (redirects to platform.claude.com)102,075917
stripe.com69,815307
docs.perplexity.ai43,171191
wix.com38,721236
hubspot.com35,564225
cloudflare.com17,17390
developers.cloudflare.com16,609111
semrush.com14,95242
nextjs.org14,59263
shopify.com12,64274
searchenginejournal.com11,18581
yoast.com7,68265
developers.openai.com6,34541
wordpress.org5,76536
vercel.com5,54028
squarespace.com (redirects to /assets/llms.txt)2,4097

15 had none (HTTP 404): google.com, developers.google.com, bing.com, microsoft.com, anthropic.com, perplexity.ai, ahrefs.com, moz.com, nytimes.com, theguardian.com, bbc.com, en.wikipedia.org, forbes.com, apple.com and amazon.com.

2 blocked our request (HTTP 403, a bot check): openai.com and searchengineland.com. We can't say whether they have one.

What stands out:

  • The search engines don't publish one. Neither Google nor Bing has an llms.txt on its main site or, for Google, its developer site.
  • AI companies put it on their docs, not their homepages. OpenAI, Anthropic and Perplexity all serve one on their developer documentation and none on their main sites. That fits its real use today: coding assistants reading API docs.
  • Website builders and dev platforms lead. Wix, Squarespace, Shopify, WordPress.org, HubSpot, Vercel, Next.js and Stripe all have one.
  • News publishers mostly don't. Of the publishers we tried, only Search Engine Journal had one.
  • Most follow the format. All 16 started with an H1. Anthropic's skipped the summary blockquote, and 7 used an ## Optional section.

How to create an llms.txt file

  1. Write the H1 and summary. Your business or site name, then one sentence on what you do and for whom, and where if you are local.
  2. List the pages a model should read first. Your about page, main service or product pages, pricing, contact details and your best guides. Ten to fifty links is plenty for most sites. Add a short note after each.
  3. Group them under H2s such as ## Services, ## Guides, ## Policies. Put nice-to-have pages under ## Optional.
  4. Save it as plain text and upload it to your site root, so it loads at https://yoursite.com/llms.txt. If you use a site builder or an SEO plugin, check whether it can generate the file for you before writing it by hand.
  5. Keep it current. A file that links to deleted pages is worse than none. Regenerate it when your main pages change; ours is rebuilt with every deploy.
  6. Check it with the script below.

Don't put anything in it you wouldn't publish. It is public, and anyone can read it.

A script to check any site's llms.txt

This is the Node.js script (version 18 or later, no packages) we used for the check above. Save it as check-llms.mjs and run node check-llms.mjs yoursite.com:

// Usage: node check-llms.mjs example.com [more.com ...]
// Fetches /llms.txt from each site and checks it against the llmstxt.org format.
const sites = process.argv.slice(2);
 
for (const site of sites) {
  const url = `https://${site}/llms.txt`;
  let res, text;
  try {
    res = await fetch(url, { redirect: 'follow', headers: { 'user-agent': 'Mozilla/5.0 (llms.txt check)' } });
    text = await res.text();
  } catch (err) {
    console.log(`${site}: could not fetch (${err.message})`);
    continue;
  }
  const type = res.headers.get('content-type') || 'none';
  const looksHtml = /^\s*<(!doctype|html)/i.test(text);
  if (!res.ok || looksHtml) {
    console.log(`${site}: no llms.txt (HTTP ${res.status}${looksHtml && res.ok ? ', HTML page instead' : ''})`);
    continue;
  }
  const lines = text.replace(/^/, '').trimEnd().split('\n');
  const firstLine = lines.find((l) => l.trim()) || '';
  const h1 = firstLine.startsWith('# ');
  const summary = lines.some((l) => l.startsWith('> '));
  const sections = lines.filter((l) => l.startsWith('## ')).length;
  const links = (text.match(/\[[^\]]*\]\([^)]+\)/g) || []).length;
  const optional = lines.some((l) => /^## Optional\s*$/i.test(l));
  console.log(
    `${site}: HTTP ${res.status}, ${type}, ${Buffer.byteLength(text)} bytes, ${lines.length} lines | ` +
      `H1 first: ${h1 ? 'yes' : 'NO'}, summary: ${summary ? 'yes' : 'no'}, ` +
      `${sections} sections, ${links} links, Optional section: ${optional ? 'yes' : 'no'}` +
      (res.url !== url ? ` | served from ${res.url}` : ''),
  );
}

Run on our own site, it printed:

thesearchpost.com: HTTP 200, text/plain; charset=utf-8, 12913 bytes, 73 lines | H1 first: yes, summary: yes, 4 sections, 55 links, Optional section: no

The "HTML page instead" case matters: some sites answer every unknown address with their normal page and a 200 code, which a quick browser check would mistake for a real file.

Our view: make one, but don't expect it to move AI citations

Make an llms.txt if you have docs, a product with an API, or a site people point AI agents at. It takes under an hour, it costs nothing to serve, and Lighthouse already checks for it. We keep one because it gives agents a clean map of the site and a full-text copy of every story.

Don't expect it to get you into AI Overviews, AI Mode, ChatGPT or Perplexity answers. Those run on their own search indexes, fed by crawlers you control with robots.txt. That is where to spend the time: let the search crawlers in (our guide to ranking in ChatGPT shows the robots.txt lines), and write pages that answer the question in their first two sentences.

What we don't know yet

  • Whether any AI search product reads llms.txt without saying so. Some site owners report seeing AI crawlers request the file in their logs; a request alone doesn't show the file changes what gets cited.
  • Whether Google will give the file any role beyond the Lighthouse check, which it calls optional "at the moment."
  • How the 16 sites above built their files, by hand or with a tool. We only checked what they serve.

Sources

Photo of Kiran Dasgupta

About the author

Kiran Dasgupta, Editor

Kiran runs Kihan Marketing, a web design, SEO and ads agency, and manages dozens of websites across real estate, home services and healthcare. He writes about SEO, AI search and paid ads news, and what it means for the people who run websites.