- llms.txt is a proposed Markdown file at the root of a website that gives AI tools a short, curated map of the pages that matter most
- Jeremy Howard of Answer.AI published the proposal in September 2024, and a revised version 2 followed in August 2026
- Google states that Google Search ignores llms.txt, and no major AI platform has confirmed it as a ranking or citation signal
- The file is cheap to publish and useful for developer documentation and AI agents, but it will not decide whether ChatGPT recommends your business
- robots.txt access for AI search crawlers, and your presence on the third-party sources AI tools quote, matter far more
llms.txt is a plain text file, written in Markdown, that sits at the root of a website and tells large language models which pages matter most. Since late 2024 it has become one of the most recommended quick wins in AI search optimisation, and one of the least understood. I run one on kaanturk.com and I have read what each major AI platform has published about it. This guide explains what the file does, what it does not do, and where it belongs in a sensible Search Engine Optimisation plan for an Australian business.
What is llms.txt?
llms.txt is a Markdown file published at yourdomain.com/llms.txt that summarises a website for large language models. It holds a title, a short summary and lists of key pages with one-line descriptions. It is a proposal, not an adopted web standard, and its job is to save AI tools from guessing which of your pages matter.
It lives beside robots.txt but does a different job. robots.txt gives crawlers permissions; llms.txt gives language models a briefing, like the one-page brief you would hand a new contractor on day one.
Because it is Markdown, a model can read it without wading through menus and scripts. The question business owners care about is whether anything that decides AI answers reads it. For terms such as AI crawler and generative engine optimisation (GEO), see my SEO and AI search glossary.
Who proposed llms.txt and why?
Jeremy Howard of Answer.AI proposed llms.txt in September 2024. His problem was practical: AI tools have limited context windows and struggle to read whole websites full of navigation, scripts and adverts. A short Markdown index lets a model or coding assistant find the right documentation quickly, especially for software libraries and APIs.
The proposal at llmstxt.org is short and clear about its scope. Its examples are developer documentation, programming libraries and project websites. Most of the marketing hype came later, when the file was repackaged as an AI ranking tactic for every kind of business.
Version 2 arrived in August 2026 with practical changes: standard link relations so pages can point to their llms.txt, two accepted URL forms for Markdown copies of pages, and subfolder files that cover the pages beneath them. It still contains no statement from OpenAI, Google, Anthropic or Perplexity that their answer engines read the file.
Does ChatGPT, Google or Perplexity use llms.txt?
No major AI platform has confirmed that it reads llms.txt to rank, cite or recommend websites. Google states in writing that Google Search, including AI Overviews and AI Mode, ignores the file. OpenAI, Perplexity and Anthropic document their crawlers in detail, and none of that documentation says those crawlers use a website's llms.txt.
Here is what each one has published, checked on 2 October 2026.
Google. Its guide to optimising for generative AI features on Google Search, updated 10 July 2026, says you do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search. It names llms.txt directly: keeping one for other systems is fine, but it will neither help nor harm your Google visibility, because "Google Search ignores them".
Chrome. Lighthouse's experimental agentic browsing audits include an llms.txt check that flags a server error and marks a missing file as not applicable, because the file is optional. It concerns AI agents moving around a site, not search rankings.
OpenAI. Its crawler documentation covers OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, all controlled through robots.txt. It says nothing about reading your llms.txt, although it links to an llms.txt index of OpenAI's own documentation.
Perplexity and Anthropic. Perplexity's bot documentation and Anthropic's crawler help article describe their crawlers and robots.txt controls. Neither says the crawlers read site owners' llms.txt files. Perplexity publishes one for its own docs, which is not the same thing as reading yours.
The best usage data I have seen is Ahrefs' analysis of llms.txt. Across 137,000 domains in its web analytics product, 28% published the file. Of roughly 38,000 domains with a valid file, 97% received zero requests for it in May 2026. The sample leans towards AI-aware sites, but the gap between publishing and reading is plain.
I'll be direct: if someone sells you llms.txt as an AI SEO tactic, ask which platform reads it when choosing sources, and where that platform has written it down.
How is llms.txt different from robots.txt and sitemap.xml?
robots.txt tells crawlers what they may and may not fetch, and the major AI platforms say they honour it. sitemap.xml lists the URLs you want indexed. llms.txt is a short, hand-picked index with descriptions, written for language models. Only robots.txt changes what AI search crawlers can access, so it is the root file to check first.
| File | Written for | Effect on AI answers |
|---|---|---|
| robots.txt | Search engine and AI crawlers | Direct: a blocked search crawler cannot fetch your pages |
| sitemap.xml | Search engine indexers | Indirect: helps pages get indexed and retrieved |
| llms.txt | Language models and AI agents | None confirmed by any major platform |
The detail that matters sits inside robots.txt, because each AI company runs separate crawlers for separate jobs. OAI-SearchBot decides whether you can appear in ChatGPT search, while GPTBot collects training content. Claude-SearchBot and ClaudeBot split the same way, and PerplexityBot surfaces sites in Perplexity's answers. Googlebot controls Google Search, including AI Overviews and AI Mode, while the Google-Extended token covers Gemini training and has no effect on Search inclusion or ranking.
So you can block the training crawlers and still allow the ones that decide whether you can be cited. When I run a technical SEO audit, robots.txt is the first root file I check for AI search, because one rule there can remove a site from ChatGPT's search results, while a missing llms.txt costs nothing measurable.
What should an llms.txt file contain?
A valid llms.txt needs only an H1 with the site or project name. Useful files add a blockquote summary, a few lines of key facts, and H2 sections that list important pages as Markdown links with one-line descriptions. An Optional section can hold secondary links that an AI tool may skip when it is short on context.
Here is a shortened version of the file I run on kaanturk.com:
# Kaan TURK, SEO & AI Search Consultant, Australia
> Kaan TURK is a Melbourne-based SEO & AI Search Consultant who helps
> Australian and New Zealand businesses grow through Search Engine
> Optimisation on Google and in AI search tools.
## Start here
- [AI SEO services](https://www.kaanturk.com/ai-search): getting cited and recommended in ChatGPT, Google AI Overviews and Perplexity
- [AI search case study](https://www.kaanturk.com/case-studies/dental-seo-ai-search-optimisation): a Blacktown dental clinic recommended by ChatGPT two weeks after a missing review profile was fixed
## How to cite
- **Name:** Kaan TURK
- **Role:** SEO & AI Search Consultant
I rewrote that file on 2 October 2026. The decisions behind it are more useful than the template:
- Facts match the rest of the site word for word. The job title, location and credentials are the same strings used on my About page and in my structured data, so a model never sees two versions of the truth.
- Fewer links, each with a reason. The sitemap already lists every URL.
- A "How to cite" section. I have no evidence that any model follows it. It simply states the facts once.
- Commercial detail came out. The earlier version carried commercial terms I no longer publish elsewhere on the site. An llms.txt goes stale like any other page, and nobody notices because nobody looks at it.
How do you create and publish an llms.txt file?
Write the file in a plain text editor using Markdown, save it as llms.txt, and upload it to the root of your domain so it loads at yourdomain.com/llms.txt as raw text. Keep every fact consistent with your About and service pages, list a short set of key URLs with descriptions, and add a last-updated date.
- Choose the pages an assistant should read first: usually the homepage, main service pages, About page, one or two case studies and the contact page.
- Write the H1 and a one-paragraph summary in plain facts. No slogans.
- Group the links under H2 headings with a one-line description after each.
- Publish it at the root. On a Next.js site like mine, it sits in the public folder. On most other platforms your developer or host uploads it to the root directory.
- Check that it loads as raw text, not an error or a styled page.
- Put it on your maintenance list next to the sitemap, and review it whenever your services or business details change.
Does your website need an llms.txt file?
Most small and mid-sized Australian businesses do not need llms.txt to appear in AI answers, and it will not change Google rankings. It is worth publishing if you have developer documentation, an API, a large help centre or detailed product specifications that AI agents read. For everyone else it is low-cost, low-impact housekeeping.
| Site type | My recommendation |
|---|---|
| Software product with documentation or an API | Publish it, plus Markdown copies of key pages. This is the use case the proposal was written for |
| eCommerce store with a large catalogue | Optional. Crawl access, product structured data and clean category pages do the real work |
| Local service business, such as a clinic or trade | Optional, after the basics. AI recommendations draw on reviews, directories and your Google Business Profile |
| Professional services site | Optional. Useful as a consistent fact sheet, not as a ranking tactic |
I publish one because my site is about AI search and it takes minutes to review. On a limited budget, spend that hour on robots.txt, your Google Business Profile and your industry's review platforms first.
What decides whether AI tools cite your business?
AI tools cite pages they can crawl, retrieve and trust. Google's AI features draw on its normal Search index and ranking systems. ChatGPT and Perplexity run their own search crawlers and also draw on third-party sources such as review sites and directories. Crawl access, consistent business facts and presence on those sources matter far more than llms.txt.
Google's generative AI guide describes the mechanism: retrieval augmented generation built on core Search ranking systems, plus query fan-out, where the model runs related searches. The route into an AI Overview is the route into Google's index. No file at your root sits on that path.
The clearest example from my own work is a dental clinic in Blacktown, in Western Sydney. It had more than 500 five-star Google reviews, yet ChatGPT did not mention it when asked for a dentist in Blacktown. When I traced where ChatGPT's dental recommendations came from, the answer was BirdEye, a review platform where the clinic had no profile. I created and claimed the profile and completed every field with consistent name, address and phone details. Within two weeks, ChatGPT listed the clinic among its top recommendations.
The fix was a profile on a third-party review site, not a file on the clinic's server. In GA4, ChatGPT went on to deliver 89.66% of the clinic's AI-platform referral sessions, and AI referral visitors engaged at 81%. The absolute numbers were modest, as the dental clinic AI search case study shows, but the lesson holds: AI visibility is decided by what the platforms can find and trust about you, mostly off your own site.
That is why my AI SEO services start with crawler access, entity facts and the sources AI tools cite. The useful next step is not a file but a baseline, and my guide on how to measure AI visibility shows how I build one.
Frequently Asked Questions
Is llms.txt a Google ranking factor?
No. Google's guidance on generative AI features says Google Search ignores llms.txt files, so publishing one neither helps nor harms your rankings, AI Overviews or AI Mode. Visibility in Google still comes from pages that are crawlable, indexable and useful.
Is llms.txt mandatory?
No. It is an optional proposal with no standards body enforcing it. Even Chrome's Lighthouse audit treats a missing llms.txt as not applicable and only flags a file that returns a server error.
What is llms-full.txt?
It is a companion file, popularised by documentation platforms, that puts a site's full documentation into one Markdown document for loading into an AI tool. It suits software docs and adds little for a service business.
Can llms.txt hurt my website?
Not in Google Search, which ignores it. The real risks are an outdated file that contradicts your current pages, or one list that makes your best content easier to scrape.
How can I tell whether AI crawlers request my llms.txt?
Check your server or CDN logs for requests to /llms.txt and note the user agent, such as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. Google Analytics will not show these requests, because crawlers do not run its tracking script.


