llms.txt: how to help AI crawlers find your content

A financial advisor in Nairobi searched for his own firm's specialty in ChatGPT. The response was thorough, named four competitors by firm name, and cited their published articles on retirement planning. His firm had written more on the topic than any of them, over three years and forty posts. The AI had not mentioned him once. His website had no clear signal telling AI systems where the useful content was. So the AI looked elsewhere.

What is an llms.txt file?

An llms.txt file is a plain text document placed at the root of your website, beside your robots.txt, telling AI language models which pages on your site are worth reading. Where robots.txt speaks to search engine crawlers, llms.txt speaks to AI systems such as ChatGPT, Perplexity, Claude, and Google's AI Overviews, so the model doesn't have to guess.

Key takeaways

• An llms.txt file is a plain text document at your site's root directing AI crawlers to your most valuable content.
• It is a proposed standard, not yet a universally adopted one, but adoption among AI tools is growing and the cost of adding one is low.
• The file uses Markdown format: a title, a short description, and a list of links with plain-language labels.
• It doesn't replace robots.txt or your sitemap; it works alongside them for a different audience.
• AI search visibility depends on more than one signal, but an llms.txt file removes a real friction point between your content and the systems deciding whether to cite it.
• Sites without one leave the citation decision entirely to the AI, which will follow the easiest path through your content, not the most informative one.

Why AI crawlers need different instructions from search crawlers

Traditional search crawlers and AI crawlers are solving different problems, and they need different instructions to solve them well. A traditional crawler like Googlebot moves through a site page by page, following links, reading HTML, and checking how fast each page loads. Google's documentation on JavaScript crawling behaviour shows pages with heavy JavaScript can sit in a rendering queue for days before Googlebot processes the full content. AI crawlers face a version of the same problem: a site full of sidebars, navigation menus, cookie banners, and footer links gives the model a noisy signal about what the page says.

An llms.txt file cuts through that noise. Instead of asking an AI model to infer which of your two hundred pages are worth reading, the file lists them directly, with a plain-language label for each. The model gets a clean index of your content without having to parse layout and navigation to find it. That is a meaningful advantage, especially for sites where the best content is buried several clicks from the homepage and would take a crawler several visits to find and assess.

What the file contains

The format is simple enough to write in a text editor. The file uses Markdown: a title at the top using a single hash symbol, a short paragraph describing the site and its purpose, and then one or more sections listing links with plain-language descriptions. An optional second file, named llms-full.txt, contains the full text of your key pages rather than links to them, for models ingesting content directly rather than following URLs.

A minimal example looks like this:

The file is intentionally lean. It doesn't embed metadata tags, doesn't use JSON, and doesn't require a developer to write. If you can create a text file and upload it to your server's root directory, you can add an llms.txt file today. The proposed standard for AI content crawling calls for a human-readable document AI systems can parse without effort, not a complex specification requiring technical implementation.

Which content belongs in the file

This is where most sites go wrong. The instinct is to list everything, the way a sitemap does. That approach defeats the purpose. A sitemap exists to ensure nothing is missed. An llms.txt file exists to ensure the right things are found first. The two documents serve opposite objectives.

Your llms.txt file should contain your highest-signal content: the pages defining your expertise, answering the questions your prospective clients are searching, and carrying the most useful information in a readable form. For a professional services firm, that typically means long-form guides, explainer articles, and pages where you set out a clear position on a question in your field. For an e-commerce business, it might mean buying guides, product comparison pages, and category explainers. Thin pages, duplicate content, and landing pages built around conversion rather than information belong elsewhere.

A useful test: if an AI system were to quote one sentence from a page, would that sentence help a reader? If the page is mostly a call to action with minimal text, leave it out of the file. If the page contains a clear, specific answer to a question your audience asks, it belongs in the file with a label naming what the page covers.

How llms.txt fits alongside robots.txt and your sitemap

Adding an llms.txt file doesn't change how robots.txt works. The two files sit at the same level on your server but speak to different readers. Your robots.txt file controls which crawlers can access which parts of your site; Google's guidance on robots.txt and crawl control confirms blocking crawlers during a site build is a standard practice, and the same logic applies when you want to exclude AI crawlers from certain content. If you want AI crawlers to skip a section of your site entirely, robots.txt is the right place to say so. Your llms.txt file isn't a restriction mechanism; it is a navigation aid.

Your sitemap and your llms.txt file also have different jobs. A sitemap is a comprehensive inventory of every indexable URL on your site. An llms.txt file is a curated reading list. The sitemap is a full catalogue; the llms.txt file is a recommended reading guide sitting at the front of it, pointing AI systems directly to what is most worth their attention.

Comparison of three files AI and search crawlers use

How widely AI systems read llms.txt files

Here is the part the industry doesn't advertise loudly enough. An Ahrefs study of llms.txt adoption across 137,000 sites found 97 percent of llms.txt files received no AI crawler requests at all. Valid files drew 96 percent bot traffic when requests did arrive, but the majority of sites with a file weren't seeing consistent AI crawler visits. This isn't a reason to skip the file. It is a reason to understand what the file does and doesn't guarantee.

An llms.txt file doesn't force an AI system to read your content or cite your site. What it does is remove a friction point. A model visiting your site has a cleaner path to your best content. A model building a knowledge base from publicly available pages, as most of them do, including ChatGPT, Claude, and Gemini, according to analysis of llms.txt adoption and impact, encounters your content in a legible, structured form rather than having to pick through layout and navigation to find the substance. That counts when a model is deciding between two sites covering the same topic: the cleaner signal tends to win.

The standard is also still developing. More AI tools are adding support for the format over time, and sites with files already in place are positioned to benefit as that support expands. The cost of adding the file is an hour of work. Waiting while others build this into their sites creates a signal gap you'll need to close at a higher cost later.

Writing the llms.txt file your site needs

Start by auditing your content for the pages with the clearest, most complete answers to questions in your field. These are your anchors. Group them by topic in the file, because AI systems reading the file learn about your content's shape from the structure you give it. A file with fifteen links in a single unsorted list gives less signal than a file with fifteen links arranged under three topic headings.

Write descriptions naming what each page covers, not what the page hopes to achieve. "Introduction to our services" tells a model almost nothing. "Plain-language guide to VAT registration for sole traders" tells a model exactly what it will find and which queries that page is relevant to. The description is how the model decides whether to follow the link. Treat it like a label on a file folder, not a headline written to attract clicks.

DataForSEO's implementation of their own llms.txt file is a useful reference: they list every documented API endpoint with a plain description of what each endpoint does, making it straightforward for a model to understand the full scope of their documentation without crawling hundreds of individual pages. The principle scales down to a ten-page business website as well as it applies to technical documentation.

Update the file when you publish significant new content, and remove links when pages are substantially changed or retired. A file pointing to outdated or redirected pages gives an AI system a broken reading list and undermines the signal you are trying to send. AI search visibility and structured data for AI are both served by keeping your signals current and accurate.

Closing reflection

The businesses showing up in AI answers over the next two years won't necessarily be the ones with the most content. They will be the ones whose content was easiest for AI systems to find, read, and quote from. An llms.txt file isn't a guarantee of that outcome. It is one of the cleaner signals available right now, at a point when most sites still haven't sent it. Adding one while adoption is low costs almost nothing. Skipping it while competitors build this into their sites creates a gap you'll need to close later at greater effort.

You shouldn't have to guess whether AI crawlers can find your best content. With Zahavah Studio, you won't.

Contact Zahavah Studio to audit your AI search visibility signals and build a clear plan for getting your content cited.

A few questions come up consistently when business owners start looking at llms.txt: whether it affects their existing SEO, how often it needs updating, and whether it works if the site is built on a platform they don't directly control. The answers are more practical than the questions suggest.

Frequently asked questions

How do I know if my llms.txt file is working?

There's no single dashboard showing you how often AI crawlers read your llms.txt file, and that is a limitation worth naming honestly. What you can check is your server access logs: look for requests to `/llms.txt` from known AI crawler user agents, including GPTBot for ChatGPT and ClaudeBot for Anthropic's Claude. A request in the log means the file was fetched. It doesn't confirm the content was used in a response, because AI training and retrieval aren't fully transparent. The more practical measure is whether your site begins appearing in AI-generated answers for queries in your area of expertise. Run a set of test queries in Perplexity, ChatGPT, and similar tools once a month, note which competitors appear, and track whether your content starts being cited. Your server logs tell you the file was read; citation tracking over three to six months tells you whether it is producing the result you are after. Use both together for a clearer picture than either provides alone.

Does an llms.txt file affect my Google rankings?

An llms.txt file has no direct effect on how Google ranks your pages in traditional search results. Google hasn't indicated it reads or acts on the file in its classic search index. The file is aimed at large language model systems, not at Googlebot. Improving the structure of your content, which is part of writing a useful llms.txt file, tends to support overall content quality, and content quality does affect Google's assessment of your pages over time. According to Google's April 2023 office hours, file accessibility and crawl signals remain important for overall search performance, but an llms.txt file isn't one of those signals. Keep it in its lane: it is a tool for AI search visibility, not a traditional SEO lever. Treat it as one layer of a broader content strategy rather than a shortcut to higher rankings in either channel.

How often should I update my llms.txt file?

Update your llms.txt file whenever you publish a new piece of content representing a significant, standalone answer to a question in your field, or whenever you substantially revise an existing page already listed. There's no fixed schedule, and monthly updates aren't necessary unless you're publishing at that pace. Accuracy counts for more than frequency: a file pointing to pages no longer existing, or to pages whose content has changed significantly, sends a confused signal to any AI system reading it. A quarterly review is a reasonable default for most small and medium businesses, coinciding with any content audit you already run. Remove retired pages, add strong new content, and update descriptions where the page's focus has shifted. Keeping the file current is a ten-minute task when done regularly; letting it drift for a year turns it into a project.

Do I need a developer to add an llms.txt file to my site?

For most websites, no. An llms.txt file is a plain text document written in Markdown, which requires no coding knowledge. You write the file in any text editor, save it as `llms.txt`, and upload it to the root directory of your site, the same folder containing your homepage files. On WordPress, that typically means using an FTP client or your hosting provider's file manager to place the file at the top level of your public directory. On other platforms, the process is similar. If your site is managed entirely through a hosted platform where you can't access the root directory, you may need to ask your developer or hosting provider to add the file. The publishing workflow your site runs on determines whether this is a five-minute task or a short developer request. Either way, the file itself takes longer to plan than to write.

Is an llms.txt file the same thing as a robots.txt file for AI?

They are related but different. A robots.txt file is a set of instructions telling crawlers what they may and may not access on your site; it controls access. An llms.txt file is a curated reading list telling AI systems which content is most worth reading; it guides attention. You can use both together: robots.txt to block AI crawlers from sections of your site you don't want ingested, and llms.txt to highlight the sections you do. The files serve different functions and neither replaces the other. A site with a robots.txt file but no llms.txt is managing access without managing guidance. Adding an llms.txt file adds the second layer, pointing AI systems toward your best content rather than leaving them to find it on their own. For your purposes as a site owner, the distinction is straightforward: one file says what AI can see, the other says what is worth seeing.

Everything on this blog is written to inform and educate. It is for information only. Nothing here is professional legal, financial, or technical advice. If you are making a significant business decision, speak to a qualified professional first. Zahavah Studio works hard to keep this content accurate and current, but is not liable for decisions made based on what you read here.

Leave a comment