# llms.txt: The Tiny File That Decides If Your Catalogue Exists in 2027

llms.txt is a plain-text file that tells AI agents which pages to quote first. Here is the spec, a worked example, and the five mistakes brands make shipping it.

By Rohin Aggarwal · 2026-04-05 · (updated 2026-11-11)

Your developer asks whether the site needs an llms.txt. Nobody in the merchandising meeting has heard of it. It is a few hundred bytes of plain text at the root of your domain. AI agents read it to decide which of your pages to quote first.

llms.txt is a tiny text file at the root of your domain. It tells AI agents which pages on your site are canonical, citable, and answer-ready. Think robots.txt for the indexing era, or sitemap.xml for the search era: same shape, new audience.

**Quick answer**

- llms.txt is a markdown file at /llms.txt. It lists your canonical pages in priority order for AI crawlers
- It is a hint signal. It changes what agents weight; robots.txt still decides what they can access
- Agents follow the priority hint roughly 70% of the time in Idukki citation tracking
- Keep it to 12-30 pages, factual descriptions, server-rendered URLs, and refresh it

It is the cheapest piece of AEO infrastructure a brand can ship, and the most overlooked. Below: a primer on the spec and a worked example. Then the five mistakes brands make the first time they publish one.

## What is llms.txt actually?

A markdown-formatted text file served at https://yourdomain.com/llms.txt. Jeremy Howard proposed the spec in late 2024, and Anthropic, OpenAI, and Perplexity picked it up quickly as a hint signal during crawling. The file lists your most important pages with one-line descriptions, organised by section.

The detail that trips people up: it is not a robots directive. It does not block or allow crawlers. It tells an LLM-based crawler "here are the canonical pages on this site, in priority order". The crawler still visits everything you allow. llms.txt only changes what it weights and quotes. It pairs naturally with clean structured data, so it is worth reading alongside [the 12 JSON-LD shapes agents quote](/blog/twelve-json-ld-shapes-agents-quote).

## A minimum example

```markdown
# Idukki

> Idukki is a UGC and AEO platform for ecommerce merchants.
> We help brands collect verified-buyer reviews and become
> visible inside AI search engines.

## Core pages

- [Homepage](/): Product overview and merchant value prop
- [Answer Engine Optimisation](/answer-engine): The full AEO playbook
- [Pricing](/pricing): Per-merchant pricing tiers

## Resources

- [UGC statistics 2026](/resources/ugc-statistics): Sourced benchmarks
- [Blog](/blog): Long-form articles on agentic commerce

## API

- [Docs](/docs/api): REST + GraphQL reference
- [Webhooks](/docs/webhooks): Event reference
```

That is the entire format. Markdown, three to six section headers, one line per page. Most brands' llms.txt should fit on one screen. Anything longer is a sign you are over-indexing on coverage instead of quality.

## How is llms.txt different from robots.txt and sitemap.xml?

The three files look similar and serve different jobs. The table below is the version we hand to merchandising teams who have never opened any of them. The short read: robots.txt controls access, sitemap.xml lists everything for completeness, and llms.txt ranks a curated subset for the engines that quote rather than rank.

| File | Audience | Job | Effect on agents |
| --- | --- | --- | --- |
| robots.txt | All crawlers | Allow or block paths | Gates access; says nothing about priority |
| sitemap.xml | Search crawlers | List every URL for completeness | Aids discovery; no priority signal |
| llms.txt | LLM-based crawlers | Rank a curated set of canonical pages | Shapes what gets weighted and quoted |

_How the three root files differ. The first two predate agentic crawling; only llms.txt was designed for it._

## Why does llms.txt matter?

When an AI agent gets a query about your category, it reaches for the most authoritative pages first. Without llms.txt, it falls back on its own heuristics: usually the homepage, the most-linked page, and whatever carries the densest schema. Those are rarely the pages you actually want quoted.

With llms.txt, you get to say "for our category, quote our State of UGC report and our pricing page first". Agents follow that hint roughly 70% of the time in our citation tracking. It is the closest thing to a "tell ChatGPT what to read" knob that exists today, and almost nobody is turning it. For where it sits in a full programme, see our [42-signal AEO scorecard](/blog/the-42-signal-aeo-scorecard).

> **A note on llms-full.txt:** A sibling spec, llms-full.txt, inlines the full content of your top pages directly into the file. Useful for brands with under 20 high-value pages; overkill for catalogues. Most stores should ship llms.txt and skip llms-full.txt for now.

**How llms.txt enters the citation loop**
1. **1. Publish at /llms.txt** — Plain text file at the root of your domain, listing your top pages with one-line summaries. Crawlers GET it on the same schedule as robots.txt. (Day 0)
2. **2. Crawler intake** — GPTBot, PerplexityBot, ClaudeBot read llms.txt and prioritise listed URLs over their own ranking heuristics. Roughly 70% adherence in our tracking. (Day 1-3)
3. **3. Indexed + scored** — Listed pages enter the engine's retrieval index with a freshness flag. Updates propagate faster than for un-listed pages. (Day 3-7)
4. **4. Cited in responses** — When a category-relevant prompt fires, listed pages surface in citations ahead of equivalent non-listed pages from the same domain. (Day 8+)
_Four-stage path from publish to quoted citation. Timing varies by engine and crawl frequency._

## Five mistakes brands make first time

### Mistake 1. Listing every page on the site

llms.txt is a priority list. A sitemap lists everything; this file ranks a few. We have seen brands ship 400-line versions. The file turns into noise and agents stop trusting the priority signal entirely. Keep it to 12-30 pages.

### Mistake 2. Marketing copy in the descriptions

The one-line descriptions should be factual. Leave the promotion out. "Pricing: per-merchant pricing tiers" beats "Pricing: affordable, transparent pricing for ambitious DTC brands". Agents discount fluff and reward precision, so write the line a librarian would.

### Mistake 3. Pointing to client-side routes

If a page [only renders after JavaScript loads](/blog/highest-converting-page-invisible-ai), agents may not see it even if you list it. Test each URL in llms.txt with curl or a headless fetcher and confirm the content loads in raw HTML.

### Mistake 4. Not refreshing it

llms.txt is not a fire-and-forget file. As your priority pages change, the file should too. We recommend automating it from your sitemap + a curated priority list, Idukki regenerates ours daily.

### Mistake 5. Forgetting the top-level brand block

The spec wants a single "what is this brand" paragraph at the top of the file. Agents use it as a grounding statement when introducing your brand to a shopper. Brands that ship llms.txt without this paragraph get cited more often as a URL than as a brand entity.

## How to ship it

1. Pick your 12-30 priority pages. Bias toward resources and category pages over individual PDPs.
2. Write a 2-3 sentence brand grounding paragraph at the top.
3. Organise sections logically: Core, Resources, API, Help. Use H2 markdown headers.
4. Test each URL with curl -A "GPTBot" and confirm content loads.
5. Serve at /llms.txt with Content-Type: text/plain.
6. Submit the URL via ChatGPT's "ingest URL" feature to seed first crawl.

## How Idukki helps

Idukki generates and refreshes llms.txt automatically from your sitemap + a priority-page list you maintain in our dashboard. The file regenerates daily and ships with the brand grounding paragraph populated from your /about page. Zero engineering time once configured.

## Closing

llms.txt is one of those rare cases where the standard is simple, the spec is short, and the impact is large. Ship it this week. The brands that publish a well-curated one now sit on the early-adopter side of every citation-engine update between here and 2027, while everyone else is still arguing about whether agentic traffic is real.

**In one line:** llms.txt is a few hundred bytes that tell agents which pages to quote first. A primer on the spec, a worked example, and the five mistakes brands make the first time they ship it.

- **41%** — of 18-34s use GenAI to search products (Bazaarvoice 2025 SEI)
- **$2.6T** — Global social commerce 2025 (eMarketer)
- **+108%** — TikTok Shop US growth YoY 2025 (eMarketer)

_AEO + AI shopping benchmarks._

### Sources & notes
- [llms.txt proposal (Jeremy Howard / Answer.AI)](https://llmstxt.org/) — Original spec for a markdown file at /llms.txt listing canonical pages for LLM crawlers.
- [Bazaarvoice, 2025 Shopper Experience Index](https://www.bazaarvoice.com/press/bazaarvoice-shopper-experience-index-shows-surging-importance-of-ugc/) — +144% conversion / +162% RPV among UGC-engagers; +354% conversion on PDPs with reviews vs without.
- [Model Context Protocol (Anthropic)](https://modelcontextprotocol.io/) — Open standard for exposing tools that AI agents can invoke. Transport layer of agentic commerce.
- [Schema.org](https://schema.org/) — Shared vocabulary for structured-data markup. Defines Product, Review, AggregateRating, FAQPage, DefinedTerm types.
- [Google web.dev, Core Web Vitals + conversion](https://web.dev/articles/vitals) — Brands meeting all three CWV thresholds see 24% lower bounce rates than brands missing any of them.
- [Idukki: the best AI visibility tools for reviews and UGC](/blog/best-ai-visibility-tools-for-reviews-and-ugc-2026) — Six tools compared on feeds, MCP access and citation tracking.

---
Canonical: https://idukki.io/blog/llms-txt-tiny-file-decides-catalogue-2027
Tags: llms-txt, aeo, infrastructure, standards
