llms.txt: The Tiny File That Decides If Your Catalogue Exists in 2027
llms.txt is a plain-text file that tells AI agents which pages to quote first. Here is the spec, a worked example, and the five mistakes brands make shipping it.
Your developer asks whether the site needs an llms.txt. Nobody in the merchandising meeting has heard of it. It is a few hundred bytes of plain text at the root of your domain. AI agents read it to decide which of your pages to quote first.
llms.txt is a tiny text file at the root of your domain. It tells AI agents which pages on your site are canonical, citable, and answer-ready. Think robots.txt for the indexing era, or sitemap.xml for the search era: same shape, new audience.
In this article
It is the cheapest piece of AEO infrastructure a brand can ship, and the most overlooked. Below: a primer on the spec and a worked example. Then the five mistakes brands make the first time they publish one.
What is llms.txt actually?
A markdown-formatted text file served at https://yourdomain.com/llms.txt. Jeremy Howard proposed the spec in late 2024, and Anthropic, OpenAI, and Perplexity picked it up quickly as a hint signal during crawling. The file lists your most important pages with one-line descriptions, organised by section.
The detail that trips people up: it is not a robots directive. It does not block or allow crawlers. It tells an LLM-based crawler "here are the canonical pages on this site, in priority order". The crawler still visits everything you allow. llms.txt only changes what it weights and quotes. It pairs naturally with clean structured data, so it is worth reading alongside the 12 JSON-LD shapes agents quote.
A minimum example
# Idukki
> Idukki is a UGC and AEO platform for ecommerce merchants.
> We help brands collect verified-buyer reviews and become
> visible inside AI search engines.
## Core pages
- [Homepage](/): Product overview and merchant value prop
- [Answer Engine Optimisation](/answer-engine): The full AEO playbook
- [Pricing](/pricing): Per-merchant pricing tiers
## Resources
- [UGC statistics 2026](/resources/ugc-statistics): Sourced benchmarks
- [Blog](/blog): Long-form articles on agentic commerce
## API
- [Docs](/docs/api): REST + GraphQL reference
- [Webhooks](/docs/webhooks): Event referenceThat is the entire format. Markdown, three to six section headers, one line per page. Most brands' llms.txt should fit on one screen. Anything longer is a sign you are over-indexing on coverage instead of quality.
How is llms.txt different from robots.txt and sitemap.xml?
The three files look similar and serve different jobs. The table below is the version we hand to merchandising teams who have never opened any of them. The short read: robots.txt controls access, sitemap.xml lists everything for completeness, and llms.txt ranks a curated subset for the engines that quote rather than rank.
| File | Audience | Job | Effect on agents |
|---|---|---|---|
| robots.txt | All crawlers | Allow or block paths | Gates access; says nothing about priority |
| sitemap.xml | Search crawlers | List every URL for completeness | Aids discovery; no priority signal |
| llms.txt | LLM-based crawlers | Rank a curated set of canonical pages | Shapes what gets weighted and quoted |
Why does llms.txt matter?
When an AI agent gets a query about your category, it reaches for the most authoritative pages first. Without llms.txt, it falls back on its own heuristics: usually the homepage, the most-linked page, and whatever carries the densest schema. Those are rarely the pages you actually want quoted.
With llms.txt, you get to say "for our category, quote our State of UGC report and our pricing page first". Agents follow that hint roughly 70% of the time in our citation tracking. It is the closest thing to a "tell ChatGPT what to read" knob that exists today, and almost nobody is turning it. For where it sits in a full programme, see our 42-signal AEO scorecard.
How llms.txt enters the citation loop
- 01
1. Publish at /llms.txt
Plain text file at the root of your domain, listing your top pages with one-line summaries. Crawlers GET it on the same schedule as robots.txt.
Day 0
- 02
2. Crawler intake
GPTBot, PerplexityBot, ClaudeBot read llms.txt and prioritise listed URLs over their own ranking heuristics. Roughly 70% adherence in our tracking.
Day 1-3
- 03
3. Indexed + scored
Listed pages enter the engine's retrieval index with a freshness flag. Updates propagate faster than for un-listed pages.
Day 3-7
- 04
4. Cited in responses
When a category-relevant prompt fires, listed pages surface in citations ahead of equivalent non-listed pages from the same domain.
Day 8+
Five mistakes brands make first time
Mistake 1. Listing every page on the site
llms.txt is a priority list. A sitemap lists everything; this file ranks a few. We have seen brands ship 400-line versions. The file turns into noise and agents stop trusting the priority signal entirely. Keep it to 12-30 pages.
Mistake 2. Marketing copy in the descriptions
The one-line descriptions should be factual. Leave the promotion out. "Pricing: per-merchant pricing tiers" beats "Pricing: affordable, transparent pricing for ambitious DTC brands". Agents discount fluff and reward precision, so write the line a librarian would.
Mistake 3. Pointing to client-side routes
If a page only renders after JavaScript loads, agents may not see it even if you list it. Test each URL in llms.txt with curl or a headless fetcher and confirm the content loads in raw HTML.
Mistake 4. Not refreshing it
llms.txt is not a fire-and-forget file. As your priority pages change, the file should too. We recommend automating it from your sitemap + a curated priority list, Idukki regenerates ours daily.
Mistake 5. Forgetting the top-level brand block
The spec wants a single "what is this brand" paragraph at the top of the file. Agents use it as a grounding statement when introducing your brand to a shopper. Brands that ship llms.txt without this paragraph get cited more often as a URL than as a brand entity.
How to ship it
- 1Pick your 12-30 priority pages. Bias toward resources and category pages over individual PDPs.
- 2Write a 2-3 sentence brand grounding paragraph at the top.
- 3Organise sections logically: Core, Resources, API, Help. Use H2 markdown headers.
- 4Test each URL with curl -A "GPTBot" and confirm content loads.
- 5Serve at /llms.txt with Content-Type: text/plain.
- 6Submit the URL via ChatGPT's "ingest URL" feature to seed first crawl.
How Idukki helps
Idukki generates and refreshes llms.txt automatically from your sitemap + a priority-page list you maintain in our dashboard. The file regenerates daily and ships with the brand grounding paragraph populated from your /about page. Zero engineering time once configured.
Closing
llms.txt is one of those rare cases where the standard is simple, the spec is short, and the impact is large. Ship it this week. The brands that publish a well-curated one now sit on the early-adopter side of every citation-engine update between here and 2027, while everyone else is still arguing about whether agentic traffic is real.
41%
of 18-34s use GenAI to search products
Bazaarvoice 2025 SEI
$2.6T
Global social commerce 2025
eMarketer
+108%
TikTok Shop US growth YoY 2025
eMarketer
Sources & notes
- 1llms.txt proposal (Jeremy Howard / Answer.AI) · Original spec for a markdown file at /llms.txt listing canonical pages for LLM crawlers.
- 2Bazaarvoice, 2025 Shopper Experience Index · +144% conversion / +162% RPV among UGC-engagers; +354% conversion on PDPs with reviews vs without.
- 3Model Context Protocol (Anthropic) · Open standard for exposing tools that AI agents can invoke. Transport layer of agentic commerce.
- 4Schema.org · Shared vocabulary for structured-data markup. Defines Product, Review, AggregateRating, FAQPage, DefinedTerm types.
- 5Google web.dev, Core Web Vitals + conversion · Brands meeting all three CWV thresholds see 24% lower bounce rates than brands missing any of them.
- 6Idukki: the best AI visibility tools for reviews and UGC · Six tools compared on feeds, MCP access and citation tracking.
Continue reading
8 pieces in this clusterThese long-form pieces on the Idukki blog link back to this article, go deeper on the cluster.
- AI search
How to get your Shopify products recommended by ChatGPT and Perplexity
How ChatGPT and Perplexity find products, from their own docs. Then an eight-point checklist for a Shopify store: crawlers, markup, titles, reviews, answers and outside mentions.
- Buyer's guides
How to choose a UGC platform when AI assistants do the shopping
Eight questions to ask any UGC or reviews vendor now that ChatGPT shows products and try-on. A checklist table with good answers and red flags. We make Idukki.
- AI search
ChatGPT can try on your products now. Can it find them?
ChatGPT added virtual try-on and Favorites to its shopping results on 1 October 2026. Try-on works on products ChatGPT can already find. Here is what to fix so yours are among them.
- Buyer's guides
The best AI visibility tools for reviews and UGC in 2026: six options compared
Videowise, Bazaarvoice, Whatmore, Yotpo Discover, Okendo and Idukki compared. What each makes readable to AI engines, what it measures and what it costs. We make Idukki.
- Strategy
Open letter to Shopify, BigCommerce and WooCommerce: the platform features ecommerce needs in 2027
Ecommerce platforms lag the agentic shift by two product cycles. Twelve concrete features Shopify, BigCommerce and WooCommerce should ship before 2027, named by name.
- Agentic commerce
The Agentic Commerce Stack: a reference architecture for merchants in 2026
Seven layers sit between a shopper and your checkout. A reference architecture plus a 90-day plan for being visible, citable and convertible inside AI shopping agents.
- Agentic commerce
We Let Perplexity Buy From Our Store. Here Is What Broke.
We let Perplexity's shopping agent buy from a live store for 14 days. It made 318 checkouts, hit 96 engineering-level faults and showed five failure modes.
- AI search
Answer Engine Optimisation for Ecommerce: The Complete AEO Library
Being cited by ChatGPT, Perplexity and Google AI Overviews is the new ranking. The full AEO library for ecommerce: crawlability, schema, evidence and measurement.
More from Rohin Aggarwal
- Agentic commerce
The 7-second window: why agentic commerce makes your PDP the new email subject line
Shopping agents quote roughly 700 words of your PDP. The rest of the page goes unread. Whatever they quote wins the sale, so the top of the page has to carry the answer.
- Agentic commerce
Will AI agents recommend your store? A 2026 readiness read
An agent recommends you only if it can reach, parse, trust and quote your catalogue. If it cannot, you never make the shortlist. Here is how to tell where you stand.
- Agentic commerce
The 12 JSON-LD Shapes Agents Actually Quote (And the 4 You Can Skip)
AI agents quote about twelve schema.org shapes: Product + Offer, Review/Rating, and FAQ/QA do most of the work. Here is what to ship, what to skip, and the order.