The llms.txt file: how e-commerce brands can get products discovered by AI
Most site owners know robots.txt and sitemap.xml, the behind-the-scenes files that guide search engine crawlers. A new file is joining them: llms.txt, designed specifically for large language models like ChatGPT, Gemini and Perplexity. Where robots.txt says what may be crawled and sitemap.xml maps pages for search engines, llms.txt acts as an instruction manual for AI bots, pointing them to the content that answers customer questions about products, brand, shipping and policies.
It matters because AI agents are changing how people shop: buyers increasingly ask an assistant for the best product meeting their criteria, and the AI chooses based on data it can find. Products whose information is hard for AI to parse are invisible to these shoppers. Setting up llms.txt means creating the file at the website root and listing direct links, one per line, to product feeds in CSV or JSON, category pages, detailed FAQs, shipping and return policies, and the brand story, with plain comments marking each section. Best practices: update it when products or policies change, keep links publicly accessible without login, and prioritize pages with clear, rich content. A simple starting point is exporting the product catalog as a CSV, hosting it at a public unlisted link, and referencing it in the file.
A related history note explains why AI shopping suddenly works: the 2017 research paper 'Attention Is All You Need' introduced the Transformer, letting computers weigh the most important words in a sentence wherever they appear rather than reading word by word. That breakthrough underpins today's chatbots, voice assistants and shopping agents, and for sellers it means smarter product search, AI agents that compare and buy on a shopper's behalf, round-the-clock automated customer service and more personalized recommendations.
Google used its I/O 2025 conference to put AI at the center of the consumer journey. AI Mode, a conversational search experience, is now live for all US users, making optimization for AI-generated answers, through schema markup and structured content, as important as classic SEO. Visual commerce is surging, with Google Lens usage up 65 percent year over year, a Search Live video feature, and photorealistic virtual try-on for fashion. A new AI agent can complete checkout details and apply coupons, while price tracking lets shoppers get instant drop alerts. Gemini 2.5 Pro and Project Astra promise proactive, context-aware assistance, and creation tools including Imagen 4, Veo 3 and Stitch automate images, video and landing pages, with SynthID watermarking AI content. Android XR smart glasses point to an augmented shopping future. The strategic advice: audit structured data, invest in video and 3D assets, test sites against AI crawlers, rethink pricing for an era of automated deal-watching, and prototype assistants that rescue carts.
Review compliance remains a live risk after a nine-figure seller reportedly lost $2 million in 10 days over review manipulation accusations; free educational guides circulating in the seller community walk through what reviews are, how to earn them legitimately and how to stay out of trouble.
Quick hits: an Amazon report credits sellers with generating $2.5 trillion and creating more than 2 million US jobs; Amazon paid out more than $1 billion refunding returns made years ago; TikTok laid off hundreds of US employees including key business development roles; and according to Pattern's data, 71 percent of shoppers who first find a product on TikTok ultimately purchase it on Amazon.
