Do You Need llms.txt? HTML, Markdown and Discovery Files

Separate authoring, publishing, and discovery

Format discussions become confusing when three different layers are treated as one. Authoring is how a team writes content. Publishing is what a public URL returns. Discovery is how people or tools find that URL.

A team can write an article in Markdown, build it into HTML, and link to it from a blog index. Another team can write semantic HTML directly and publish the same kind of page. Neither choice alone tells you whether the content is accurate, useful, or discoverable.

An optional file such as llms.txt belongs to a separate question: does a particular consumer use that file as a guide to the site's content? A file's existence is not evidence that a search engine used it, cited the site, or sent a customer.

What a format experiment can and cannot show

Profound's Markdown experiment reported no statistically significant difference in AI bot traffic between its tested Markdown and HTML conditions. That result concerns the sample, timeframe, and traffic metric studied. It does not prove that all formats behave identically in every system, and it does not measure business enquiries.

The useful response is to test a real need before adding another publishing system. Ask which tool needs the representation, what problem it has with the current output, and how success will be measured. “AI prefers this” is too vague to be an acceptance criterion.

If an integration demonstrably consumes Markdown, providing it may be worthwhile. If the only issue is that an essential service description appears after several scripts run, fixing the public HTML is a more direct response to that problem.

Choose by usefulness and maintenance cost

Every additional representation creates a consistency obligation. A service price in HTML, Markdown, a PDF, and an AI directory can diverge if four people edit four copies. That is an operational problem even before search behavior enters the discussion.

Use one approved content source

Choose one source for article titles, descriptions, dates, and body content. Generate the public page and related discovery files from that source where practical. A new article should not require a person to remember five separate URL lists.

For a small static site, a build can generate the article HTML, blog index, sitemap, and feed together. The build can reject duplicate routes, missing descriptions, broken related-article references, or invalid dates before replacing the last good output.

Keep the writing source out of the public output unless publishing it serves a purpose. A content repository may contain editorial notes that are not meant to become public copy. Distinguish those internal notes from the article the reader receives.

When a public alternate representation is needed, use the same approved text and stable links. Check how its intended consumer discovers and interprets it. Do not assume a filename or response format creates a universal protocol.

How we keep FoundUp’s public formats aligned

We found that our earlier llms-full file contained an expanded company description plus article links, rather than the full page text its name suggested. It now derives readable text from the same rendered pages. Navigation, scripts and form controls are omitted; source URLs identify the canonical pages. This removes a separate manual copy of every article.

At the September 28 observation, these counts described the local release and did not confirm live publication or crawler adoption. The later public check is recorded below. Maintaining these files improves consistency for consumers that use them. It does not add a guarantee of indexing or AI citation.

Verify the output that a customer receives

After a build, open the final URL directly and inspect the response without relying on a navigation click. Confirm the main heading, summary, service facts, and contact route are present. Verify that relative asset paths still work on a nested article URL.

Then follow internal links and request an invalid URL. A static server that sends the homepage for every unknown path may hide broken links during development. Test with the deployment platform's routing behavior before drawing conclusions.

For optional llms files, check that the content contains only public facts and valid destinations. Keep access controls in the appropriate systems; a text directory is not a security mechanism. Remove stale descriptions when services change.

Finally, measure the outcome that motivated the change. A smaller response may help a particular consumer, but a rise in bot requests does not establish more citations or customers. Use the technical audit to verify delivery and the measurement guide to keep outcomes separate.

Check the platform before adding another format

Google's current AI optimization guide says llms.txt is not required for Google Search and does not positively or negatively affect visibility or rankings there. It also does not require special Markdown files or a fixed content format for AI features. Maintain an extra representation only when a real consumer or workflow justifies it, and keep it consistent with the canonical HTML.

This is separate from Profound's experiment: a measured change in crawler traffic would not by itself demonstrate more citations or customers. State which outcome was measured before applying research to your own publishing system.

A file can load and still contain outdated facts

At the September 29, 2026 production check, the live llms-full.txt still described an enquiry form that only prepared a local JSON brief. The live Terms described a form that submits details to FoundUp. A file can return HTTP 200 and still contain outdated facts. Generate reference files from the same approved page content and check them after deployment.

The local generator was revised to extract the main content from the same rendered pages used for the release. That is an implementation change, not proof of production publication or crawler adoption. Compare the generated form description with the visible Privacy Policy, Terms and actual form behaviour after a release.

The reference files after publication

At 07:42 UTC on September 29, 2026, the live sitemap listed 25 canonical pages, llms.txt indexed those 25 pages, and llms-full.txt contained 25 readable main sections. The RSS feed contained ten articles. All five reference resources returned HTTP 200, and the 25 HTML pages were accessible. The earlier local-only counts were therefore confirmed in that public release.

The dated website check records the transition from missing detail routes to accessible pages. Continue checking manual introductions as well as generated sections: a full-text generator can keep the bodies aligned while a hand-written company summary still describes an older business stage. Publication, content agreement, search indexing and AI citation remain separate observations.