thedaviddias/Front-End-Checklist / skills/llms-txt
llms-txt
VerifiedUse when auditing public documentation portals, API references, help centers, SDK docs, or large knowledge bases. Check the final file served at `/llms.txt` and verify that the linked pages are stable, high-value, and accessible without relying on a JavaScript-only interface.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 2.5 KB |
| references/rule.md | markdown | 6.1 KB |
references/rule.md
View on GitHub# Publish llms.txt for documentation-heavy sites
> Offer an optional llms.txt index that points AI tools to high-value documentation pages and, when useful, a fuller llms-full.txt companion.
**Priority:** medium · **Difficulty:** intermediate · **Time:** 20 min
---
`llms.txt` is an optional plain-text or Markdown file that can give AI tools a cleaner starting point for navigating large documentation sets. It is most useful for public API docs, help centers, SDK references, and knowledge bases where the most important content would otherwise be buried inside navigation trees or app-like UIs.
## Code Examples
```md
# Example API Docs
> Official documentation for Example API, including quickstarts, concepts, and reference material.
## Docs
- [Getting Started](https://docs.example.com/getting-started): Setup, authentication, and your first request.
- [Concepts](https://docs.example.com/concepts): Core mental model, resources, and lifecycle concepts.
- [API Reference](https://docs.example.com/api): Endpoint-by-endpoint request and response details.
- [Examples](https://docs.example.com/examples): Copy-paste examples for common integration patterns.
## Optional
- [Full Reference](https://docs.example.com/llms-full.txt): Expanded documentation index for tools that can handle larger context files.
```
```md
# ❌ Poor example
- [Home](https://example.com/)
- [Blog](https://example.com/blog)
- [Pricing](https://example.com/pricing)
- [Login](https://example.com/login)
- [Random changelog entry](https://example.com/changelog/2024-01-11)
```
## Why It Matters
Documentation portals often bury the most important pages behind dense navigation, search UIs, or framework-specific layouts. A curated `llms.txt` gives AI tools a cleaner entry point without replacing the normal crawl and indexing mechanisms of the web.
## Best Practices
- Publish the file at exactly `/llms.txt` on the production domain.
- Keep it short and curated: link to the most useful guides, concepts, references, and examples first.
- Use absolute URLs and one-sentence descriptions so each entry makes sense on its own.
- Link only to public, stable content that returns HTTP 200 and does not require login.
- If you provide `llms-full.txt`, treat it as an optional companion file and link to it from `llms.txt`.
## Common Mistakes
- Treating `llms.txt` as a replacement for `robots.txt`, XML sitemaps, or structured data.
- Dumping every page from the sitemap instead of curating the highest-signal documentation URLs.
- Listing marketing pages, pricing pages, or login flows instead of task-oriented documentation.
- Linking to content that is only visible after search, tabs, or client-side rendering complete.
- Claiming that `llms.txt` is required for search ranking, AI Overviews, or chat assistant inclusion.
## Implementation Notes
Use `llms.txt` when your site already has substantial documentation and you can maintain a curated index over time. Small brochure sites, landing pages, and simple marketing sites usually do not benefit enough to justify another public file.
`llms-full.txt` is commonly used in the ecosystem as a larger companion file, but it is optional. Only publish it if you can keep it synchronized with the documentation that you actually want AI tools to consume.
```ts
// app/llms.txt/route.ts
const body = `# Example Docs
> Public documentation for Example product.
## Docs
- [Getting Started](https://docs.example.com/getting-started): Setup and first steps.
- [API Reference](https://docs.example.com/api): Request and response details.
## Optional
- [Full Reference](https://docs.example.com/llms-full.txt): Expanded docs index.
`
return new Response(body, {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, max-age=3600'
}
})
}
```
## Tools & Validation
- Fetch `/llms.txt` on the production domain and confirm it returns HTTP 200.
- Spot-check every listed URL and remove links that redirect, 404, or require authentication.
- Compare `llms.txt` against your sitemap and docs IA to make sure it highlights the most useful public docs rather than duplicating everything, and check the overall shape against the [llms.txt convention](https://llmstxt.org/).
- If you publish `llms-full.txt`, verify that it is linked from `llms.txt` and clearly positioned as an expanded companion file.
- Directories such as `llmstxthub.com` can help you compare real-world implementations before you define your own file structure.
- Helper tooling such as `npx -y @thedaviddias/mcp-llms-txt-explorer` can speed up audits when you want to inspect sites that already publish these files.
## Exceptions
- Small marketing sites, portfolios, or single-page sites often do not need `llms.txt`; strong page-level structure is usually enough.
- Private documentation behind authentication may need an internal equivalent rather than a public `llms.txt`.
- If your docs platform already exposes a clean, stable, public documentation index and your team cannot maintain another file, skipping `llms.txt` is reasonable.
## Standards
- Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
- Check the implementation against llms.txt before treating the rule as satisfied.
- Check the implementation against Google Search Central: AI features and your website before treating the rule as satisfied.
## Verification
### Automated Checks
- Fetch [the live `/llms.txt` file](https://llmstxt.org/) and confirm your production version returns HTTP 200 from the exact root path.
- Validate that every URL listed in the file returns HTTP 200 and resolves to a canonical-url public page.
- If `llms-full.txt` is present, fetch `/llms-full.txt` on the production domain and confirm it also returns HTTP 200.
### Manual Checks
- Confirm the file is curated around developer or support tasks, not generic site navigation.
- Confirm that the most important docs pages are understandable without requiring a site search box, hidden tabs, or authentication.
- Confirm that crawl and indexing rules are still managed through `robots.txt`, canonical URLs, sitemaps, and page-level metadata.Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |