thedaviddias/Front-End-Checklist / skills/robots-txt
robots-txt
VerifiedUse when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 1.9 KB |
| references/rule.md | markdown | 3.0 KB |
SKILL.md
View on GitHub---
name: robots-txt
description: "Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps."
metadata:
category: seo
priority: high
difficulty: beginner
estimatedTime: "10"
source: frontendchecklist.io
url: https://frontendchecklist.io/rules/seo/robots-txt
---
# Publish a robots.txt file
robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.
## Quick Reference
- Serve a valid `robots.txt` at `/robots.txt` on the production domain, returning HTTP 200
- Include a `Sitemap:` directive pointing to your XML sitemap
- Never disallow crawling of CSS/JS assets that render your pages
- Avoid blocking all crawlers with `Disallow: /` on a live site
## Check
Fetch `/robots.txt` on the live domain and verify it returns HTTP 200, uses correct `User-agent` / `Disallow` / `Allow` syntax, and includes a `Sitemap:` directive pointing to the XML sitemap. Check for accidental `Disallow: /` directives.
## Fix
Create or update `robots.txt` at the web root with valid directives. Add a live `Sitemap:` line for the production sitemap URL. Remove any `Disallow: /` rules that block the whole site or resources needed for rendering.
## Explain
Explain how robots.txt controls crawler access, why an accidental `Disallow: /` can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.
## Code Review
Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.
---
For full implementation details, code examples, and framework-specific guidance,
see `references/rule.md`.
Rule page: https://frontendchecklist.io/rules/seo/robots-txt
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |