Skip to content

thedaviddias/Front-End-Checklist / skills/robots-txt

robots-txt

Verified

Use when applies to any public website. Use when auditing technical SEO foundations or diagnosing crawl coverage gaps.

Publish a robots.txt file

robots.txt is the first file crawlers fetch; misconfigured directives can silently block search engines from crawling your entire site, killing organic visibility.

Quick Reference

  • Serve a valid robots.txt at /robots.txt on the production domain, returning HTTP 200
  • Include a Sitemap: directive pointing to your XML sitemap
  • Never disallow crawling of CSS/JS assets that render your pages
  • Avoid blocking all crawlers with Disallow: / on a live site

Check

Fetch /robots.txt on the live domain and verify it returns HTTP 200, uses correct User-agent / Disallow / Allow syntax, and includes a Sitemap: directive pointing to the XML sitemap. Check for accidental Disallow: / directives.

Fix

Create or update robots.txt at the web root with valid directives. Add a live Sitemap: line for the production sitemap URL. Remove any Disallow: / rules that block the whole site or resources needed for rendering.

Explain

Explain how robots.txt controls crawler access, why an accidental Disallow: / can delist a site, and why CSS/JS must remain accessible for rendering-based indexing.

Code Review

Review metadata generation, rendered HTML, structured data, and response headers related to Publish a robots.txt file. Flag exact routes or templates where search-facing output violates the rule, and describe how to verify the final page output.


For full implementation details, code examples, and framework-specific guidance, see references/rule.md.

Rule page: https://frontendchecklist.io/rules/seo/robots-txt

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall