Skip to content

sergekostenchuk/seo-llm-skill-cluster / skills/server-log-crawler-analyst

server-log-crawler-analyst

Verified

Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.

references/data-source-tiers.md

View on GitHub
# Monitoring Data Source Tiers

## Tier 0: Public Fetch

Examples:

- `curl -I https://example.com/`
- `robots.txt`
- `llms.txt`
- `sitemap.xml`
- `news-sitemap.xml`
- RSS feed

Can prove:

- endpoint availability;
- HTTP status;
- content type;
- file size;
- presence of public discovery surfaces.

Cannot prove:

- ranking;
- indexing;
- assistant citation;
- real crawler visits.

## Tier 1: Server Logs

Examples:

- nginx/apache access logs;
- CDN request logs;
- reverse proxy logs.

Can prove:

- a request happened;
- path, status, byte count, timestamp, and user agent;
- user-agent string claiming a bot.

Cannot prove without extra verification:

- true bot identity;
- indexing;
- ranking;
- citation.

Privacy:

- raw IP must be redacted, hashed, or kept local;
- query strings and admin paths should be removed from public reports.

## Tier 2: Exported Search/Analytics Summaries

Examples:

- Search Console exported rows;
- analytics dashboard screenshots or CSV summaries;
- server aggregate reports.

Can prove:

- metrics as represented by the export;
- date range, query/page/country/device if present.

Requires:

- timestamp;
- export source;
- privacy review;
- no raw private rows in public skill artifacts.

## Tier 3: Credentialed APIs

Examples:

- Search Console API;
- analytics API;
- paid rank tracking API;
- assistant/search monitoring APIs.

This skill does not implement these. Hand off to future credentialed monitoring skills.

## User-Agent Classification

Classify conservatively:

- `known_search_bot_claim`: user-agent claims Googlebot/Bingbot/Yandex/etc.
- `known_ai_bot_claim`: user-agent claims GPTBot/OAI-SearchBot/ChatGPT-User/ClaudeBot/PerplexityBot/etc.
- `generic_bot`: user-agent contains bot/crawl/spider but is not recognized.
- `browser_like`: normal browser-like UA.
- `suspicious_probe`: paths such as `/wp-admin`, `/.env`, `/cgi-bin`, traversal attempts, or auth probes.

Use "claim" unless bot identity is verified by provider-specific method.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall