sergekostenchuk/seo-llm-skill-cluster / skills/server-log-crawler-analyst
server-log-crawler-analyst
VerifiedAnalyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.1 KB |
| agents/openai.yaml | config | 388 B |
| assets/crawler-monitor-report.template.md | markdown | 732 B |
| evals.json | config | 1.8 KB |
| fixtures/sample-access.log | other | 575 B |
| references/data-source-tiers.md | markdown | 2.0 KB |
| references/monitoring-backlog.md | markdown | 1.4 KB |
| scripts/analyze_access_log.py | script | 3.8 KB |
| validation-report.md | markdown | 1.2 KB |
references/data-source-tiers.md
View on GitHub# Monitoring Data Source Tiers
## Tier 0: Public Fetch
Examples:
- `curl -I https://example.com/`
- `robots.txt`
- `llms.txt`
- `sitemap.xml`
- `news-sitemap.xml`
- RSS feed
Can prove:
- endpoint availability;
- HTTP status;
- content type;
- file size;
- presence of public discovery surfaces.
Cannot prove:
- ranking;
- indexing;
- assistant citation;
- real crawler visits.
## Tier 1: Server Logs
Examples:
- nginx/apache access logs;
- CDN request logs;
- reverse proxy logs.
Can prove:
- a request happened;
- path, status, byte count, timestamp, and user agent;
- user-agent string claiming a bot.
Cannot prove without extra verification:
- true bot identity;
- indexing;
- ranking;
- citation.
Privacy:
- raw IP must be redacted, hashed, or kept local;
- query strings and admin paths should be removed from public reports.
## Tier 2: Exported Search/Analytics Summaries
Examples:
- Search Console exported rows;
- analytics dashboard screenshots or CSV summaries;
- server aggregate reports.
Can prove:
- metrics as represented by the export;
- date range, query/page/country/device if present.
Requires:
- timestamp;
- export source;
- privacy review;
- no raw private rows in public skill artifacts.
## Tier 3: Credentialed APIs
Examples:
- Search Console API;
- analytics API;
- paid rank tracking API;
- assistant/search monitoring APIs.
This skill does not implement these. Hand off to future credentialed monitoring skills.
## User-Agent Classification
Classify conservatively:
- `known_search_bot_claim`: user-agent claims Googlebot/Bingbot/Yandex/etc.
- `known_ai_bot_claim`: user-agent claims GPTBot/OAI-SearchBot/ChatGPT-User/ClaudeBot/PerplexityBot/etc.
- `generic_bot`: user-agent contains bot/crawl/spider but is not recognized.
- `browser_like`: normal browser-like UA.
- `suspicious_probe`: paths such as `/wp-admin`, `/.env`, `/cgi-bin`, traversal attempts, or auth probes.
Use "claim" unless bot identity is verified by provider-specific method.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |