sergekostenchuk/seo-llm-skill-cluster / skills/server-log-crawler-analyst
server-log-crawler-analyst
VerifiedAnalyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.1 KB |
| agents/openai.yaml | config | 388 B |
| assets/crawler-monitor-report.template.md | markdown | 732 B |
| evals.json | config | 1.8 KB |
| fixtures/sample-access.log | other | 575 B |
| references/data-source-tiers.md | markdown | 2.0 KB |
| references/monitoring-backlog.md | markdown | 1.4 KB |
| scripts/analyze_access_log.py | script | 3.8 KB |
| validation-report.md | markdown | 1.2 KB |
SKILL.md
View on GitHub---
name: server-log-crawler-analyst
description: Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.
---
# Server Log Crawler Analyst
Use this skill when credential-free monitoring evidence is enough, or when credentialed
Search Console, rank tracking, analytics, or assistant citation tools are unavailable.
Read [references/data-source-tiers.md](references/data-source-tiers.md) before producing a monitoring report.
## Owns
- credential-free monitoring baseline;
- server access-log summary with IP privacy protection;
- crawler/user-agent observation reports;
- public HTTP checks for robots, llms.txt, sitemaps, RSS, and representative URLs;
- data-source tiering;
- monitoring limitations and unknowns;
- backlog handoff for Search Console, rank/SERP, analytics, and LLM citation monitoring.
## Does Not Own
- credentialed Search Console or analytics API work;
- scraping search engines;
- bypassing bot protections;
- live firewall or WAF changes;
- claims about rankings or assistant citations without direct evidence;
- storing raw IP logs in public artifacts.
## Workflow
1. Identify available evidence: public URL fetches, access-log snippet, exported summary, or synthetic fixture.
2. Label the evidence tier and collection time.
3. If logs are provided, summarize requests by path, status, user-agent class, and redacted IP hash.
4. Separate known crawler hits from unknown bot-like traffic.
5. Check public discovery endpoints when requested.
6. Record limitations: no Search Console, no rank data, no assistant citation data unless explicitly provided.
7. Produce a monitoring report using [assets/crawler-monitor-report.template.md](assets/crawler-monitor-report.template.md).
8. Hand off credentialed or citation tasks to the backlog.
## Non-Negotiables
- Do not publish raw IP addresses.
- Do not store credentials, cookies, API keys, or private analytics exports in skill files.
- Do not infer ranking, indexing, or citation success from server logs alone.
- Do not claim a user-agent is verified Google/OpenAI/Claude/Perplexity unless verification method is documented.
- Do not scrape SERPs or assistants against terms of service.
- Do not change firewall, WAF, robots, or server config from this skill.
## Safety And Privacy Boundaries
- Treat raw logs as private by default.
- Use aggregated counts and redacted identifiers.
- Keep exact timestamps only when needed for evidence; otherwise aggregate by hour/day.
- If the user asks to publish a report, remove raw IP, query strings, session IDs, auth paths, and admin URLs unless explicitly needed and safe.
- Public reports should say "observed in provided logs" rather than "definitely crawled by" unless bot verification is complete.
## Evidence Labels
- `Observed`: directly in log, HTTP response, or exported report.
- `Verified`: observed and checked by an accepted verification method.
- `Inferred`: likely based on user-agent/path/status, but not verified.
- `Open`: no evidence available in this run.
## Validation
Validate skill edits with:
```bash
python3 <codex-skills-dir>/senior-skill-architect/scripts/lint_production_skill.py ./skills/server-log-crawler-analyst
python3 ./skills/server-log-crawler-analyst/scripts/analyze_access_log.py ./skills/server-log-crawler-analyst/fixtures/sample-access.log --report <tmp-dir>/server-log-crawler-report.json
python3 ./plans/seo-llm-skill-cluster/scripts/lint_skill_cluster.py .
```
Forward tests live in [evals.json](evals.json).
## Output Shape
Return:
1. Evidence source and timestamp.
2. Data tier and privacy handling.
3. Observed crawler/user-agent classes.
4. Public endpoint status checks if performed.
5. Suspicious or irrelevant bot traffic summary.
6. Unknowns and claims refused.
7. Monitoring backlog and next evidence needed.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |