moonlight-lupin/agent-skills / web-scraping/website-scraping
website-scraping
VerifiedGeneric playbook for extracting structured data from any website — hotel prices, flight fares, e-commerce listings, real estate, jobs, competitor product catalogues, anything where the goal is to turn one or more URLs into clean records on disk. Use this skill whenever the user mentions scraping, harvesting, extracting, or pulling data from a website; names a specific site or competitor they want data from; asks how to handle JavaScript-rendered pages, Cloudflare blocks, bot detection, Turnstile challenges, or Playwright; wants to monitor prices over time; needs to automate a copy-paste research task; or says things like "just get the data from X" or "I need a script that grabs N from site Y". Covers recon, picking the lightest extraction tool that works, surviving anti-bot defences, and writing clean JSONL output with a run manifest. Scope is scraping only — landing data in a database, scheduling, and downstream pipeline work are explicitly out of scope and live in other tools.
evals/routing-fixtures.json
View on GitHub{
"skill_name": "website-scraping",
"purpose": "Lightweight routing + output-contract fixtures for website-scraping. Specs, not executed live in CI; validated by tests/test_routing_fixtures.py. Complements evals/evals.json (which holds runnable end-to-end scraping tasks) by capturing routing boundaries and the required output contract. ",
"fixtures": [
{
"id": "ws-01-structured-extract",
"request": "Pull every product on the first page of example-shop.com — name, price, and product URL — into a file I can load later.",
"expected_routing": {
"skill": "website-scraping",
"reason": "Turning one or more URLs into clean structured records is the core website-scraping job.",
"not": ["people-enrichment", "entity-research"]
},
"required_output_fields": ["JSONL records (one per line)", "per-record fields", "run manifest"],
"forbidden_patterns": ["(?i)as an ai language model", "(?i)i cannot (access|browse)"]
},
{
"id": "ws-02-js-rendered",
"request": "Get all the quotes, authors, and tags from a JavaScript-rendered listing page and save them as records.",
"expected_routing": {
"skill": "website-scraping",
"reason": "Client-rendered extraction (escalate to in-page JSON or Playwright) is still website-scraping.",
"not": []
},
"required_output_fields": ["JSONL records (one per line)", "run manifest", "extraction strategy noted"],
"forbidden_patterns": ["(?i)as an ai language model"]
},
{
"id": "ws-03-boundary-entity",
"request": "Research the background and any adverse media on Acme Corp before we sign a contract.",
"expected_routing": {
"skill": "entity-research",
"reason": "Background/vetting of a named company is entity-research, not generic site scraping.",
"not": ["website-scraping"]
},
"required_output_fields": ["entity identity", "adverse media", "sources"],
"forbidden_patterns": ["(?i)\\bcleared\\b", "(?i)compliance determination"]
},
{
"id": "ws-04-boundary-enrichment",
"request": "I have a CSV of 200 people — add each person's current employer and LinkedIn URL.",
"expected_routing": {
"skill": "people-enrichment",
"reason": "Bulk person/company enrichment via a licensed data provider is people-enrichment, not DIY scraping (and LinkedIn scraping is out of scope).",
"not": ["website-scraping"]
},
"required_output_fields": ["per-person enriched fields", "styled .xlsx output"],
"forbidden_patterns": ["(?i)scrape linkedin", "(?i)as an ai language model"]
}
]
}
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |