Skip to content

moonlight-lupin/agent-skills / web-scraping/website-scraping

website-scraping

Verified

Generic playbook for extracting structured data from any website — hotel prices, flight fares, e-commerce listings, real estate, jobs, competitor product catalogues, anything where the goal is to turn one or more URLs into clean records on disk. Use this skill whenever the user mentions scraping, harvesting, extracting, or pulling data from a website; names a specific site or competitor they want data from; asks how to handle JavaScript-rendered pages, Cloudflare blocks, bot detection, Turnstile challenges, or Playwright; wants to monitor prices over time; needs to automate a copy-paste research task; or says things like "just get the data from X" or "I need a script that grabs N from site Y". Covers recon, picking the lightest extraction tool that works, surviving anti-bot defences, and writing clean JSONL output with a run manifest. Scope is scraping only — landing data in a database, scheduling, and downstream pipeline work are explicitly out of scope and live in other tools.

examples/agentic-browsing-quotes-demo/plan.md

View on GitHub
# Task

On http://quotes.toscrape.com/search.aspx, use the two dependent dropdowns to
filter quotes: select author "Albert Einstein", then select one of his tags
(populated by AJAX after the author is chosen), submit the search, and extract
the resulting quote text(s).

# Critical Points

- [ ] CP1: Author dropdown set to "Albert Einstein" (visible selected value)
- [ ] CP2: Tag dropdown repopulated via AJAX for that author, and a specific tag selected
- [ ] CP3: Search submitted and result quotes displayed for (author, tag)
- [ ] CP4: Final datum — the extracted quote text(s) — printed to the log
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall