archibate/dotfiles-opencode / skills/scrapling
scrapling
UnverifiedScrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. This skill should be used when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; write spiders.
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 1.5 KB |
| SKILL.md | skill md | 19.3 KB |
| examples/.DS_Store | binary | 6.0 KB |
| examples/01_fetcher_session.py | script | 855 B |
| examples/02_dynamic_session.py | script | 951 B |
| examples/03_stealthy_session.py | script | 952 B |
| examples/04_spider.py | script | 1.9 KB |
| examples/README.md | markdown | 1.7 KB |
| references/fetching/choosing.md | markdown | 6.3 KB |
| references/fetching/dynamic.md | markdown | 20.4 KB |
| references/fetching/static.md | markdown | 17.2 KB |
| references/fetching/stealthy.md | markdown | 20.1 KB |
| references/mcp-server.md | markdown | 10.6 KB |
| references/migrating_from_beautifulsoup.md | markdown | 11.0 KB |
| references/parsing/adaptive.md | markdown | 10.5 KB |
| references/parsing/main_classes.md | markdown | 26.7 KB |
| references/parsing/selection.md | markdown | 22.1 KB |
| references/spiders/advanced.md | markdown | 10.5 KB |
| references/spiders/architecture.md | markdown | 6.7 KB |
| references/spiders/getting-started.md | markdown | 4.6 KB |
| references/spiders/proxy-blocking.md | markdown | 8.7 KB |
| references/spiders/requests-responses.md | markdown | 8.9 KB |
| references/spiders/sessions.md | markdown | 8.5 KB |
examples/04_spider.py
View on GitHub"""
Example 4: Python - Spider (auto-crawling framework)
Scrapes ALL pages of quotes.toscrape.com by following "Next" pagination links
automatically. No manual page looping needed.
The spider yields structured items (text + author + tags) and exports them to JSON.
Best for: multi-page crawls, full-site scraping, anything needing pagination or
link following across many pages.
Outputs:
- Live stats to terminal during crawl
- Final crawl stats at the end
- quotes.json in the current directory
"""
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
concurrent_requests = 5 # Fetch up to 5 pages at once
async def parse(self, response: Response):
# Extract all quotes on the current page
for quote in response.css(".quote"):
yield {
"text": quote.css(".text::text").get(),
"author": quote.css(".author::text").get(),
"tags": quote.css(".tags .tag::text").getall(),
}
# Follow the "Next" button to the next page (if it exists)
next_page = response.css(".next a")
if next_page:
yield response.follow(next_page[0].attrib["href"])
if __name__ == "__main__":
result = QuotesSpider().start()
print(f"\n{'=' * 50}")
print(f"Scraped : {result.stats.items_scraped} quotes")
print(f"Requests: {result.stats.requests_count}")
print(f"Time : {result.stats.elapsed_seconds:.2f}s")
print(f"Speed : {result.stats.requests_per_second:.2f} req/s")
print(f"{'=' * 50}\n")
for i, item in enumerate(result.items, 1):
print(f"{i:>3}. [{item['author']}] {item['text']}")
if item["tags"]:
print(f" Tags: {', '.join(item['tags'])}")
# Export to JSON
result.items.to_json("quotes.json", indent=True)
print("\nExported to quotes.json")
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |