archibate/dotfiles-opencode / skills/scrapling
scrapling
UnverifiedScrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. This skill should be used when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; write spiders.
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 1.5 KB |
| SKILL.md | skill md | 19.3 KB |
| examples/.DS_Store | binary | 6.0 KB |
| examples/01_fetcher_session.py | script | 855 B |
| examples/02_dynamic_session.py | script | 951 B |
| examples/03_stealthy_session.py | script | 952 B |
| examples/04_spider.py | script | 1.9 KB |
| examples/README.md | markdown | 1.7 KB |
| references/fetching/choosing.md | markdown | 6.3 KB |
| references/fetching/dynamic.md | markdown | 20.4 KB |
| references/fetching/static.md | markdown | 17.2 KB |
| references/fetching/stealthy.md | markdown | 20.1 KB |
| references/mcp-server.md | markdown | 10.6 KB |
| references/migrating_from_beautifulsoup.md | markdown | 11.0 KB |
| references/parsing/adaptive.md | markdown | 10.5 KB |
| references/parsing/main_classes.md | markdown | 26.7 KB |
| references/parsing/selection.md | markdown | 22.1 KB |
| references/spiders/advanced.md | markdown | 10.5 KB |
| references/spiders/architecture.md | markdown | 6.7 KB |
| references/spiders/getting-started.md | markdown | 4.6 KB |
| references/spiders/proxy-blocking.md | markdown | 8.7 KB |
| references/spiders/requests-responses.md | markdown | 8.9 KB |
| references/spiders/sessions.md | markdown | 8.5 KB |
references/spiders/getting-started.md
View on GitHub# Getting started
## Your First Spider
A spider is a class that defines how to crawl and extract data from websites. Here's the simplest possible spider:
```python
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
async def parse(self, response: Response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(""),
"author": quote.css("small.author::text").get(""),
}
```
Every spider needs three things:
1. **`name`** — A unique identifier for the spider.
2. **`start_urls`** — A list of URLs to start crawling from.
3. **`parse()`** — An async generator method that processes each response and yields results.
The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](parsing/main_classes.md#selector)/[Response](fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
## Running the Spider
To run your spider, create an instance and call `start()`:
```python
result = QuotesSpider().start()
```
The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
Those stats are in the returned `CrawlResult` object, which gives you everything you need:
```python
result = QuotesSpider().start()
# Access scraped items
for item in result.items:
print(item["text"], "-", item["author"])
# Check statistics
print(f"Scraped {result.stats.items_scraped} items")
print(f"Made {result.stats.requests_count} requests")
print(f"Took {result.stats.elapsed_seconds:.1f} seconds")
# Did the crawl finish or was it paused?
print(f"Completed: {result.completed}")
```
## Following Links
Most crawls need to follow links across multiple pages. Use `response.follow()` to create follow-up requests:
```python
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
async def parse(self, response: Response):
# Extract items from the current page
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(""),
"author": quote.css("small.author::text").get(""),
}
# Follow the "next page" link
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)
```
`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default.
You can point follow-up requests at different callback methods for different page types:
```python
async def parse(self, response: Response):
for link in response.css("a.product-link::attr(href)").getall():
yield response.follow(link, callback=self.parse_product)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
```
**Note:** All callback methods must be async generators (using `async def` and `yield`).
## Exporting Data
The `ItemList` returned in `result.items` has built-in export methods:
```python
result = QuotesSpider().start()
# Export as JSON
result.items.to_json("quotes.json")
# Export as JSON with pretty-printing
result.items.to_json("quotes.json", indent=True)
# Export as JSON Lines (one JSON object per line)
result.items.to_jsonl("quotes.jsonl")
```
Both methods create parent directories automatically if they don't exist.
## Filtering Domains
Use `allowed_domains` to restrict the spider to specific domains. This prevents it from accidentally following links to external websites:
```python
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
allowed_domains = {"example.com"}
async def parse(self, response: Response):
for link in response.css("a::attr(href)").getall():
# Links to other domains are silently dropped
yield response.follow(link, callback=self.parse)
```
Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 467k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |