archibate/dotfiles-opencode / skills/scrapling
scrapling
UnverifiedScrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. This skill should be used when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; write spiders.
| File | Kind | Size |
|---|---|---|
| LICENSE.txt | markdown | 1.5 KB |
| SKILL.md | skill md | 19.3 KB |
| examples/.DS_Store | binary | 6.0 KB |
| examples/01_fetcher_session.py | script | 855 B |
| examples/02_dynamic_session.py | script | 951 B |
| examples/03_stealthy_session.py | script | 952 B |
| examples/04_spider.py | script | 1.9 KB |
| examples/README.md | markdown | 1.7 KB |
| references/fetching/choosing.md | markdown | 6.3 KB |
| references/fetching/dynamic.md | markdown | 20.4 KB |
| references/fetching/static.md | markdown | 17.2 KB |
| references/fetching/stealthy.md | markdown | 20.1 KB |
| references/mcp-server.md | markdown | 10.6 KB |
| references/migrating_from_beautifulsoup.md | markdown | 11.0 KB |
| references/parsing/adaptive.md | markdown | 10.5 KB |
| references/parsing/main_classes.md | markdown | 26.7 KB |
| references/parsing/selection.md | markdown | 22.1 KB |
| references/spiders/advanced.md | markdown | 10.5 KB |
| references/spiders/architecture.md | markdown | 6.7 KB |
| references/spiders/getting-started.md | markdown | 4.6 KB |
| references/spiders/proxy-blocking.md | markdown | 8.7 KB |
| references/spiders/requests-responses.md | markdown | 8.9 KB |
| references/spiders/sessions.md | markdown | 8.5 KB |
references/mcp-server.md
View on GitHub# Scrapling MCP Server
The Scrapling MCP server exposes six web scraping tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results) and three levels of scraping capability: plain HTTP, browser-rendered, and stealth (anti-bot bypass).
All tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str).
## Tools
### `get` -- HTTP request (single URL)
Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable for static pages with no/low bot protection.
**Key parameters:**
| Parameter | Type | Default | Description |
|---------------------|------------------------------------|--------------|--------------------------------------------------------------------|
| `url` | str | required | URL to fetch |
| `extraction_type` | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format |
| `css_selector` | str or null | null | CSS selector to narrow content (applied after `main_content_only`) |
| `main_content_only` | bool | true | Restrict to `<body>` content |
| `impersonate` | str | `"chrome"` | Browser fingerprint to impersonate |
| `proxy` | str or null | null | Proxy URL, e.g. `"http://user:pass@host:port"` |
| `proxy_auth` | dict or null | null | `{"username": "...", "password": "..."}` |
| `auth` | dict or null | null | HTTP basic auth, same format as proxy_auth |
| `timeout` | number | 30 | Seconds before timeout |
| `retries` | int | 3 | Retry attempts on failure |
| `retry_delay` | int | 1 | Seconds between retries |
| `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer |
| `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |
| `follow_redirects` | bool | true | Follow HTTP redirects |
| `max_redirects` | int | 30 | Max redirects (-1 for unlimited) |
| `headers` | dict or null | null | Custom request headers |
| `cookies` | dict or null | null | Request cookies |
| `params` | dict or null | null | Query string parameters |
| `verify` | bool | true | Verify HTTPS certificates |
### `bulk_get` -- HTTP request (multiple URLs)
Async concurrent version of `get`. Same parameters except `url` is replaced by `urls` (list of strings). All URLs are fetched in parallel. Returns a list of `ResponseModel`.
### `fetch` -- Browser fetch (single URL)
Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynamic/SPA pages with no/low bot protection.
**Key parameters (beyond shared ones):**
| Parameter | Type | Default | Description |
|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|
| `url` | str | required | URL to fetch |
| `extraction_type` | str | `"markdown"` | `"markdown"` / `"html"` / `"text"` |
| `css_selector` | str or null | null | Narrow content before extraction |
| `main_content_only` | bool | true | Restrict to `<body>` |
| `headless` | bool | true | Run browser hidden (true) or visible (false) |
| `proxy` | str or dict or null | null | String URL or `{"server": "...", "username": "...", "password": "..."}` |
| `timeout` | number | 30000 | Timeout in **milliseconds** |
| `wait` | number | 0 | Extra wait (ms) after page load before extraction |
| `wait_selector` | str or null | null | CSS selector to wait for before extraction |
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
| `network_idle` | bool | false | Wait until no network activity for 500ms |
| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
| `google_search` | bool | true | Set a Google referer header |
| `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium |
| `cdp_url` | str or null | null | Connect to existing browser via CDP URL |
| `extra_headers` | dict or null | null | Additional request headers |
| `useragent` | str or null | null | Custom user-agent (auto-generated if null) |
| `cookies` | list or null | null | Playwright-format cookies |
| `timezone_id` | str or null | null | Browser timezone, e.g. `"America/New_York"` |
| `locale` | str or null | null | Browser locale, e.g. `"en-GB"` |
### `bulk_fetch` -- Browser fetch (multiple URLs)
Concurrent browser version of `fetch`. Same parameters except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
### `stealthy_fetch` -- Stealth browser fetch (single URL)
Anti-bot bypass fetcher with fingerprint spoofing. Use this for sites with Cloudflare Turnstile/Interstitial or other strong protections.
**Additional parameters (beyond those in `fetch`):**
| Parameter | Type | Default | Description |
|--------------------|--------------|---------|------------------------------------------------------------------|
| `solve_cloudflare` | bool | false | Automatically solve Cloudflare Turnstile/Interstitial challenges |
| `hide_canvas` | bool | false | Add noise to canvas operations to prevent fingerprinting |
| `block_webrtc` | bool | false | Force WebRTC to respect proxy settings (prevents IP leak) |
| `allow_webgl` | bool | true | Keep WebGL enabled (disabling is detectable by WAFs) |
| `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |
All parameters from `fetch` are also accepted.
### `bulk_stealthy_fetch` -- Stealth browser fetch (multiple URLs)
Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
## Tool selection guide
| Scenario | Tool |
|------------------------------------------|---------------------------------------------------------------|
| Static page, no bot protection | `get` |
| Multiple static pages | `bulk_get` |
| JavaScript-rendered / SPA page | `fetch` |
| Multiple JS-rendered pages | `bulk_fetch` |
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
| Multiple protected pages | `bulk_stealthy_fetch` |
Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked.
## Content extraction tips
- Use `css_selector` to narrow results before they reach the model -- this saves significant tokens.
- `main_content_only=true` (default) strips nav/footer by restricting to `<body>`.
- `extraction_type="markdown"` (default) is best for readability. Use `"text"` for minimal output, `"html"` when structure matters.
- If a `css_selector` matches multiple elements, all are returned in the `content` list.
## Setup
Start the server (stdio transport, used by most MCP clients):
```bash
scrapling mcp
```
Or with Streamable HTTP transport:
```bash
scrapling mcp --http
scrapling mcp --http --host 127.0.0.1 --port 8000
```
Docker alternative:
```bash
docker pull pyd4vinci/scrapling
docker run -i --rm scrapling mcp
```
The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling` binary and the argument is `mcp`.Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 468k | Updated | Claude CodeCodex | Unverified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |