aboylabs/ai-agent-skills / skills/research/web-scraping
web-scraping
VerifiedExtract data from websites, including JavaScript-rendered SPAs and dynamic content
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.5 KB |
| references/browser-use-cloud-api.md | markdown | 5.3 KB |
| references/browser-use-cloud.md | markdown | 3.5 KB |
| references/browserbase-api.md | markdown | 7.6 KB |
| references/cloudflare-bypass-strategies.md | markdown | 6.4 KB |
| references/cloudflare-turnstile-bypass.md | markdown | 4.5 KB |
| references/nextjs-ssr-scraping.md | markdown | 2.6 KB |
| references/race-condition-order-matching.md | markdown | 9.2 KB |
| references/security-audit-pattern.md | markdown | 7.1 KB |
| references/security-audit-workflow.md | markdown | 6.8 KB |
| references/spa-api-discovery.md | markdown | 5.5 KB |
| references/tinyfish-js-reverse-engineering.md | markdown | 6.7 KB |
| references/trendshift-analysis.md | markdown | 2.2 KB |
references/cloudflare-bypass-strategies.md
View on GitHub# Cloudflare Bypass Strategies
When a site returns Cloudflare challenge page ("Just a moment...") or 403 errors on all endpoints, standard scraping tools fail. This document covers escalation strategies.
## Detection
**Cloudflare challenge page:**
```html
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
<meta http-equiv="refresh" content="360">
```
**403 on all endpoints:**
```bash
curl -s https://example.com/api/listings -o /dev/null -w "HTTP %{http_code}\n"
# Output: HTTP 403
```
**Bot detection error from cloud services:**
```json
{"results":[],"errors":[{"url":"https://example.com/","error":"bot_blocked"}]}
```
## Escalation Ladder
### Level 1: Headers + User-Agent (Success Rate: ~20%)
```bash
curl -s -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "Connection: keep-alive" \
"https://example.com/api/listings"
```
**When it works:** Basic bot detection that only checks User-Agent.
**When it fails:** Cloudflare Turnstile, fingerprinting, or IP reputation checks.
### Level 2: TinyFish Fetch API (Success Rate: ~40%)
```bash
curl -X POST https://api.fetch.tinyfish.ai \
-H "X-API-Key: $TINYFISH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/"]}'
```
**When it works:** Sites with moderate Cloudflare protection.
**When it fails:** Advanced bot detection, Turnstile challenges, or IP-based blocks.
**rpow2swap.com result (May 2026):**
```json
{"results":[],"errors":[{"url":"https://rpow2swap.com/","error":"bot_blocked"}]}
```
### Level 3: Browserbase Cloud Browser (Success Rate: ~70%)
Requires API key from https://browserbase.com/
```python
import requests
# Create session
response = requests.post(
"https://www.browserbase.com/v1/sessions",
headers={"X-BB-API-Key": API_KEY, "Content-Type": "application/json"},
json={"projectId": PROJECT_ID}
)
session = response.json()
session_id = session['id']
# Navigate to page
requests.post(
f"https://www.browserbase.com/v1/sessions/{session_id}/navigate",
headers={"X-BB-API-Key": API_KEY, "Content-Type": "application/json"},
json={"url": "https://example.com"}
)
# Get page content
content = requests.get(
f"https://www.browserbase.com/v1/sessions/{session_id}/content",
headers={"X-BB-API-Key": API_KEY}
).text
```
**When it works:** Most Cloudflare-protected sites (uses real Chrome with residential proxies).
**When it fails:** Sites with advanced fingerprinting or manual review.
**Cost:** Paid service, ~$0.01-0.05 per session.
### Level 4: Stealth Browser Libraries (Success Rate: ~60%)
**Puppeteer with stealth plugin:**
```javascript
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox', '--disable-setuid-sandbox']
});
const page = await browser.newPage();
await page.goto('https://example.com', {waitUntil: 'networkidle2'});
const content = await page.content();
```
**When it works:** Sites that check for headless browser markers.
**When it fails:** Cloudflare Turnstile (requires human interaction).
**Note:** May fail on VPS with sandbox errors — use `--no-sandbox` flag.
### Level 5: Manual Browser + DevTools (Success Rate: 100%)
When all automation fails, use real browser:
1. Open site in Chrome/Firefox
2. Open DevTools → Network tab
3. Filter by `Fetch/XHR`
4. Interact with page (scroll, click, navigate)
5. Find API endpoint in Network tab
6. Right-click → Copy as cURL
7. Extract auth headers/cookies
8. Replicate in script
**When to use:** High-value targets, one-time scraping, or when automation is blocked.
**Limitation:** Not scalable, requires manual intervention.
## rpow2swap.com Case Study (May 2026)
**Protection level:** Cloudflare with bot detection
**Attempts:**
1. ❌ curl with User-Agent → 403
2. ❌ TinyFish Fetch API → `bot_blocked`
3. ❌ Hermes browser tool → timeout (stuck on challenge)
4. ⏸️ Browserbase → not configured (API key missing)
5. ✅ Manual browser inspection → **recommended approach**
**Outcome:** Site requires manual browser access for initial API discovery. Once endpoint found, may be accessible via curl with proper headers/cookies.
**Next steps for automation:**
1. Use manual browser to find API endpoint
2. Extract auth headers/cookies from DevTools
3. Test endpoint with curl + headers
4. If works → automate with curl
5. If fails → use Browserbase for automation
## Decision Tree
```
Site returns Cloudflare challenge?
├─ Yes → Try TinyFish Fetch
│ ├─ Works → Use TinyFish
│ └─ Fails → Try Browserbase (if configured)
│ ├─ Works → Use Browserbase
│ └─ Fails → Manual browser + DevTools
└─ No → Use curl with User-Agent
```
## Cost-Benefit Analysis
| Method | Cost | Speed | Success Rate | Scalability |
|--------|------|-------|--------------|-------------|
| curl + headers | Free | Fast | 20% | High |
| TinyFish | Free | Medium | 40% | High |
| Browserbase | $0.01-0.05/req | Slow | 70% | Medium |
| Stealth browser | Free | Slow | 60% | Medium |
| Manual browser | Free | Very slow | 100% | None |
## Pitfalls
- **Don't retry failed methods** — if curl fails, TinyFish likely fails too (same IP reputation)
- **Cloudflare learns** — repeated automation attempts may trigger stricter protection
- **Session cookies expire** — manual browser approach requires periodic re-authentication
- **Rate limiting** — even successful bypass may trigger rate limits on API endpoints
- **Legal considerations** — bypassing protection may violate ToS (check robots.txt and terms)
## When to Give Up
If all methods fail and manual browser is not feasible:
1. Check if site has official API
2. Contact site owner for API access
3. Use alternative data sources
4. Accept that automation is not possible
**Red flags:**
- Site requires CAPTCHA on every request
- IP gets banned after 1-2 requests
- Site uses device fingerprinting + IP reputation
- Manual browser access also gets blocked
## References
- TinyFish Docs: https://docs.tinyfish.ai/
- Browserbase Docs: https://docs.browserbase.com/
- Puppeteer Stealth: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |