aboylabs/ai-agent-skills / skills/research/web-scraping
web-scraping
VerifiedExtract data from websites, including JavaScript-rendered SPAs and dynamic content
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.5 KB |
| references/browser-use-cloud-api.md | markdown | 5.3 KB |
| references/browser-use-cloud.md | markdown | 3.5 KB |
| references/browserbase-api.md | markdown | 7.6 KB |
| references/cloudflare-bypass-strategies.md | markdown | 6.4 KB |
| references/cloudflare-turnstile-bypass.md | markdown | 4.5 KB |
| references/nextjs-ssr-scraping.md | markdown | 2.6 KB |
| references/race-condition-order-matching.md | markdown | 9.2 KB |
| references/security-audit-pattern.md | markdown | 7.1 KB |
| references/security-audit-workflow.md | markdown | 6.8 KB |
| references/spa-api-discovery.md | markdown | 5.5 KB |
| references/tinyfish-js-reverse-engineering.md | markdown | 6.7 KB |
| references/trendshift-analysis.md | markdown | 2.2 KB |
references/cloudflare-turnstile-bypass.md
View on GitHub# Cloudflare Turnstile Bypass Strategies
Session: rpow2swap.com analysis (May 2026)
## Problem
Site protected by Cloudflare Turnstile — advanced bot detection that blocks:
- curl (403 Forbidden)
- TinyFish fetch API (`bot_blocked`)
- Standard Puppeteer headless (`HeadlessChrome` detected)
- Puppeteer + stealth plugin (still stuck on "Just a moment..." challenge)
## Escalation Ladder
### Level 1: Direct API Access (FAILED)
```bash
# All endpoints return 403
curl -X POST https://rpow2swap.com/api/payout \
-H "Content-Type: application/json" \
-d '{"recipient":"user@example.com","amount":100000}'
# → 403 Forbidden (Cloudflare challenge page)
```
**Result:** Cloudflare blocks all programmatic access.
### Level 2: TinyFish Fetch (FAILED)
```bash
curl -X POST https://api.fetch.tinyfish.ai \
-H "X-API-Key: $TINYFISH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://rpow2swap.com/"]}'
# → {"results":[],"errors":[{"url":"...","error":"bot_blocked"}]}
```
**Result:** TinyFish cloud browser also detected and blocked.
### Level 3: Puppeteer Headless (FAILED)
```javascript
const puppeteer = require('puppeteer');
const browser = await puppeteer.launch({ headless: true });
// → Stuck on "Just a moment..." Cloudflare challenge
```
**Detection signals:**
- `HeadlessChrome` in User-Agent
- Missing browser fingerprints (WebGL, Canvas, Audio)
- Automation flags (`navigator.webdriver = true`)
### Level 4: Puppeteer + Stealth Plugin (FAILED)
```javascript
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
// → Still stuck on challenge (Turnstile too advanced)
```
**Result:** Cloudflare Turnstile v2 detects even stealth browsers via:
- Mouse movement patterns
- Timing analysis
- Browser API inconsistencies
### Level 5: Browserbase Cloud (NOT TESTED)
```python
import requests
response = requests.post(
"https://www.browserbase.com/v1/sessions",
headers={"X-BB-API-Key": API_KEY},
json={"projectId": PROJECT_ID}
)
session_id = response.json()['id']
# Connect via CDP and navigate
```
**Status:** Not tested (requires API key configuration).
**Expected:** May work (residential IPs + real browser fingerprints).
### Level 6: Manual Browser Inspection (RECOMMENDED)
**When all automation fails, guide user to extract API manually:**
1. User opens site in real browser (Chrome/Firefox)
2. Wait for Cloudflare challenge to complete
3. Open DevTools → Network tab
4. Interact with site (trigger API calls)
5. Copy API endpoint + headers + payload
6. Replicate with curl using captured cookies/tokens
**Pros:** Always works (real browser = no detection)
**Cons:** Requires manual user action
## Cloudflare Turnstile Detection Signals
Based on failed bypass attempts:
1. **User-Agent patterns:**
- `HeadlessChrome` → instant block
- Missing browser version details → suspicious
2. **Browser fingerprints:**
- WebGL renderer hash
- Canvas fingerprint
- Audio context fingerprint
- Screen resolution + color depth
3. **Behavioral signals:**
- Mouse movement (or lack thereof)
- Keyboard timing
- Scroll patterns
- Time-to-interactive
4. **Automation flags:**
- `navigator.webdriver = true`
- Missing Chrome DevTools Protocol artifacts
- Inconsistent `window.chrome` object
## Recommendations
**For rpow2swap.com specifically:**
1. **Short-term:** Manual browser inspection (Level 6)
- User opens site → extracts API calls from DevTools
- Agent replicates with captured auth tokens
2. **Medium-term:** Browserbase integration
- Configure API key in `~/.hermes/.env`
- Use cloud browser for Cloudflare-protected sites
3. **Long-term:** Contact site owner
- Request API documentation
- Ask for programmatic access method
- Report bug (orders reappearing after purchase)
## Related Issues
**Order Reappearance Bug:**
- Orders show as available after being purchased
- Likely causes:
- Race condition in order matching
- Cache invalidation failure
- Frontend state not updated via websocket
- Database replication lag
See `references/race-condition-order-matching.md` for debugging approach.
## Key Takeaway
**Cloudflare Turnstile v2 is extremely effective** against automated scraping. When facing it:
1. Try TinyFish first (easiest)
2. Try Browserbase if available (most reliable)
3. Fall back to manual browser inspection (always works)
4. **Never waste time** on Puppeteer stealth plugins — Turnstile detects them easily
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |