Skip to content

aboylabs/ai-agent-skills / skills/research/web-scraping

web-scraping

Verified

Extract data from websites, including JavaScript-rendered SPAs and dynamic content

references/cloudflare-turnstile-bypass.md

View on GitHub
# Cloudflare Turnstile Bypass Strategies

Session: rpow2swap.com analysis (May 2026)

## Problem

Site protected by Cloudflare Turnstile — advanced bot detection that blocks:
- curl (403 Forbidden)
- TinyFish fetch API (`bot_blocked`)
- Standard Puppeteer headless (`HeadlessChrome` detected)
- Puppeteer + stealth plugin (still stuck on "Just a moment..." challenge)

## Escalation Ladder

### Level 1: Direct API Access (FAILED)
```bash
# All endpoints return 403
curl -X POST https://rpow2swap.com/api/payout \
  -H "Content-Type: application/json" \
  -d '{"recipient":"user@example.com","amount":100000}'
# → 403 Forbidden (Cloudflare challenge page)
```

**Result:** Cloudflare blocks all programmatic access.

### Level 2: TinyFish Fetch (FAILED)
```bash
curl -X POST https://api.fetch.tinyfish.ai \
  -H "X-API-Key: $TINYFISH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://rpow2swap.com/"]}'
# → {"results":[],"errors":[{"url":"...","error":"bot_blocked"}]}
```

**Result:** TinyFish cloud browser also detected and blocked.

### Level 3: Puppeteer Headless (FAILED)
```javascript
const puppeteer = require('puppeteer');
const browser = await puppeteer.launch({ headless: true });
// → Stuck on "Just a moment..." Cloudflare challenge
```

**Detection signals:**
- `HeadlessChrome` in User-Agent
- Missing browser fingerprints (WebGL, Canvas, Audio)
- Automation flags (`navigator.webdriver = true`)

### Level 4: Puppeteer + Stealth Plugin (FAILED)
```javascript
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
// → Still stuck on challenge (Turnstile too advanced)
```

**Result:** Cloudflare Turnstile v2 detects even stealth browsers via:
- Mouse movement patterns
- Timing analysis
- Browser API inconsistencies

### Level 5: Browserbase Cloud (NOT TESTED)
```python
import requests

response = requests.post(
    "https://www.browserbase.com/v1/sessions",
    headers={"X-BB-API-Key": API_KEY},
    json={"projectId": PROJECT_ID}
)
session_id = response.json()['id']
# Connect via CDP and navigate
```

**Status:** Not tested (requires API key configuration).  
**Expected:** May work (residential IPs + real browser fingerprints).

### Level 6: Manual Browser Inspection (RECOMMENDED)
**When all automation fails, guide user to extract API manually:**

1. User opens site in real browser (Chrome/Firefox)
2. Wait for Cloudflare challenge to complete
3. Open DevTools → Network tab
4. Interact with site (trigger API calls)
5. Copy API endpoint + headers + payload
6. Replicate with curl using captured cookies/tokens

**Pros:** Always works (real browser = no detection)  
**Cons:** Requires manual user action

## Cloudflare Turnstile Detection Signals

Based on failed bypass attempts:

1. **User-Agent patterns:**
   - `HeadlessChrome` → instant block
   - Missing browser version details → suspicious

2. **Browser fingerprints:**
   - WebGL renderer hash
   - Canvas fingerprint
   - Audio context fingerprint
   - Screen resolution + color depth

3. **Behavioral signals:**
   - Mouse movement (or lack thereof)
   - Keyboard timing
   - Scroll patterns
   - Time-to-interactive

4. **Automation flags:**
   - `navigator.webdriver = true`
   - Missing Chrome DevTools Protocol artifacts
   - Inconsistent `window.chrome` object

## Recommendations

**For rpow2swap.com specifically:**

1. **Short-term:** Manual browser inspection (Level 6)
   - User opens site → extracts API calls from DevTools
   - Agent replicates with captured auth tokens

2. **Medium-term:** Browserbase integration
   - Configure API key in `~/.hermes/.env`
   - Use cloud browser for Cloudflare-protected sites

3. **Long-term:** Contact site owner
   - Request API documentation
   - Ask for programmatic access method
   - Report bug (orders reappearing after purchase)

## Related Issues

**Order Reappearance Bug:**
- Orders show as available after being purchased
- Likely causes:
  - Race condition in order matching
  - Cache invalidation failure
  - Frontend state not updated via websocket
  - Database replication lag

See `references/race-condition-order-matching.md` for debugging approach.

## Key Takeaway

**Cloudflare Turnstile v2 is extremely effective** against automated scraping. When facing it:
1. Try TinyFish first (easiest)
2. Try Browserbase if available (most reliable)
3. Fall back to manual browser inspection (always works)
4. **Never waste time** on Puppeteer stealth plugins — Turnstile detects them easily
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall