aboylabs/ai-agent-skills / skills/research/web-scraping
web-scraping
VerifiedExtract data from websites, including JavaScript-rendered SPAs and dynamic content
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.5 KB |
| references/browser-use-cloud-api.md | markdown | 5.3 KB |
| references/browser-use-cloud.md | markdown | 3.5 KB |
| references/browserbase-api.md | markdown | 7.6 KB |
| references/cloudflare-bypass-strategies.md | markdown | 6.4 KB |
| references/cloudflare-turnstile-bypass.md | markdown | 4.5 KB |
| references/nextjs-ssr-scraping.md | markdown | 2.6 KB |
| references/race-condition-order-matching.md | markdown | 9.2 KB |
| references/security-audit-pattern.md | markdown | 7.1 KB |
| references/security-audit-workflow.md | markdown | 6.8 KB |
| references/spa-api-discovery.md | markdown | 5.5 KB |
| references/tinyfish-js-reverse-engineering.md | markdown | 6.7 KB |
| references/trendshift-analysis.md | markdown | 2.2 KB |
references/nextjs-ssr-scraping.md
View on GitHub# Next.js SSR Site Scraping Example
## Case: MegaETH KPIs Page
**URL:** https://flux.megaeth.com/kpis
**Framework:** Next.js with server-side rendering
**Challenge:** Content rendered client-side, no API endpoint available
## Failed Approaches
### 1. Browser Tool (Playwright via Hermes)
```
Error: Chrome exited early without writing DevToolsActivePort
FATAL: No usable sandbox! Ubuntu 23.10+ AppArmor restrictions
```
**Root cause:** WSL2 Ubuntu 24.04 lacks unprivileged user namespaces for Chrome sandbox.
### 2. Python + Playwright
```
Error: externally-managed-environment
python3-venv package not installed
```
**Root cause:** Ubuntu 24.04 requires explicit `python3.12-venv` package.
## Working Solution: Node.js + Puppeteer
### Setup
```bash
cd /tmp && mkdir scraper && cd scraper
npm init -y
npm install puppeteer
```
### Script
```javascript
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox', '--disable-setuid-sandbox'] // Critical for WSL2
});
const page = await browser.newPage();
try {
await page.goto('https://flux.megaeth.com/kpis', {
waitUntil: 'networkidle2',
timeout: 60000
});
// Wait for React hydration
await new Promise(resolve => setTimeout(resolve, 5000));
// Extract all text
const content = await page.evaluate(() => document.body.innerText);
console.log(content);
// Extract structured data
const kpiData = await page.evaluate(() => {
const percentages = Array.from(document.querySelectorAll('*'))
.filter(el => el.innerText && el.innerText.includes('%'))
.map(el => el.innerText.trim());
return { percentages: [...new Set(percentages)] };
});
console.log(JSON.stringify(kpiData, null, 2));
} finally {
await browser.close();
}
})();
```
### Key Learnings
1. **`--no-sandbox` is mandatory** in WSL2/containers — Chrome won't launch without it
2. **Wait after networkidle2** — React/Next.js needs time to hydrate
3. **Extract text first** — verify content is present before refining selectors
4. **Use `new Promise(setTimeout)` not `page.waitForTimeout`** — Puppeteer API changed
## Results
Successfully extracted:
- 13 KPIs with 0% completion
- 5.3B MEGA total rewards ($834.93M)
- All tranche milestones and requirements
- No date-based unlocks (all milestone-based)
## Takeaway
For Next.js/React sites in WSL2:
1. Skip browser tool if sandbox errors appear
2. Use Node.js + Puppeteer with `--no-sandbox`
3. Always wait 3-5 seconds post-load for hydration
4. Extract incrementally (text → structured data)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |