aboylabs/ai-agent-skills / skills/research/web-scraping
web-scraping
VerifiedExtract data from websites, including JavaScript-rendered SPAs and dynamic content
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.5 KB |
| references/browser-use-cloud-api.md | markdown | 5.3 KB |
| references/browser-use-cloud.md | markdown | 3.5 KB |
| references/browserbase-api.md | markdown | 7.6 KB |
| references/cloudflare-bypass-strategies.md | markdown | 6.4 KB |
| references/cloudflare-turnstile-bypass.md | markdown | 4.5 KB |
| references/nextjs-ssr-scraping.md | markdown | 2.6 KB |
| references/race-condition-order-matching.md | markdown | 9.2 KB |
| references/security-audit-pattern.md | markdown | 7.1 KB |
| references/security-audit-workflow.md | markdown | 6.8 KB |
| references/spa-api-discovery.md | markdown | 5.5 KB |
| references/tinyfish-js-reverse-engineering.md | markdown | 6.7 KB |
| references/trendshift-analysis.md | markdown | 2.2 KB |
references/trendshift-analysis.md
View on GitHub# Analyzing Trending Repositories (TrendShift Pattern)
When user asks to analyze trending repos from TrendShift.io or similar platforms:
## Data Extraction
TrendShift uses client-side rendering with embedded JSON:
- Fetch HTML: `curl -s "https://trendshift.io/"`
- Extract JSON from `<script>` tags containing repo data
- Parse embedded data structure (usually in `self.__next_f.push()` calls)
## Analysis Framework
### 1. Categorization
Group repos by:
- **Type:** AI agent, tools, learning resources, libraries, frameworks
- **Domain:** DevOps, ML/AI, web dev, security, data science
- **Language:** Python, TypeScript, Rust, Go, etc.
- **Tags:** From repo metadata
### 2. Quality Indicators
Assess each repo:
- **Stars:** Popularity signal
- **Forks:** Community engagement
- **Score:** TrendShift engagement score (if available)
- **Social mentions:** Reddit, HackerNews validation
- **Recency:** Last updated, creation date
- **Activity:** Commit frequency, issue response
### 3. Trend Analysis
Identify patterns:
- **Dominant categories:** What's hot right now?
- **Language preferences:** Which languages trending?
- **Community signals:** Social mention correlation
- **Emerging patterns:** New paradigms, approaches
### 4. Practical Assessment
For each category, answer:
- **Bagus gak?** (Is it good?) — Star rating
- **Why it matters:** Practical value
- **Use cases:** When to use
- **Top picks:** Best repos in category
## Output Format
User prefers:
- **Bahasa Indonesia** (casual style)
- **Direct, no-BS** explanations
- **Practical focus:** "Gunanya apa?" not just "What is it?"
- **Clear recommendations:** Top 5 must-try
- **Visual aids:** Tables, comparisons
- **Actionable insights:** What to do next
## Deliverables
Generate:
1. **Main report** (Bahasa Indonesia, comprehensive)
2. **Quick summary** (TL;DR version)
3. **Data tables** (structured reference)
4. **Raw data** (JSON for further analysis)
5. **Categorized data** (organized by type)
## Pitfalls
- **Don't scrape rendered content:** TrendShift is client-side rendered, extract from embedded JSON
- **Don't just list repos:** Provide analysis, insights, recommendations
- **Don't use English:** User prefers Bahasa Indonesia (casual)
- **Don't be verbose:** Direct, practical, actionable
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |