apify/agent-skills / skills/apify-ultimate-scraper
apify-ultimate-scraper
VerifiedUniversal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis, influencer discovery, trend research, content analytics, audience analysis, review analysis, SEO intelligence, recruitment, or any data extraction task.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.5 KB |
| references/actor-index.md | markdown | 9.7 KB |
| references/gotchas.md | markdown | 7.3 KB |
| references/workflows/brand-monitoring.md | markdown | 3.7 KB |
| references/workflows/company-research.md | markdown | 4.6 KB |
| references/workflows/competitive-intel.md | markdown | 3.5 KB |
| references/workflows/contact-enrichment.md | markdown | 3.6 KB |
| references/workflows/content-and-seo.md | markdown | 5.1 KB |
| references/workflows/ecommerce-price-monitoring.md | markdown | 4.4 KB |
| references/workflows/influencer-vetting.md | markdown | 3.9 KB |
| references/workflows/job-market-and-recruitment.md | markdown | 3.5 KB |
| references/workflows/knowledge-base-and-rag.md | markdown | 3.7 KB |
| references/workflows/lead-generation.md | markdown | 5.2 KB |
| references/workflows/real-estate-and-hospitality.md | markdown | 3.7 KB |
| references/workflows/review-analysis.md | markdown | 4.1 KB |
| references/workflows/social-media-analytics.md | markdown | 3.3 KB |
| references/workflows/trend-research.md | markdown | 3.3 KB |
references/workflows/content-and-seo.md
View on GitHub# Content and SEO workflows
## Website content extraction for RAG
**When:** User wants to crawl a website and extract clean text for AI/LLM pipelines or knowledge bases.
### Pipeline
1. **Crawl website** -> `apify/website-content-crawler`
- Key input: `startUrls`, `maxCrawlPages`, `crawlerType` ("cheerio" for speed, "playwright" for JS sites)
### Output fields
Step 1: `url`, `title`, `text`, `markdown`, `metadata`, `links[]`
### Gotcha
For JS-heavy sites (SPAs), set `crawlerType: "playwright"`. For static sites, use `"cheerio"` (10x faster). For anti-bot sites, use `apify/camoufox-scraper` instead.
## SERP-based SEO content brief generation
**When:** User wants to generate a content brief for a target keyword by analyzing competitor SERP results and heading structures.
### Pipeline
1. **SERP scrape** -> `apify/google-search-scraper`
- Key input: `queries`, `countryCode`, `maxResultsPerPage`
2. **Extract heading structure** -> `apify/cheerio-scraper`
- Pipe: `results[].organicResults[].url` -> `startUrls` (filter non-article URLs first)
- Key input: `startUrls`, `pageFunction` (extract h1/h2/h3 nodes)
### Output fields
Step 1: `organicResults[].url`, `organicResults[].title`, `organicResults[].snippet`
Step 2: heading structure (h1/h2/h3 text), word count per page
### Gotcha
Use `apify/cheerio-scraper` for heading extraction - it's 10x faster than `website-content-crawler` for HTML-only pages. Only escalate to `website-content-crawler` when you need full body text for AI synthesis.
## Sitemap content audit
**When:** User wants to crawl all URLs on a competitor's site from the sitemap and build a keyword and topic inventory.
### Pipeline
1. **Extract sitemap URLs** -> `apify/sitemap-extractor`
- Key input: `startUrls` (sitemap.xml URL)
2. **Crawl each URL** -> `apify/website-content-crawler`
- Pipe: `results[].urls[]` -> `startUrls`
- Key input: `startUrls`, `maxCrawlPages`, `htmlTransformer` (readableText)
### Output fields
Step 1: `urls[]` (all discovered page URLs)
Step 2: `text`, `metadata`, `url`
### Gotcha
Large sitemaps (1,000+ URLs) can be expensive. Filter step 1 results to a specific path prefix (e.g., `/blog/`) before passing to step 2 to avoid crawling low-value pages like tag archives and pagination.
## Keyword rank tracking with alerts
**When:** User wants weekly tracking of keyword positions for own domain and competitors, with Slack alerts on drops over 5 positions.
### Pipeline
1. **SERP scrape** -> `apify/google-search-scraper`
- Key input: `queries` (tracked keywords array), `countryCode`, `device` (desktop/mobile), `maxResultsPerPage` (set to 100)
### Output fields
Step 1: `organicResults[].position`, `organicResults[].url`, `organicResults[].domain`
### Cost estimate
`apify/google-search-scraper` is a fixed-cost Actor. 100 keywords weekly ≈ $1-3/month depending on query volume.
### Gotcha
Set `maxResultsPerPage: 100` to capture positions 11-100. Default is 10 results, which misses any keyword ranking outside page 1 - making it impossible to detect rank drops from position 12 to 18.
## SERP analysis
**When:** User wants to analyze search engine results for specific keywords.
### Pipeline
1. **Google SERP** -> `apify/google-search-scraper`
- Key input: `queries`, `maxPagesPerQuery`, `countryCode`, `languageCode`
### Output fields
Step 1: `organicResults[]` (title, url, description, position), `paidResults[]`, `peopleAlsoAsk[]`, `relatedSearches[]`
## Deep research agent
**When:** User wants an AI agent that takes a research question, generates search queries, crawls top results, and synthesizes findings into a structured report.
### Pipeline
1. **Generate search queries** (LLM node - 3 queries per research question)
2. **SERP scrape** -> `apify/google-search-scraper`
- Pipe: generated queries -> `queries`
- Key input: `queries`, `maxResultsPerPage`
3. **Extract content** -> `apify/rag-web-browser`
- Pipe: `results[].organicResults[].url` (AI-selected relevant URLs) -> `query`
- Key input: `query`, `maxResults`, `requestTimeoutSecs`
4. **Synthesize** (LLM node - final report generation)
### Output fields
Step 2: `organicResults[].url`, `organicResults[].snippet`
Step 3: `text`, `url`, `metadata.title`
### Gotcha
Use `apify/rag-web-browser` at step 3 rather than `website-content-crawler` - it's optimized for agent-based single-URL retrieval and returns clean markdown with lower latency. Reserve `website-content-crawler` for bulk batch crawls.
## Domain authority and backlink analysis
**When:** User wants SEO metrics for specific domains.
### Pipeline
1. **Traffic overview** -> `radeance/similarweb-scraper`
- Key input: `urls`
2. **Backlink profile** -> `radeance/ahrefs-scraper`
- Key input: `urls`
3. **Domain authority** -> `radeance/semrush-scraper`
- Key input: `urls`
### Output fields
Step 1: `globalRank`, `monthlyVisits`, `bounceRate`, `trafficSources`
Step 2: `domainRating`, `backlinks`, `referringDomains`, `organicKeywords`
Step 3: `authorityScore`, `organicSearchTraffic`, `paidSearchTraffic`
### Cost estimate
All radeance/ SEO Actors are PPE at $0.005-0.0275/result. Running all 3 for one domain costs ~$0.05-0.08. For 50 domains, estimate $2.50-$4.00.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |