Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/crawl4ai

crawl4ai

Verified

references/crawl4ai-summary.md

View on GitHub
# Crawl4AI Analysis: Quick Reference Guide

## Key Findings at a Glance

### Core Design Patterns (6 Major Patterns)

| Pattern | Primary Use | Key Classes |
|---------|-------------|-------------|
| **Strategy** | Extraction methods | `JsonCssExtractionStrategy`, `LLMExtractionStrategy`, `*DeepCrawlStrategy` |
| **Builder** | Configuration | `BrowserConfig`, `CrawlerRunConfig` |
| **Context Manager** | Resource lifecycle | `AsyncWebCrawler` |
| **Hook/Callback** | Extension points | `BrowserConfig.hooks` dictionary |
| **Factory** | Content-type routing | Auto-selection of strategies (implicit) |
| **Composite** | Multiple strategies | Combining CSS + LLM extraction |

---

## Primary Data Model

### CrawlResult (The Core Output)
```
CrawlResult
├── url: str                                    [Where we crawled]
├── status: CrawlStatus                         [Success/failure]
├── markdown: MarkdownGenerationResult          [LLM-ready content]
│   ├── raw_markdown: str                       [Original extraction]
│   ├── fit_markdown: str                       [Cleaned version]
│   └── fit_html: str                           [HTML version]
├── extracted_data: Dict[str, Any]              [Structured extraction]
├── media: Dict[str, List]                      [Images, videos, etc.]
├── metadata: Dict[str, Any]                    [Page metadata]
├── raw_html: str                               [Original HTML]
├── pdf: bytes                                  [PDF if requested]
├── screenshot: bytes                           [Page screenshot]
└── session_id: str                             [Browser session ID]
```

---

## Extension Points (5 Main Categories)

### 1. Extraction Strategies
- **CSS-based:** Fast, deterministic, no LLM cost
- **LLM-based:** Semantic understanding, Pydantic schema validation
- **PDF-specific:** Specialized PDF parsing strategies

### 2. Hooks/Callbacks
- `on_page_context_created` - Perfect for login automation
- `on_before_fetch`, `on_after_fetch`, `on_content_ready`
- Receive Playwright page object for full control

### 3. JavaScript Execution
- Custom JS injection via `js_code` parameter
- Trigger dynamic content loading
- Wait for specific conditions with `wait_for` parameter

### 4. Deep Crawling
- `BFSDeepCrawlStrategy` - Breadth-first discovery
- `DFSDeepCrawlStrategy` - Depth-first discovery
- URL filtering: include/exclude patterns, domain limits

### 5. Content Filtering
- `css_filters` - Remove noise (nav, ads, footer, etc.)
- Multiple markdown versions: raw vs. cleaned
- Post-processing capabilities

---

## Configuration Hierarchy

### Minimum Config
```python
BrowserConfig()
CrawlerRunConfig(url="https://example.com")
```

### Full Config
```python
# Browser setup (persistent)
BrowserConfig(
    headless=True,                              # Headless mode
    use_managed_browser=True,                   # Persistent profile
    user_data_dir="./profiles/auth",            # Profile location
    browser_type="chromium",                    # Playwright browser
    viewport_size=(1920, 1080),                 # Page size
    proxy_type="http",                          # Proxy support
    hooks={"on_page_context_created": fn}      # Lifecycle hooks
)

# Run-specific config (per crawl)
CrawlerRunConfig(
    url="https://example.com",                  # Target URL
    extraction_strategy=JsonCssExtractionStrategy(...),  # How to extract
    js_code="// Custom JS",                     # Dynamic content
    wait_for="selector:.loaded",                # Wait condition
    delay=2.0,                                  # Rate limiting
    css_filters=["nav", ".ad"],                 # Remove elements
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2)  # Recursive
)
```

---

## Design Principles

### 1. LLM-Ready by Default
- Converts HTML → Clean Markdown
- Removes noise automatically
- Multiple output formats (markdown, HTML, JSON)

### 2. Async-First Architecture
- Non-blocking operations via `async`/`await`
- Browser pool for concurrency
- Efficient resource management with context managers

### 3. Composable Strategies
- Mix & match extraction approaches
- CSS for structured data, LLM for semantic understanding
- Switch strategies at runtime

### 4. Type-Safe Extraction
- Pydantic schema validation
- JSON Schema generation
- Automatic type coercion

### 5. Extensible Pipeline
- Multiple hooks for custom logic
- JavaScript injection for dynamic content
- CSS filtering for content pruning

---

## Common Patterns & Use Cases

### Quick Reference (Pattern → Implementation)

| Use Case | Pattern | Key Components |
|----------|---------|-----------------|
| Extract product prices | CSS Extraction | `JsonCssExtractionStrategy` + CSS selectors |
| Login to protected site | Hook + Session | `on_page_context_created` hook + `use_managed_browser` |
| Handle dynamic JS | JS Injection | `js_code` parameter + `wait_for` selector |
| Crawl entire site | Deep Crawling | `BFSDeepCrawlStrategy` or `DFSDeepCrawlStrategy` |
| Extract structured data | LLM Extraction | `LLMExtractionStrategy` + Pydantic schema |
| Clean for LLM input | Content Filtering | `css_filters` + `fit_markdown` |
| Parse PDF documents | PDF Strategy | Auto-detection + `PDFCrawlerStrategy` |
| Integrate with pipeline | dlt resource | Wrap in `@dlt.resource` decorator |
| Multi-page comparison | Multi-URL crawl | Loop with same extraction strategy |
| Get clean markdown | Content Pruning | Enable filters + use `fit_markdown` |

---

## Integration Ecosystem

### Upstream Sources
- Static HTML pages
- Dynamic JavaScript-heavy pages
- PDF documents
- Protected sites (with auth)
- Multi-page websites (deep crawling)

### Downstream Destinations
- **Data Pipelines:** dlt → DuckDB, PostgreSQL, Parquet
- **Vector DBs:** LanceDB, pgvector (for semantic search)
- **AI Models:** Direct input for RAG, fine-tuning
- **Document Processors:** Docling (complex PDFs), Unstract (structured extraction)
- **AI Agents:** Agno, LangChain (agentic workflows)

---

## Best Practices Checklist

### Performance
- [ ] Use CSS extraction first (cheaper/faster)
- [ ] Enable caching for repeated URLs
- [ ] Set appropriate `delay` for rate limiting
- [ ] Batch multiple URLs
- [ ] Use `css_filters` to reduce markdown size

### Reliability
- [ ] Check `result.status.is_success()` before processing
- [ ] Implement retry logic with exponential backoff
- [ ] Store original HTML alongside markdown
- [ ] Use session management for auth
- [ ] Log full results for debugging

### Data Quality
- [ ] Validate with Pydantic schemas
- [ ] Combine CSS (fast) + LLM (accurate)
- [ ] Store both `raw_markdown` and `fit_markdown`
- [ ] Use media extraction for images
- [ ] Test with real pages first

### Cost Management (LLM)
- [ ] CSS extraction → fallback to LLM
- [ ] Batch LLM requests
- [ ] Use cheaper models (GPT-3.5 Turbo, Claude Sonnet)
- [ ] Cache results by URL
- [ ] Monitor API costs

### Ethics & Legal
- [ ] Respect robots.txt
- [ ] Use appropriate `delay` values
- [ ] Set User-Agent headers
- [ ] Only crawl authorized content
- [ ] Get user consent for auth flows

---

## Architecture Strengths

1. **Flexibility:** Multiple strategies, swappable at runtime
2. **LLM-Native:** Built for AI/ML consumption from ground up
3. **Extensible:** Hooks, custom strategies, integration points
4. **Modern:** Async/await, Pydantic, type hints
5. **Practical:** Real-world features (auth, JS, PDF, proxies)

---

## Key Code Examples

### Basic Crawl
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(CrawlerRunConfig(url="https://example.com"))
    print(result.markdown.fit_markdown)
```

### Structured Extraction
```python
from crawl4ai.extraction import JsonCssExtractionStrategy

strategy = JsonCssExtractionStrategy(
    extractions=[
        {"name": "title", "css": "h1", "type": "text"},
        {"name": "price", "css": ".price", "type": "text"},
    ]
)
config = CrawlerRunConfig(url="https://shop.com", extraction_strategy=strategy)
result = await crawler.arun(config)
print(result.extracted_data)  # {'title': '...', 'price': '...'}
```

### Login Automation
```python
async def login(page, context, **kwargs):
    await page.fill('input[name="email"]', "user@example.com")
    await page.fill('input[name="password"]', "password")
    await page.click("button[type='submit']")
    await page.wait_for_url("**/dashboard")

config = BrowserConfig(hooks={"on_page_context_created": login})
```

### dlt Integration
```python
@dlt.resource
async def web_crawler():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(CrawlerRunConfig(url="https://api.example.com"))
        yield result.extracted_data

pipeline = dlt.pipeline("web_crawler", destination="duckdb")
load_info = pipeline.run(web_crawler())
```

---

## Files in Repository

- `/home/user/hackathon/CRAWL4AI_ANALYSIS.md` - Comprehensive 758-line analysis
- `/home/user/hackathon/research/pdf/ingestion/crawl4ai_dlt.md` - Integration with dlt tutorial
- `/home/user/hackathon/infrastructure/compose/agno/cookbook/tools/crawl4ai_tools.py` - Agno integration example

---

## Next Steps for Usage

1. **Start Simple:** Basic crawl without extraction
2. **Add Extraction:** CSS selectors for known structure
3. **Handle Auth:** BrowserProfiler for login
4. **Scale Up:** Deep crawl strategies for entire sites
5. **Integrate:** Connect with dlt, vector DBs, AI models
6. **Optimize:** Monitor costs, cache, batch requests
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall