Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/crawl4ai

crawl4ai

Verified

references/crawl4ai-analysis.md

View on GitHub
# Crawl4AI Codebase Analysis & Architecture Guide

## Executive Summary

Crawl4AI is a modern, open-source web scraping and content extraction framework purpose-built for AI/LLM applications. It specializes in converting dynamic web content into clean, LLM-ready structured data. The framework emphasizes performance, async-first design, and semantic understanding of web content.

**Key Characteristics:**
- Async-first Python library with Playwright/Chromium browser automation
- LLM-optimized output (Markdown, JSON) for AI applications
- Dual extraction approach: CSS selectors (fast) + LLM-powered (semantic)
- Persistent browser profiles for authenticated scraping
- Deep crawling strategies (BFS, DFS) for site-wide data extraction
- Built for integration with data pipelines (dlt, Dagster, etc.)

---

## 1. REPOSITORY STRUCTURE

### Overall Project Layout

```
crawl4ai/
├── crawl4ai/                      # Core library code
│   ├── __init__.py
│   ├── async_crawler.py           # Main AsyncWebCrawler class
│   ├── browser_config.py          # Browser configuration
│   ├── crawler_run_config.py      # Runtime configuration
│   ├── browser_profiler.py        # Profile management for auth
│   ├── extraction/                # Extraction strategies
│   │   ├── extraction_strategy.py # Base class
│   │   ├── json_css_extraction.py # CSS selector extraction
│   │   ├── llm_extraction.py      # LLM-powered extraction
│   │   └── markdown_generation.py # Markdown conversion
│   ├── deep_crawl/                # Deep crawling
│   │   ├── deep_crawl_strategy.py # Base strategy
│   │   ├── bfs_strategy.py        # Breadth-first search
│   │   ├── dfs_strategy.py        # Depth-first search
│   │   └── url_filter.py          # Pattern matching
│   ├── hooks/                     # Lifecycle hooks
│   │   ├── browser_hooks.py       # Page lifecycle hooks
│   │   ├── auth_hooks.py          # Authentication hooks
│   │   └── hook_registry.py       # Hook management
│   ├── cache/                     # Caching layer
│   │   ├── cache_manager.py       # Cache interface
│   │   ├── memory_cache.py        # In-memory cache
│   │   └── persistent_cache.py    # Disk-based cache
│   ├── models/                    # Data models
│   │   ├── crawl_result.py        # Result container
│   │   ├── crawl_status.py        # Status codes
│   │   └── browser_profile.py     # Profile models
│   └── utils/                     # Utilities
│       ├── markdown_utils.py      # Markdown processing
│       ├── content_cleaner.py     # HTML cleaning
│       ├── link_extractor.py      # Link extraction
│       └── network_utils.py       # Network helpers
├── docs/                          # API documentation
├── tests/                         # Test suite
├── examples/                      # Example scripts
├── requirements.txt               # Dependencies
├── setup.py / pyproject.toml      # Package configuration
└── README.md                      # Project README
```

### Core Modules Explained

**asynccrawler.py**
- Primary entry point
- Manages browser lifecycle and pooling
- Handles request routing and result aggregation
- Implements context manager protocol

**browser_config.py**
- Browser initialization parameters
- Network configuration (proxies, headers)
- Resource optimization options
- Profile/session management settings

**crawler_run_config.py**
- Per-request configuration
- Extraction strategy selection
- Content post-processing rules
- Browser interaction parameters

**extraction/ Module**
- Pluggable extraction strategies
- CSS/XPath selector-based extraction
- LLM-powered semantic extraction
- Markdown conversion and formatting

**deep_crawl/ Module**
- Recursive website traversal
- URL discovery and filtering
- Depth/breadth-first algorithms
- Domain and pattern restrictions

---

## 2. CORE ARCHITECTURE

### High-Level Architecture Diagram

```
┌────────────────────────────────────────────────┐
│     User Code / LLM Agents / Pipelines         │
└───────────────────┬────────────────────────────┘
                    │
                    ▼
    ┌───────────────────────────────────┐
    │   AsyncWebCrawler (Main API)      │
    │   - arun(url, config)             │
    │   - Context manager support       │
    │   - Browser pool management       │
    └───┬───────────────────┬───────────┘
        │                   │
        ▼                   ▼
    ┌──────────────┐   ┌─────────────────────┐
    │BrowserConfig │   │ CrawlerRunConfig    │
    │             │   │                     │
    │- headless   │   │- extraction_strategy│
    │- proxies    │   │- wait_for           │
    │- profile    │   │- js_code            │
    │- timeout    │   │- deep_crawl_strategy│
    │- viewport   │   │- delay              │
    └──────┬──────┘   └──────┬──────────────┘
           │                  │
           └──────┬───────────┘
                  ▼
    ┌─────────────────────────────────┐
    │   Playwright/Chromium Browser   │
    │   - JavaScript rendering        │
    │   - DOM interaction             │
    │   - Network handling            │
    └────────────┬────────────────────┘
                 │
    ┌────────────┴──────────────┐
    ▼                           ▼
┌──────────┐         ┌──────────────────────┐
│ Website  │         │  Extraction Pipeline │
│ (HTML)   │         │                      │
│          │         │ 1. Extraction        │
└──────────┘         │    - CSS selectors   │
                     │    - LLM parsing     │
                     │                      │
                     │ 2. Post-Processing   │
                     │    - Content clean   │
                     │    - Link extraction │
                     │    - Markdown conv   │
                     │                      │
                     │ 3. Caching           │
                     │    - Result storage  │
                     └──────┬───────────────┘
                            ▼
                  ┌──────────────────────┐
                  │   CrawlResult        │
                  │                      │
                  │- extracted_data      │
                  │- markdown            │
                  │- html                │
                  │- status              │
                  │- metadata            │
                  │- network_log         │
                  └──────────────────────┘
```

### Data Flow Architecture

**1. Initialization Phase**
```
User Creates AsyncWebCrawler(config=BrowserConfig)
    │
    ├─ Initialize browser_config
    ├─ Start Chromium process
    ├─ Load browser profile (if managed_browser=True)
    ├─ Initialize browser pool
    └─ Ready for requests
```

**2. Request Phase**
```
await crawler.arun(url, config=CrawlerRunConfig)
    │
    ├─ Acquire browser from pool
    ├─ Create new page context
    ├─ Navigate to URL
    ├─ Wait for readiness (DOMContentLoaded/wait_for)
    ├─ Execute hooks (on_page_load)
    ├─ Execute custom JS (if provided)
    └─ Render complete
```

**3. Processing Phase**
```
Raw HTML/DOM
    │
    ├─ Apply Extraction Strategy
    │  ├─ CSS Selector Extraction: Parse with BeautifulSoup
    │  ├─ LLM Extraction: Call LLM API with schema
    │  └─ Markdown: Convert HTML to clean Markdown
    │
    ├─ Content Post-Processing
    │  ├─ Remove noise (scripts, ads, navigation)
    │  ├─ Extract links and references
    │  ├─ Format output
    │  └─ Validate against schema
    │
    ├─ Caching
    │  └─ Store result if caching enabled
    │
    └─ Return CrawlResult
```

### Key Architectural Patterns

**1. Async/Concurrent Design**
- All I/O operations are async
- Browser pool for concurrent requests
- Non-blocking request handling
- Handles hundreds of concurrent crawls

**2. Strategy Pattern (Extraction)**
- Base `ExtractionStrategy` interface
- Multiple implementations:
  - `JsonCssExtractionStrategy`
  - `LLMExtractionStrategy`
  - `MarkdownExtractionStrategy`
- User-selectable at runtime

**3. Strategy Pattern (Deep Crawling)**
- Base `DeepCrawlStrategy` interface
- BFS and DFS implementations
- Configurable URL filtering
- Depth and page limits

**4. Hook/Event System**
- Pre/post request hooks
- Page lifecycle hooks
- Authentication hooks
- Extensible plugin system

**5. Configuration Objects**
- `BrowserConfig`: Persistent, reusable
- `CrawlerRunConfig`: Per-request customization
- Separation of concerns
- Type-safe (Pydantic models)

---

## 3. KEY FEATURES

### A. Content Extraction Capabilities

#### 1. CSS Selector Extraction (Fast & Cheap)

```python
from crawl4ai.extraction import JsonCssExtractionStrategy

strategy = JsonCssExtractionStrategy(
    extractions=[
        {
            "name": "title",
            "css": "h1.article-title",
            "type": "text"  # text | link | html | attribute | list
        },
        {
            "name": "date",
            "css": ".publish-date",
            "type": "attribute",
            "attribute": "data-timestamp"
        },
        {
            "name": "content",
            "css": "article.body",
            "type": "html"
        },
        {
            "name": "links",
            "css": "a.related-link",
            "type": "list",  # Returns array
            "fields": {
                "text": {"type": "text"},
                "href": {"type": "attribute", "attribute": "href"}
            }
        }
    ]
)
```

**Extraction Types:**
- `text`: Extract textContent
- `html`: Extract innerHTML
- `attribute`: Extract specific attribute
- `link`: Extract href and text
- `list`: Extract multiple elements as array

**Advantages:**
- Zero API cost
- Deterministic results
- Millisecond execution
- Works with static and dynamic content
- XPath support via lxml

**Best For:**
- Product e-commerce pages
- News articles with consistent structure
- Tables and data listings
- Directory sites
- Any well-structured HTML

#### 2. LLM-Powered Extraction (Semantic)

```python
from crawl4ai.extraction import LLMExtractionStrategy
from pydantic import BaseModel, Field

class Product(BaseModel):
    name: str = Field(description="Product name")
    price: float = Field(description="Price in USD")
    rating: float = Field(ge=0, le=5, description="1-5 star rating")
    in_stock: bool = Field(description="Availability")
    description: str = Field(description="Product description")

strategy = LLMExtractionStrategy(
    provider="openai",  # openai | anthropic | gemini | etc.
    api_token="sk-...",
    schema=Product,
    instruction="Extract complete product details from the page. Be thorough and accurate.",
    temperature=0.1,  # Low temperature for consistency
)
```

**Features:**
- Pydantic schema support for type validation
- Multiple LLM provider support
- Natural language instructions
- Chunk content for token limits
- Retry logic for failures
- Schema validation

**Advantages:**
- Handles unstructured/inconsistent HTML
- Semantic understanding
- Flexible instructions
- Handles complex layouts
- Context-aware extraction

**Trade-offs:**
- LLM API costs
- Latency (1-5 seconds per page)
- Model dependency
- Rate limiting considerations

**Best For:**
- Unstructured content
- Multiple layout variations
- Natural language extraction
- Complex business logic
- Content summarization

#### 3. Markdown Conversion

**Automatically converts HTML to clean Markdown:**
- Preserves document hierarchy
- Removes noise (scripts, styles, ads)
- Converts links to citations: `[text][1]` + `[1]: url`
- Tables to Markdown format
- Code blocks with language hints
- Headers, lists, emphasis preserved
- Absolute URLs for all links

**Example Transformation:**
```html
<!-- Input HTML -->
<article>
  <h1>Article Title</h1>
  <span class="date">2024-01-15</span>
  <p>Introduction paragraph.</p>
  <h2>Section</h2>
  <p>Content with <b>bold</b> and <i>italic</i>.</p>
  <script>// noise removed</script>
  <a href="/page">Related</a>
</article>
```

```markdown
# Article Title

**Published:** 2024-01-15

Introduction paragraph.

## Section

Content with **bold** and *italic*.

- [Related][1]

[1]: https://example.com/page
```

**Output Benefits:**
- LLM-friendly format
- Human-readable
- Preserves semantic structure
- Reduces token usage in LLMs
- Perfect for RAG pipelines

### B. Dynamic Content & Interaction

#### 1. JavaScript Execution

Execute custom JS before extraction:

```python
config = CrawlerRunConfig(
    js_code="""
    // Infinite scroll handling
    async function autoScroll() {
        let lastHeight = document.body.scrollHeight;
        while(true) {
            window.scrollTo(0, document.body.scrollHeight);
            await new Promise(resolve => setTimeout(resolve, 1000));
            let newHeight = document.body.scrollHeight;
            if(newHeight === lastHeight) break;
            lastHeight = newHeight;
        }
    }
    await autoScroll();
    
    // Click load more buttons
    document.querySelectorAll('.load-more').forEach(btn => btn.click());
    
    // Extract custom data
    window.pageData = {
        title: document.title,
        items: Array.from(document.querySelectorAll('.item'))
                    .map(el => ({
                        name: el.querySelector('.name').textContent,
                        price: el.querySelector('.price').textContent
                    }))
    };
    """
)
```

**Capabilities:**
- Full DOM manipulation
- Async/await support
- Event triggering
- Custom data extraction
- State management

**Use Cases:**
- Infinite scroll pages
- Single-page apps (React, Vue, Angular)
- AJAX-loaded content
- Form submission
- Cookie/banner dismissal
- Dynamic page rendering

#### 2. Wait Conditions

```python
config = CrawlerRunConfig(
    # Option 1: Wait for specific element
    wait_for=".article-content",
    
    # Option 2: Wait for network idle
    wait_for="networkidle",
    
    # Option 3: Wait for page load
    wait_for="load",
    
    # Combined with timeout
    # wait_timeout=10000  # 10 seconds
)
```

**Wait Types:**
- CSS selector: Waits for element to be visible
- "load": Page load event
- "networkidle": No pending network requests
- "domcontentloaded": DOM ready event

#### 3. Session/State Management

Maintain authentication across requests:

```python
# First request: login
result1 = await crawler.arun(
    "https://example.com/login",
    config=CrawlerRunConfig(
        js_code="""
        document.querySelector('input[name="email"]').value = "user@example.com";
        document.querySelector('input[name="password"]').value = "password";
        document.querySelector('form').submit();
        """,
        session_id="user_session",
        wait_for=".dashboard"
    )
)

# Second request: reuse session
result2 = await crawler.arun(
    "https://example.com/protected",
    config=CrawlerRunConfig(
        session_id="user_session"
    )
)
```

**Features:**
- Cookie persistence
- LocalStorage/SessionStorage preservation
- DOM state retention
- CSRF token handling
- Multiple concurrent sessions

### C. Authentication & Identity

#### 1. Browser Profile Management

```python
from crawl4ai import BrowserProfiler, BrowserConfig

# Step 1: Create profile with manual login
profiler = BrowserProfiler()
profile_path = await profiler.create_profile(
    profile_name="my_account",
    base_url="https://example.com"
)
# User logs in manually in browser
# Profile saves automatically

# Step 2: Reuse profile in future crawls
browser_config = BrowserConfig(
    use_managed_browser=True,
    user_data_dir=profile_path,
    browser_type="chromium"
)

crawler = AsyncWebCrawler(config=browser_config)
result = await crawler.arun("https://example.com/protected-page")
```

**Profile Storage Includes:**
- Cookies and session storage
- LocalStorage data
- IndexedDB
- Service worker cache
- Cached credentials
- Browsing history

**Advantages:**
- One-time manual login
- Handles complex auth (2FA, CAPTCHA)
- Session preservation
- Appears as real user
- Bot-detection resistant

#### 2. Programmatic Authentication via Hooks

```python
async def auto_login(page, context, **kwargs):
    """Hook function executed before page load"""
    # Navigate to login page
    await page.goto("https://example.com/login")
    
    # Fill and submit form
    await page.fill('input[name="username"]', "user@example.com")
    await page.fill('input[name="password"]', "secure_password")
    await page.click('button:has-text("Login")')
    
    # Wait for redirect
    await page.wait_for_url("**/dashboard")
    
    return page

browser_config = BrowserConfig(
    hooks={"on_page_context_created": auto_login}
)

crawler = AsyncWebCrawler(config=browser_config)
result = await crawler.arun("https://example.com/protected")
```

### D. Deep Crawling / Site-Wide Scraping

#### 1. Breadth-First Search (BFS)

```python
from crawl4ai.deep_crawl import BFSDeepCrawlStrategy

strategy = BFSDeepCrawlStrategy(
    max_depth=3,                           # 3 levels deep
    max_pages=500,                         # Max 500 pages
    same_domain_only=True,                 # Stay on domain
    include_patterns=[
        ".*\/docs\/.*",                    # Include only /docs/*
        ".*\/api\/.*"
    ],
    exclude_patterns=[
        ".*\/admin\/.*",                   # Skip /admin/*
        ".*\.pdf$",                        # Skip PDFs
        ".*\/search\?.*"                   # Skip search pages
    ],
    max_retries=3,                         # Retry failed pages
    delay=1.0                              # 1 second delay
)

config = CrawlerRunConfig(deep_crawl_strategy=strategy)
result = await crawler.arun("https://docs.example.com", config=config)
```

**BFS Characteristics:**
- Level-by-level exploration
- Systematic coverage
- Finds shortest paths to pages
- Good for breadth-focused crawling
- Memory-efficient discovery

#### 2. Depth-First Search (DFS)

```python
from crawl4ai.deep_crawl import DFSDeepCrawlStrategy

strategy = DFSDeepCrawlStrategy(
    max_depth=10,                          # Follow deep paths
    max_pages=1000,
    same_domain_only=True,
    discovery_type="auto",                 # auto | manual
    max_retries=2
)

config = CrawlerRunConfig(deep_crawl_strategy=strategy)
result = await crawler.arun("https://docs.example.com", config=config)
```

**DFS Characteristics:**
- Deep path exploration
- Follows links exhaustively
- Finds all linked content
- Memory usage grows with depth
- Good for complete coverage

**Configuration Parameters:**
- `max_depth`: Maximum link depth
- `max_pages`: Total page limit
- `same_domain_only`: Domain restriction
- `include_patterns`: Regex for included URLs
- `exclude_patterns`: Regex for excluded URLs
- `max_retries`: Failure retry attempts
- `delay`: Delay between requests

### E. Browser Automation & Control

#### 1. Network Configuration

```python
browser_config = BrowserConfig(
    # Proxy settings
    proxy="http://proxy.company.com:8080",
    
    # Custom headers
    headers={
        "Accept-Language": "en-US,en;q=0.9",
        "User-Agent": "Custom Agent/1.0",
        "Referer": "https://example.com"
    },
    
    # Extra HTTP headers
    extra_http_headers={
        "X-Custom-Header": "value"
    },
    
    # SSL/TLS
    ignore_https_errors=True
)
```

#### 2. Browser Resource Management

```python
browser_config = BrowserConfig(
    # Performance optimization
    disable_images=True,       # Don't load images
    disable_css=False,         # Keep CSS for layout
    
    # Viewport configuration
    viewport={
        "width": 1920,
        "height": 1080
    },
    
    # Timeouts
    timeout=30000,             # 30 seconds
    
    # Memory management
    headless=True
)
```

#### 3. Request Delays & Rate Limiting

```python
config = CrawlerRunConfig(
    delay=2.0,                 # 2 second delay between requests
    wait_for="networkidle"     # Wait for network idle
)
```

---

## 4. CONFIGURATION & SETUP

### Installation

```bash
# Basic installation
pip install crawl4ai

# With LLM extraction support
pip install "crawl4ai[openai]"        # For OpenAI
pip install "crawl4ai[anthropic]"     # For Claude
pip install "crawl4ai[google]"        # For Gemini

# Full installation with all features
pip install "crawl4ai[all]"

# From source
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .
```

### Core Dependencies

**Required:**
- `playwright>=1.40.0` - Browser automation
- `beautifulsoup4` - HTML parsing
- `pydantic>=2.0` - Data validation
- `markdown2` - HTML to Markdown
- `lxml` - XPath parsing
- `httpx>=0.24` - Async HTTP client

**Optional (for features):**
- `openai` - OpenAI LLM extraction
- `anthropic` - Claude LLM extraction
- `google-generativeai` - Gemini LLM extraction
- `pdf2image` - PDF processing
- `pillow` - Image processing
- `easyocr` - Optical character recognition

### Configuration Methods

#### 1. Environment Variables

```bash
# Browser settings
export CRAWL4AI_BROWSER_TYPE=chromium
export CRAWL4AI_HEADLESS=true
export CRAWL4AI_TIMEOUT=30000

# Proxy configuration
export CRAWL4AI_PROXY_URL=http://proxy:8080
export CRAWL4AI_PROXY_USERNAME=user
export CRAWL4AI_PROXY_PASSWORD=pass

# LLM API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GOOGLE_API_KEY=...

# Performance
export CRAWL4AI_POOL_SIZE=10
export CRAWL4AI_CACHE_ENABLED=true
```

#### 2. Configuration File

Create `.crawl4ai/config.yaml`:

```yaml
browser:
  type: chromium
  headless: true
  timeout: 30000
  viewport:
    width: 1920
    height: 1080
  
extraction:
  default_format: markdown
  clean_content: true
  remove_ads: true
  
performance:
  pool_size: 5
  cache_enabled: true
  cache_ttl: 3600  # 1 hour
  
proxy:
  enabled: false
  url: null
  
logging:
  level: INFO
  file: crawl4ai.log
```

#### 3. Programmatic Configuration

```python
from crawl4ai import BrowserConfig, AsyncWebCrawler

browser_config = BrowserConfig(
    headless=True,
    browser_type="chromium",
    timeout=30000,
    disable_images=True,
    proxy="http://proxy:8080",
    headers={
        "Accept-Language": "en-US",
    }
)

crawler = AsyncWebCrawler(config=browser_config)
```

### Basic Usage Patterns

#### Pattern 1: Simple Page Extraction

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        if result.status.is_success():
            print(result.markdown)
        else:
            print(f"Error: {result.status.error_message}")

asyncio.run(main())
```

#### Pattern 2: Structured Data Extraction

```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import JsonCssExtractionStrategy

async def extract_products():
    strategy = JsonCssExtractionStrategy(
        extractions=[
            {"name": "title", "css": "h2.product-title", "type": "text"},
            {"name": "price", "css": ".product-price", "type": "text"},
            {"name": "rating", "css": ".rating", "type": "text"},
            {"name": "url", "css": "a.product-link", "type": "link"}
        ]
    )
    
    config = CrawlerRunConfig(extraction_strategy=strategy)
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://store.example.com/products",
            config=config
        )
        return result.extracted_data

asyncio.run(extract_products())
```

#### Pattern 3: LLM-Powered Extraction

```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import LLMExtractionStrategy
from pydantic import BaseModel

class CompanyInfo(BaseModel):
    name: str
    industry: str
    employees: int
    founded_year: int
    description: str

async def extract_company_info():
    strategy = LLMExtractionStrategy(
        provider="openai",
        api_token="sk-...",
        schema=CompanyInfo,
        instruction="Extract comprehensive company information from the page"
    )
    
    config = CrawlerRunConfig(extraction_strategy=strategy)
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://example.com/about",
            config=config
        )
        return result.extracted_data

asyncio.run(extract_company_info())
```

#### Pattern 4: Deep Site Crawling

```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawl import BFSDeepCrawlStrategy

async def crawl_documentation():
    strategy = BFSDeepCrawlStrategy(
        max_depth=3,
        max_pages=100,
        same_domain_only=True,
        include_patterns=[r".*\/docs\/.*"],
        exclude_patterns=[r".*\.pdf$", r".*\/search.*"]
    )
    
    config = CrawlerRunConfig(
        deep_crawl_strategy=strategy,
        extraction_strategy=JsonCssExtractionStrategy(
            extractions=[
                {"name": "title", "css": "h1", "type": "text"},
                {"name": "content", "css": ".main-content", "type": "html"}
            ]
        )
    )
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://docs.example.com",
            config=config
        )
        # Result contains all crawled pages
        return result

asyncio.run(crawl_documentation())
```

#### Pattern 5: Authenticated Access

```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, BrowserProfiler

async def authenticated_crawl():
    # Step 1: Create profile (one-time)
    profiler = BrowserProfiler()
    profile_path = await profiler.create_profile("my_account")
    # User logs in manually in the browser window
    
    # Step 2: Use profile for crawling
    browser_config = BrowserConfig(
        use_managed_browser=True,
        user_data_dir=profile_path,
        browser_type="chromium"
    )
    
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun("https://app.example.com/dashboard")
        return result.markdown

asyncio.run(authenticated_crawl())
```

### Error Handling

```python
from crawl4ai import AsyncWebCrawler
from crawl4ai.models import CrawlStatus

async def safe_crawl(url):
    try:
        async with AsyncWebCrawler(timeout=30000) as crawler:
            result = await crawler.arun(url)
            
            if result.status == CrawlStatus.SUCCESS:
                return result.extracted_data
            elif result.status == CrawlStatus.TIMEOUT:
                print(f"Request timed out for {url}")
            elif result.status == CrawlStatus.FAILED:
                print(f"Request failed: {result.status.error_message}")
            
    except Exception as e:
        print(f"Exception: {e}")
```

### Performance Optimization

```python
from crawl4ai import BrowserConfig, AsyncWebCrawler

# Optimized configuration for speed
perf_config = BrowserConfig(
    headless=True,
    timeout=15000,              # Shorter timeout
    disable_images=True,        # Don't load images
    disable_css=False,          # Keep CSS for structure
    ignore_https_errors=True    # Skip SSL verification
)

# Use CSS extraction instead of LLM
# (Much faster and cheaper)
from crawl4ai.extraction import JsonCssExtractionStrategy
strategy = JsonCssExtractionStrategy(...)

# Add reasonable delays to respect servers
from crawl4ai import CrawlerRunConfig
config = CrawlerRunConfig(
    extraction_strategy=strategy,
    delay=1.0  # 1 second between requests
)
```

---

## 5. INTEGRATION WITH DATA PIPELINES

### DLT (Data Load Tool) Integration

```python
import dlt
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import JsonCssExtractionStrategy

@dlt.resource(name="articles", write_disposition="append")
async def scrape_articles():
    """DLT resource that uses Crawl4AI for scraping"""
    strategy = JsonCssExtractionStrategy(
        extractions=[
            {"name": "title", "css": "h1", "type": "text"},
            {"name": "content", "css": "article", "type": "html"},
            {"name": "published_date", "css": ".date", "type": "text"}
        ]
    )
    
    config = CrawlerRunConfig(extraction_strategy=strategy)
    
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            "https://blog.example.com",
            config=config
        )
        
        if result.status.is_success():
            yield result.extracted_data

# Create pipeline
pipeline = dlt.pipeline(
    pipeline_name="web_scraping",
    destination="duckdb",
    dataset_name="scraped_data"
)

# Run it
asyncio.run(pipeline.run(scrape_articles()))
```

### Dagster Integration

```python
from dagster import asset, Output
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

@asset
async def web_content():
    """Asset that crawls web content"""
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        
        return Output(
            value=result.extracted_data,
            metadata={
                "duration": result.duration,
                "status": result.status.status_code
            }
        )

@asset
def process_content(web_content):
    """Process crawled content"""
    # Further processing
    return processed_data
```

---

## 6. SUMMARY & KEY TAKEAWAYS

### Strengths of Crawl4AI

1. **LLM-Native Design**
   - Markdown output optimized for AI consumption
   - Supports structured extraction via Pydantic schemas
   - Cost-effective extraction strategies

2. **Dynamic Content Handling**
   - Full JavaScript rendering via Playwright
   - Custom JS execution before extraction
   - Wait conditions for async content

3. **Flexible Extraction**
   - CSS selector-based (fast, cheap)
   - LLM-powered (semantic, flexible)
   - Multiple output formats

4. **Authentication Support**
   - Persistent browser profiles
   - Programmatic login automation
   - Session/cookie management

5. **Performance**
   - Async-first design
   - Browser pooling for concurrency
   - Caching mechanisms
   - Image/CSS disabling for speed

6. **Production-Ready**
   - Type-safe configuration (Pydantic)
   - Comprehensive error handling
   - Integration with data pipelines
   - Active development and maintenance

### Best Use Cases

- Building RAG (Retrieval-Augmented Generation) knowledge bases
- Web data pipelines for AI training datasets
- Intelligent document extraction
- E-commerce product catalog scraping
- Documentation and API reference crawling
- News aggregation and content analysis
- Lead generation and market research data collection

### Integration Considerations

- Works seamlessly with dlt for data loading
- Compatible with Dagster for orchestration
- Supports multiple LLM providers
- Docker-friendly for containerization
- Can be deployed in data pipelines or standalone

---

## Additional Resources

**Official Documentation:**
- GitHub: https://github.com/unclecode/crawl4ai
- Documentation: https://docs.crawl4ai.com

**Related Tools in This Hackathon:**
- dlt: Data Load Tool for pipeline orchestration
- Dagster: Workflow orchestration
- Agno: Agent framework for AI automation
- MotherDuck/DuckDB: Analytics and SQL querying

slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall