# Crawl4AI Codebase Analysis & Architecture Guide
## Executive Summary
Crawl4AI is a modern, open-source web scraping and content extraction framework purpose-built for AI/LLM applications. It specializes in converting dynamic web content into clean, LLM-ready structured data. The framework emphasizes performance, async-first design, and semantic understanding of web content.
**Key Characteristics:**
- Async-first Python library with Playwright/Chromium browser automation
- LLM-optimized output (Markdown, JSON) for AI applications
- Dual extraction approach: CSS selectors (fast) + LLM-powered (semantic)
- Persistent browser profiles for authenticated scraping
- Deep crawling strategies (BFS, DFS) for site-wide data extraction
- Built for integration with data pipelines (dlt, Dagster, etc.)
---
## 1. REPOSITORY STRUCTURE
### Overall Project Layout
```
crawl4ai/
├── crawl4ai/ # Core library code
│ ├── __init__.py
│ ├── async_crawler.py # Main AsyncWebCrawler class
│ ├── browser_config.py # Browser configuration
│ ├── crawler_run_config.py # Runtime configuration
│ ├── browser_profiler.py # Profile management for auth
│ ├── extraction/ # Extraction strategies
│ │ ├── extraction_strategy.py # Base class
│ │ ├── json_css_extraction.py # CSS selector extraction
│ │ ├── llm_extraction.py # LLM-powered extraction
│ │ └── markdown_generation.py # Markdown conversion
│ ├── deep_crawl/ # Deep crawling
│ │ ├── deep_crawl_strategy.py # Base strategy
│ │ ├── bfs_strategy.py # Breadth-first search
│ │ ├── dfs_strategy.py # Depth-first search
│ │ └── url_filter.py # Pattern matching
│ ├── hooks/ # Lifecycle hooks
│ │ ├── browser_hooks.py # Page lifecycle hooks
│ │ ├── auth_hooks.py # Authentication hooks
│ │ └── hook_registry.py # Hook management
│ ├── cache/ # Caching layer
│ │ ├── cache_manager.py # Cache interface
│ │ ├── memory_cache.py # In-memory cache
│ │ └── persistent_cache.py # Disk-based cache
│ ├── models/ # Data models
│ │ ├── crawl_result.py # Result container
│ │ ├── crawl_status.py # Status codes
│ │ └── browser_profile.py # Profile models
│ └── utils/ # Utilities
│ ├── markdown_utils.py # Markdown processing
│ ├── content_cleaner.py # HTML cleaning
│ ├── link_extractor.py # Link extraction
│ └── network_utils.py # Network helpers
├── docs/ # API documentation
├── tests/ # Test suite
├── examples/ # Example scripts
├── requirements.txt # Dependencies
├── setup.py / pyproject.toml # Package configuration
└── README.md # Project README
```
### Core Modules Explained
**asynccrawler.py**
- Primary entry point
- Manages browser lifecycle and pooling
- Handles request routing and result aggregation
- Implements context manager protocol
**browser_config.py**
- Browser initialization parameters
- Network configuration (proxies, headers)
- Resource optimization options
- Profile/session management settings
**crawler_run_config.py**
- Per-request configuration
- Extraction strategy selection
- Content post-processing rules
- Browser interaction parameters
**extraction/ Module**
- Pluggable extraction strategies
- CSS/XPath selector-based extraction
- LLM-powered semantic extraction
- Markdown conversion and formatting
**deep_crawl/ Module**
- Recursive website traversal
- URL discovery and filtering
- Depth/breadth-first algorithms
- Domain and pattern restrictions
---
## 2. CORE ARCHITECTURE
### High-Level Architecture Diagram
```
┌────────────────────────────────────────────────┐
│ User Code / LLM Agents / Pipelines │
└───────────────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────┐
│ AsyncWebCrawler (Main API) │
│ - arun(url, config) │
│ - Context manager support │
│ - Browser pool management │
└───┬───────────────────┬───────────┘
│ │
▼ ▼
┌──────────────┐ ┌─────────────────────┐
│BrowserConfig │ │ CrawlerRunConfig │
│ │ │ │
│- headless │ │- extraction_strategy│
│- proxies │ │- wait_for │
│- profile │ │- js_code │
│- timeout │ │- deep_crawl_strategy│
│- viewport │ │- delay │
└──────┬──────┘ └──────┬──────────────┘
│ │
└──────┬───────────┘
▼
┌─────────────────────────────────┐
│ Playwright/Chromium Browser │
│ - JavaScript rendering │
│ - DOM interaction │
│ - Network handling │
└────────────┬────────────────────┘
│
┌────────────┴──────────────┐
▼ ▼
┌──────────┐ ┌──────────────────────┐
│ Website │ │ Extraction Pipeline │
│ (HTML) │ │ │
│ │ │ 1. Extraction │
└──────────┘ │ - CSS selectors │
│ - LLM parsing │
│ │
│ 2. Post-Processing │
│ - Content clean │
│ - Link extraction │
│ - Markdown conv │
│ │
│ 3. Caching │
│ - Result storage │
└──────┬───────────────┘
▼
┌──────────────────────┐
│ CrawlResult │
│ │
│- extracted_data │
│- markdown │
│- html │
│- status │
│- metadata │
│- network_log │
└──────────────────────┘
```
### Data Flow Architecture
**1. Initialization Phase**
```
User Creates AsyncWebCrawler(config=BrowserConfig)
│
├─ Initialize browser_config
├─ Start Chromium process
├─ Load browser profile (if managed_browser=True)
├─ Initialize browser pool
└─ Ready for requests
```
**2. Request Phase**
```
await crawler.arun(url, config=CrawlerRunConfig)
│
├─ Acquire browser from pool
├─ Create new page context
├─ Navigate to URL
├─ Wait for readiness (DOMContentLoaded/wait_for)
├─ Execute hooks (on_page_load)
├─ Execute custom JS (if provided)
└─ Render complete
```
**3. Processing Phase**
```
Raw HTML/DOM
│
├─ Apply Extraction Strategy
│ ├─ CSS Selector Extraction: Parse with BeautifulSoup
│ ├─ LLM Extraction: Call LLM API with schema
│ └─ Markdown: Convert HTML to clean Markdown
│
├─ Content Post-Processing
│ ├─ Remove noise (scripts, ads, navigation)
│ ├─ Extract links and references
│ ├─ Format output
│ └─ Validate against schema
│
├─ Caching
│ └─ Store result if caching enabled
│
└─ Return CrawlResult
```
### Key Architectural Patterns
**1. Async/Concurrent Design**
- All I/O operations are async
- Browser pool for concurrent requests
- Non-blocking request handling
- Handles hundreds of concurrent crawls
**2. Strategy Pattern (Extraction)**
- Base `ExtractionStrategy` interface
- Multiple implementations:
- `JsonCssExtractionStrategy`
- `LLMExtractionStrategy`
- `MarkdownExtractionStrategy`
- User-selectable at runtime
**3. Strategy Pattern (Deep Crawling)**
- Base `DeepCrawlStrategy` interface
- BFS and DFS implementations
- Configurable URL filtering
- Depth and page limits
**4. Hook/Event System**
- Pre/post request hooks
- Page lifecycle hooks
- Authentication hooks
- Extensible plugin system
**5. Configuration Objects**
- `BrowserConfig`: Persistent, reusable
- `CrawlerRunConfig`: Per-request customization
- Separation of concerns
- Type-safe (Pydantic models)
---
## 3. KEY FEATURES
### A. Content Extraction Capabilities
#### 1. CSS Selector Extraction (Fast & Cheap)
```python
from crawl4ai.extraction import JsonCssExtractionStrategy
strategy = JsonCssExtractionStrategy(
extractions=[
{
"name": "title",
"css": "h1.article-title",
"type": "text" # text | link | html | attribute | list
},
{
"name": "date",
"css": ".publish-date",
"type": "attribute",
"attribute": "data-timestamp"
},
{
"name": "content",
"css": "article.body",
"type": "html"
},
{
"name": "links",
"css": "a.related-link",
"type": "list", # Returns array
"fields": {
"text": {"type": "text"},
"href": {"type": "attribute", "attribute": "href"}
}
}
]
)
```
**Extraction Types:**
- `text`: Extract textContent
- `html`: Extract innerHTML
- `attribute`: Extract specific attribute
- `link`: Extract href and text
- `list`: Extract multiple elements as array
**Advantages:**
- Zero API cost
- Deterministic results
- Millisecond execution
- Works with static and dynamic content
- XPath support via lxml
**Best For:**
- Product e-commerce pages
- News articles with consistent structure
- Tables and data listings
- Directory sites
- Any well-structured HTML
#### 2. LLM-Powered Extraction (Semantic)
```python
from crawl4ai.extraction import LLMExtractionStrategy
from pydantic import BaseModel, Field
class Product(BaseModel):
name: str = Field(description="Product name")
price: float = Field(description="Price in USD")
rating: float = Field(ge=0, le=5, description="1-5 star rating")
in_stock: bool = Field(description="Availability")
description: str = Field(description="Product description")
strategy = LLMExtractionStrategy(
provider="openai", # openai | anthropic | gemini | etc.
api_token="sk-...",
schema=Product,
instruction="Extract complete product details from the page. Be thorough and accurate.",
temperature=0.1, # Low temperature for consistency
)
```
**Features:**
- Pydantic schema support for type validation
- Multiple LLM provider support
- Natural language instructions
- Chunk content for token limits
- Retry logic for failures
- Schema validation
**Advantages:**
- Handles unstructured/inconsistent HTML
- Semantic understanding
- Flexible instructions
- Handles complex layouts
- Context-aware extraction
**Trade-offs:**
- LLM API costs
- Latency (1-5 seconds per page)
- Model dependency
- Rate limiting considerations
**Best For:**
- Unstructured content
- Multiple layout variations
- Natural language extraction
- Complex business logic
- Content summarization
#### 3. Markdown Conversion
**Automatically converts HTML to clean Markdown:**
- Preserves document hierarchy
- Removes noise (scripts, styles, ads)
- Converts links to citations: `[text][1]` + `[1]: url`
- Tables to Markdown format
- Code blocks with language hints
- Headers, lists, emphasis preserved
- Absolute URLs for all links
**Example Transformation:**
```html
<!-- Input HTML -->
<article>
<h1>Article Title</h1>
<span class="date">2024-01-15</span>
<p>Introduction paragraph.</p>
<h2>Section</h2>
<p>Content with <b>bold</b> and <i>italic</i>.</p>
<script>// noise removed</script>
<a href="/page">Related</a>
</article>
```
```markdown
# Article Title
**Published:** 2024-01-15
Introduction paragraph.
## Section
Content with **bold** and *italic*.
- [Related][1]
[1]: https://example.com/page
```
**Output Benefits:**
- LLM-friendly format
- Human-readable
- Preserves semantic structure
- Reduces token usage in LLMs
- Perfect for RAG pipelines
### B. Dynamic Content & Interaction
#### 1. JavaScript Execution
Execute custom JS before extraction:
```python
config = CrawlerRunConfig(
js_code="""
// Infinite scroll handling
async function autoScroll() {
let lastHeight = document.body.scrollHeight;
while(true) {
window.scrollTo(0, document.body.scrollHeight);
await new Promise(resolve => setTimeout(resolve, 1000));
let newHeight = document.body.scrollHeight;
if(newHeight === lastHeight) break;
lastHeight = newHeight;
}
}
await autoScroll();
// Click load more buttons
document.querySelectorAll('.load-more').forEach(btn => btn.click());
// Extract custom data
window.pageData = {
title: document.title,
items: Array.from(document.querySelectorAll('.item'))
.map(el => ({
name: el.querySelector('.name').textContent,
price: el.querySelector('.price').textContent
}))
};
"""
)
```
**Capabilities:**
- Full DOM manipulation
- Async/await support
- Event triggering
- Custom data extraction
- State management
**Use Cases:**
- Infinite scroll pages
- Single-page apps (React, Vue, Angular)
- AJAX-loaded content
- Form submission
- Cookie/banner dismissal
- Dynamic page rendering
#### 2. Wait Conditions
```python
config = CrawlerRunConfig(
# Option 1: Wait for specific element
wait_for=".article-content",
# Option 2: Wait for network idle
wait_for="networkidle",
# Option 3: Wait for page load
wait_for="load",
# Combined with timeout
# wait_timeout=10000 # 10 seconds
)
```
**Wait Types:**
- CSS selector: Waits for element to be visible
- "load": Page load event
- "networkidle": No pending network requests
- "domcontentloaded": DOM ready event
#### 3. Session/State Management
Maintain authentication across requests:
```python
# First request: login
result1 = await crawler.arun(
"https://example.com/login",
config=CrawlerRunConfig(
js_code="""
document.querySelector('input[name="email"]').value = "user@example.com";
document.querySelector('input[name="password"]').value = "password";
document.querySelector('form').submit();
""",
session_id="user_session",
wait_for=".dashboard"
)
)
# Second request: reuse session
result2 = await crawler.arun(
"https://example.com/protected",
config=CrawlerRunConfig(
session_id="user_session"
)
)
```
**Features:**
- Cookie persistence
- LocalStorage/SessionStorage preservation
- DOM state retention
- CSRF token handling
- Multiple concurrent sessions
### C. Authentication & Identity
#### 1. Browser Profile Management
```python
from crawl4ai import BrowserProfiler, BrowserConfig
# Step 1: Create profile with manual login
profiler = BrowserProfiler()
profile_path = await profiler.create_profile(
profile_name="my_account",
base_url="https://example.com"
)
# User logs in manually in browser
# Profile saves automatically
# Step 2: Reuse profile in future crawls
browser_config = BrowserConfig(
use_managed_browser=True,
user_data_dir=profile_path,
browser_type="chromium"
)
crawler = AsyncWebCrawler(config=browser_config)
result = await crawler.arun("https://example.com/protected-page")
```
**Profile Storage Includes:**
- Cookies and session storage
- LocalStorage data
- IndexedDB
- Service worker cache
- Cached credentials
- Browsing history
**Advantages:**
- One-time manual login
- Handles complex auth (2FA, CAPTCHA)
- Session preservation
- Appears as real user
- Bot-detection resistant
#### 2. Programmatic Authentication via Hooks
```python
async def auto_login(page, context, **kwargs):
"""Hook function executed before page load"""
# Navigate to login page
await page.goto("https://example.com/login")
# Fill and submit form
await page.fill('input[name="username"]', "user@example.com")
await page.fill('input[name="password"]', "secure_password")
await page.click('button:has-text("Login")')
# Wait for redirect
await page.wait_for_url("**/dashboard")
return page
browser_config = BrowserConfig(
hooks={"on_page_context_created": auto_login}
)
crawler = AsyncWebCrawler(config=browser_config)
result = await crawler.arun("https://example.com/protected")
```
### D. Deep Crawling / Site-Wide Scraping
#### 1. Breadth-First Search (BFS)
```python
from crawl4ai.deep_crawl import BFSDeepCrawlStrategy
strategy = BFSDeepCrawlStrategy(
max_depth=3, # 3 levels deep
max_pages=500, # Max 500 pages
same_domain_only=True, # Stay on domain
include_patterns=[
".*\/docs\/.*", # Include only /docs/*
".*\/api\/.*"
],
exclude_patterns=[
".*\/admin\/.*", # Skip /admin/*
".*\.pdf$", # Skip PDFs
".*\/search\?.*" # Skip search pages
],
max_retries=3, # Retry failed pages
delay=1.0 # 1 second delay
)
config = CrawlerRunConfig(deep_crawl_strategy=strategy)
result = await crawler.arun("https://docs.example.com", config=config)
```
**BFS Characteristics:**
- Level-by-level exploration
- Systematic coverage
- Finds shortest paths to pages
- Good for breadth-focused crawling
- Memory-efficient discovery
#### 2. Depth-First Search (DFS)
```python
from crawl4ai.deep_crawl import DFSDeepCrawlStrategy
strategy = DFSDeepCrawlStrategy(
max_depth=10, # Follow deep paths
max_pages=1000,
same_domain_only=True,
discovery_type="auto", # auto | manual
max_retries=2
)
config = CrawlerRunConfig(deep_crawl_strategy=strategy)
result = await crawler.arun("https://docs.example.com", config=config)
```
**DFS Characteristics:**
- Deep path exploration
- Follows links exhaustively
- Finds all linked content
- Memory usage grows with depth
- Good for complete coverage
**Configuration Parameters:**
- `max_depth`: Maximum link depth
- `max_pages`: Total page limit
- `same_domain_only`: Domain restriction
- `include_patterns`: Regex for included URLs
- `exclude_patterns`: Regex for excluded URLs
- `max_retries`: Failure retry attempts
- `delay`: Delay between requests
### E. Browser Automation & Control
#### 1. Network Configuration
```python
browser_config = BrowserConfig(
# Proxy settings
proxy="http://proxy.company.com:8080",
# Custom headers
headers={
"Accept-Language": "en-US,en;q=0.9",
"User-Agent": "Custom Agent/1.0",
"Referer": "https://example.com"
},
# Extra HTTP headers
extra_http_headers={
"X-Custom-Header": "value"
},
# SSL/TLS
ignore_https_errors=True
)
```
#### 2. Browser Resource Management
```python
browser_config = BrowserConfig(
# Performance optimization
disable_images=True, # Don't load images
disable_css=False, # Keep CSS for layout
# Viewport configuration
viewport={
"width": 1920,
"height": 1080
},
# Timeouts
timeout=30000, # 30 seconds
# Memory management
headless=True
)
```
#### 3. Request Delays & Rate Limiting
```python
config = CrawlerRunConfig(
delay=2.0, # 2 second delay between requests
wait_for="networkidle" # Wait for network idle
)
```
---
## 4. CONFIGURATION & SETUP
### Installation
```bash
# Basic installation
pip install crawl4ai
# With LLM extraction support
pip install "crawl4ai[openai]" # For OpenAI
pip install "crawl4ai[anthropic]" # For Claude
pip install "crawl4ai[google]" # For Gemini
# Full installation with all features
pip install "crawl4ai[all]"
# From source
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai
pip install -e .
```
### Core Dependencies
**Required:**
- `playwright>=1.40.0` - Browser automation
- `beautifulsoup4` - HTML parsing
- `pydantic>=2.0` - Data validation
- `markdown2` - HTML to Markdown
- `lxml` - XPath parsing
- `httpx>=0.24` - Async HTTP client
**Optional (for features):**
- `openai` - OpenAI LLM extraction
- `anthropic` - Claude LLM extraction
- `google-generativeai` - Gemini LLM extraction
- `pdf2image` - PDF processing
- `pillow` - Image processing
- `easyocr` - Optical character recognition
### Configuration Methods
#### 1. Environment Variables
```bash
# Browser settings
export CRAWL4AI_BROWSER_TYPE=chromium
export CRAWL4AI_HEADLESS=true
export CRAWL4AI_TIMEOUT=30000
# Proxy configuration
export CRAWL4AI_PROXY_URL=http://proxy:8080
export CRAWL4AI_PROXY_USERNAME=user
export CRAWL4AI_PROXY_PASSWORD=pass
# LLM API keys
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GOOGLE_API_KEY=...
# Performance
export CRAWL4AI_POOL_SIZE=10
export CRAWL4AI_CACHE_ENABLED=true
```
#### 2. Configuration File
Create `.crawl4ai/config.yaml`:
```yaml
browser:
type: chromium
headless: true
timeout: 30000
viewport:
width: 1920
height: 1080
extraction:
default_format: markdown
clean_content: true
remove_ads: true
performance:
pool_size: 5
cache_enabled: true
cache_ttl: 3600 # 1 hour
proxy:
enabled: false
url: null
logging:
level: INFO
file: crawl4ai.log
```
#### 3. Programmatic Configuration
```python
from crawl4ai import BrowserConfig, AsyncWebCrawler
browser_config = BrowserConfig(
headless=True,
browser_type="chromium",
timeout=30000,
disable_images=True,
proxy="http://proxy:8080",
headers={
"Accept-Language": "en-US",
}
)
crawler = AsyncWebCrawler(config=browser_config)
```
### Basic Usage Patterns
#### Pattern 1: Simple Page Extraction
```python
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com")
if result.status.is_success():
print(result.markdown)
else:
print(f"Error: {result.status.error_message}")
asyncio.run(main())
```
#### Pattern 2: Structured Data Extraction
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import JsonCssExtractionStrategy
async def extract_products():
strategy = JsonCssExtractionStrategy(
extractions=[
{"name": "title", "css": "h2.product-title", "type": "text"},
{"name": "price", "css": ".product-price", "type": "text"},
{"name": "rating", "css": ".rating", "type": "text"},
{"name": "url", "css": "a.product-link", "type": "link"}
]
)
config = CrawlerRunConfig(extraction_strategy=strategy)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://store.example.com/products",
config=config
)
return result.extracted_data
asyncio.run(extract_products())
```
#### Pattern 3: LLM-Powered Extraction
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import LLMExtractionStrategy
from pydantic import BaseModel
class CompanyInfo(BaseModel):
name: str
industry: str
employees: int
founded_year: int
description: str
async def extract_company_info():
strategy = LLMExtractionStrategy(
provider="openai",
api_token="sk-...",
schema=CompanyInfo,
instruction="Extract comprehensive company information from the page"
)
config = CrawlerRunConfig(extraction_strategy=strategy)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://example.com/about",
config=config
)
return result.extracted_data
asyncio.run(extract_company_info())
```
#### Pattern 4: Deep Site Crawling
```python
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.deep_crawl import BFSDeepCrawlStrategy
async def crawl_documentation():
strategy = BFSDeepCrawlStrategy(
max_depth=3,
max_pages=100,
same_domain_only=True,
include_patterns=[r".*\/docs\/.*"],
exclude_patterns=[r".*\.pdf$", r".*\/search.*"]
)
config = CrawlerRunConfig(
deep_crawl_strategy=strategy,
extraction_strategy=JsonCssExtractionStrategy(
extractions=[
{"name": "title", "css": "h1", "type": "text"},
{"name": "content", "css": ".main-content", "type": "html"}
]
)
)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://docs.example.com",
config=config
)
# Result contains all crawled pages
return result
asyncio.run(crawl_documentation())
```
#### Pattern 5: Authenticated Access
```python
from crawl4ai import AsyncWebCrawler, BrowserConfig, BrowserProfiler
async def authenticated_crawl():
# Step 1: Create profile (one-time)
profiler = BrowserProfiler()
profile_path = await profiler.create_profile("my_account")
# User logs in manually in the browser window
# Step 2: Use profile for crawling
browser_config = BrowserConfig(
use_managed_browser=True,
user_data_dir=profile_path,
browser_type="chromium"
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun("https://app.example.com/dashboard")
return result.markdown
asyncio.run(authenticated_crawl())
```
### Error Handling
```python
from crawl4ai import AsyncWebCrawler
from crawl4ai.models import CrawlStatus
async def safe_crawl(url):
try:
async with AsyncWebCrawler(timeout=30000) as crawler:
result = await crawler.arun(url)
if result.status == CrawlStatus.SUCCESS:
return result.extracted_data
elif result.status == CrawlStatus.TIMEOUT:
print(f"Request timed out for {url}")
elif result.status == CrawlStatus.FAILED:
print(f"Request failed: {result.status.error_message}")
except Exception as e:
print(f"Exception: {e}")
```
### Performance Optimization
```python
from crawl4ai import BrowserConfig, AsyncWebCrawler
# Optimized configuration for speed
perf_config = BrowserConfig(
headless=True,
timeout=15000, # Shorter timeout
disable_images=True, # Don't load images
disable_css=False, # Keep CSS for structure
ignore_https_errors=True # Skip SSL verification
)
# Use CSS extraction instead of LLM
# (Much faster and cheaper)
from crawl4ai.extraction import JsonCssExtractionStrategy
strategy = JsonCssExtractionStrategy(...)
# Add reasonable delays to respect servers
from crawl4ai import CrawlerRunConfig
config = CrawlerRunConfig(
extraction_strategy=strategy,
delay=1.0 # 1 second between requests
)
```
---
## 5. INTEGRATION WITH DATA PIPELINES
### DLT (Data Load Tool) Integration
```python
import dlt
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.extraction import JsonCssExtractionStrategy
@dlt.resource(name="articles", write_disposition="append")
async def scrape_articles():
"""DLT resource that uses Crawl4AI for scraping"""
strategy = JsonCssExtractionStrategy(
extractions=[
{"name": "title", "css": "h1", "type": "text"},
{"name": "content", "css": "article", "type": "html"},
{"name": "published_date", "css": ".date", "type": "text"}
]
)
config = CrawlerRunConfig(extraction_strategy=strategy)
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
"https://blog.example.com",
config=config
)
if result.status.is_success():
yield result.extracted_data
# Create pipeline
pipeline = dlt.pipeline(
pipeline_name="web_scraping",
destination="duckdb",
dataset_name="scraped_data"
)
# Run it
asyncio.run(pipeline.run(scrape_articles()))
```
### Dagster Integration
```python
from dagster import asset, Output
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
@asset
async def web_content():
"""Asset that crawls web content"""
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com")
return Output(
value=result.extracted_data,
metadata={
"duration": result.duration,
"status": result.status.status_code
}
)
@asset
def process_content(web_content):
"""Process crawled content"""
# Further processing
return processed_data
```
---
## 6. SUMMARY & KEY TAKEAWAYS
### Strengths of Crawl4AI
1. **LLM-Native Design**
- Markdown output optimized for AI consumption
- Supports structured extraction via Pydantic schemas
- Cost-effective extraction strategies
2. **Dynamic Content Handling**
- Full JavaScript rendering via Playwright
- Custom JS execution before extraction
- Wait conditions for async content
3. **Flexible Extraction**
- CSS selector-based (fast, cheap)
- LLM-powered (semantic, flexible)
- Multiple output formats
4. **Authentication Support**
- Persistent browser profiles
- Programmatic login automation
- Session/cookie management
5. **Performance**
- Async-first design
- Browser pooling for concurrency
- Caching mechanisms
- Image/CSS disabling for speed
6. **Production-Ready**
- Type-safe configuration (Pydantic)
- Comprehensive error handling
- Integration with data pipelines
- Active development and maintenance
### Best Use Cases
- Building RAG (Retrieval-Augmented Generation) knowledge bases
- Web data pipelines for AI training datasets
- Intelligent document extraction
- E-commerce product catalog scraping
- Documentation and API reference crawling
- News aggregation and content analysis
- Lead generation and market research data collection
### Integration Considerations
- Works seamlessly with dlt for data loading
- Compatible with Dagster for orchestration
- Supports multiple LLM providers
- Docker-friendly for containerization
- Can be deployed in data pipelines or standalone
---
## Additional Resources
**Official Documentation:**
- GitHub: https://github.com/unclecode/crawl4ai
- Documentation: https://docs.crawl4ai.com
**Related Tools in This Hackathon:**
- dlt: Data Load Tool for pipeline orchestration
- Dagster: Workflow orchestration
- Agno: Agent framework for AI automation
- MotherDuck/DuckDB: Analytics and SQL querying