Skip to content

cianfhoghlaim/cianchosaint / .agents/skills/crawl4ai

crawl4ai

Verified

references/crawl4ai-openapi-research.md

View on GitHub
# Crawl4AI OpenAPI Specification Research

**Date:** 2025-11-22
**Repository:** https://github.com/unclecode/crawl4ai
**Documentation:** https://docs.crawl4ai.com/
**Stars:** 53.7k+ (as of research date)

## Executive Summary

**Does an official OpenAPI specification exist?** **YES**, but it's dynamically generated by FastAPI.

Crawl4AI is an open-source LLM-friendly web crawler that provides a comprehensive REST API built with FastAPI. While there isn't a static OpenAPI specification file committed to the repository, the FastAPI server automatically generates OpenAPI documentation that can be accessed when the service is running.

## OpenAPI Specification Access

### Method 1: Running Docker Container (Recommended)

When you run the Crawl4AI Docker container, the OpenAPI specification is automatically available:

```bash
docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest
```

**Available endpoints:**
- **OpenAPI JSON:** `http://localhost:11235/openapi.json`
- **Swagger UI (Interactive Docs):** `http://localhost:11235/docs`
- **ReDoc (Alternative Docs):** `http://localhost:11235/redoc`
- **Interactive Playground:** `http://localhost:11235/playground`

### Method 2: Third-Party Platforms

**Apify Platform:**
- Primary: https://apify.com/janbuchar/crawl4ai/api/openapi
- Alternative (AI Web Scraper): https://apify.com/raizen/ai-web-scraper/api/openapi

These are wrapper implementations that expose Crawl4AI functionality through Apify's platform.

### Method 3: Postman Collection

A community-maintained Postman collection is available:
- **Collection URL:** https://www.postman.com/pixelao/pixel-public-workspace/collection/c26yn3l/crawl4ai-api

*Note: This link returned a 503 error during research, so availability may vary.*

## OpenAPI Specification Details

### Version Information

- **OpenAPI Version:** Likely 3.0.x or 3.1.x (FastAPI supports both)
- **Crawl4AI Version:** 0.7.x (current as of research)
- **Framework:** FastAPI
- **Server Port:** 11235 (default)

### Architecture

**Repository Structure:**
```
crawl4ai/
├── deploy/docker/
│   ├── server.py          # FastAPI server with endpoint definitions
│   ├── api.py             # API endpoint handler functions
│   ├── schemas.py         # Pydantic models for request/response
│   └── config.yml         # Server configuration
```

**Key Components:**
- **FastAPI Server:** Provides automatic OpenAPI documentation
- **Redis:** Task management and caching
- **Supervisord:** Process management in Docker
- **Gunicorn:** WSGI server for production deployment

## API Endpoints Coverage

### Core Crawling Endpoints

#### POST /crawl
**Description:** Synchronous crawling of one or more URLs
**Parameters:**
- `urls` (array): URLs to crawl
- `browser_config`: Browser configuration settings
- `crawler_config`: Crawler configuration settings
- `extraction_strategy`: Strategy for content extraction
- `hooks`: Custom hook functions

**Returns:** `CrawlResult` with markdown, HTML, extracted content, and links

#### POST /crawl/stream
**Description:** Streaming crawl results in NDJSON format
**Parameters:** Same as `/crawl` with `stream: true`
**Returns:** Line-delimited JSON results as they complete

#### POST /crawl/job
**Description:** Asynchronous job submission with webhook support
**Parameters:**
- `urls`: URLs to crawl
- `cache_mode`: Caching strategy
- `extraction_strategy`: Content extraction method
- `webhook_config`: Webhook configuration for notifications

**Returns:**
```json
{
  "task_id": "crawl_1698765432",
  "message": "Crawl job submitted"
}
```

### Specialized Processing Endpoints

#### POST /html
**Description:** Crawls URL and returns preprocessed HTML optimized for schema extraction

#### POST /screenshot
**Description:** Captures full-page PNG screenshots
**Parameters:**
- `screenshot_wait_for`: Wait condition before capturing

#### POST /pdf
**Description:** Generates PDF documents from URLs

#### POST /execute_js
**Description:** Runs JavaScript snippets on pages
**Parameters:**
- `scripts`: JavaScript code to execute

#### POST /llm/job
**Description:** Asynchronous LLM extraction
**Parameters:**
- `url`: Target URL
- `q`: Query/question for LLM
- `provider`: LLM provider (e.g., OpenAI, Anthropic)
- `schema`: Expected output schema

### Job Management & Monitoring

#### GET /job/{task_id}
**Description:** Retrieve job status and results
**Path Parameters:** `task_id` - Job identifier

#### GET /health
**Description:** Server health status check

#### GET /metrics
**Description:** Prometheus-format metrics endpoint

#### GET /hooks/info
**Description:** Information about available hook points and signatures

#### GET /mcp/schema
**Description:** MCP tool schemas and parameters

### MCP (Model Context Protocol) Endpoints

#### GET /mcp/sse
**Description:** Server-Sent Events endpoint for MCP connections

#### WebSocket /mcp/ws
**Description:** WebSocket endpoint for MCP
**URL:** `ws://localhost:11235/mcp/ws`

### Configuration & Utility

#### GET /playground
**Description:** Interactive web interface for testing API requests
**Features:**
- Configure CrawlerRunConfig and BrowserConfig
- Test crawling operations directly
- Generate corresponding JSON for REST API requests

## Key Features

### 1. Dual API Approaches

**Synchronous API (`/crawl`):**
- Client waits for entire crawl to complete
- Immediate results returned in response
- Suitable for simple, quick crawls

**Asynchronous Job Queue API (`/crawl/job`, `/llm/job`):**
- Submit jobs and receive task ID
- Poll for results or use webhooks
- Real-time notifications when jobs finish
- Better for long-running operations

### 2. Webhook Support

**Introduced in v0.7.6:**
- Real-time notifications for `/crawl/job` and `/llm/job`
- Exponential backoff retry mechanism
- Custom headers support
- Flexible delivery modes

### 3. Authentication

**JWT Token Authentication:**
- Built-in security for API access
- Token-based request authorization

### 4. Content Output Formats

- **Markdown:** LLM-ready clean markdown
- **JSON:** Structured data extraction
- **HTML:** Preprocessed HTML
- **PDF:** Document generation
- **Screenshots:** PNG images

### 5. Extraction Strategies

- CSS Selector-based extraction
- XPath-based extraction
- LLM-powered extraction
- Repeated pattern parsing

### 6. Browser Control

- Hooks support
- Proxy configuration
- Stealth modes
- Session re-use
- JavaScript execution

## Configuration Format

### JSON Structure for API Calls

When calling the REST API directly, configuration objects must be converted to JSON with this pattern:

```json
{
  "type": "ClassName",
  "params": {
    "param1": "value1",
    "param2": "value2"
  }
}
```

### Example Request

```json
{
  "urls": ["https://example.com"],
  "crawler_config": {
    "type": "CrawlerRunConfig",
    "params": {
      "cache_mode": "ENABLED",
      "verbose": true
    }
  },
  "browser_config": {
    "type": "BrowserConfig",
    "params": {
      "headless": true,
      "viewport_width": 1920,
      "viewport_height": 1080
    }
  }
}
```

## Python SDK vs REST API

### Python SDK (Primary Interface)

**Installation:**
```bash
pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor  # Verify installation
```

**Usage:**
```python
from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url="https://example.com")
```

**Key Classes:**
- `AsyncWebCrawler` - Main asynchronous crawler class
- `arun()` - Primary crawling method
- `arun_many()` - Batch crawling method
- `CrawlResult` - Result object

### REST API (Docker-based)

**Best for:**
- Language-agnostic integration
- Microservices architecture
- Distributed systems
- Production deployments

## Monitoring & Observability

### Prometheus Metrics

Available at `/metrics` endpoint:
- Request counts
- Response times
- Error rates
- Browser pool status
- Queue depth

### Enterprise Features

- Comprehensive monitoring dashboard
- WebSocket streaming for real-time updates
- Smart browser pool management
- Production-ready observability

## Generating OpenAPI Specification

### Method 1: Extract from Running Server

```bash
# Start the Docker container
docker run -d -p 11235:11235 --name crawl4ai unclecode/crawl4ai:latest

# Download the OpenAPI spec
curl http://localhost:11235/openapi.json > crawl4ai-openapi.json

# Or use wget
wget http://localhost:11235/openapi.json -O crawl4ai-openapi.json
```

### Method 2: Using FastAPI CLI (if running from source)

```bash
# If running the FastAPI app directly
fastapi dev server.py --port 11235

# In another terminal
curl http://localhost:11235/openapi.json > crawl4ai-openapi.json
```

## Use Cases

### 1. RAG (Retrieval-Augmented Generation)
- Clean markdown output perfect for LLM ingestion
- Optimized for vector database storage
- Structured content extraction

### 2. AI Agents & Automation
- Asynchronous job queue for agent workflows
- Webhook notifications for event-driven architectures
- MCP protocol support

### 3. Data Pipelines
- Batch processing with `arun_many()`
- Redis-backed task management
- Configurable caching strategies

### 4. Model Training
- Structured data extraction
- Schema-based scraping
- Repeated pattern parsing

### 5. Content Analysis
- Sentiment analysis ready
- LLM-powered extraction
- Multi-format output

## Limitations & Considerations

### No Static OpenAPI File

**Why it doesn't exist in the repo:**
- FastAPI generates OpenAPI spec dynamically
- Schema evolves with code changes
- Reduces maintenance burden
- Always up-to-date with actual implementation

**How to get it:**
- Must run the server
- Download from `/openapi.json` endpoint
- Alternative: Use Apify platform versions

### Version Compatibility

- Different versions may have different endpoints
- Check release notes for API changes
- Use semantic versioning for production

### Rate Limiting

- Not explicitly documented in OpenAPI
- May vary by deployment method
- Check documentation for production limits

## Integration Patterns

### 1. Direct REST API Calls

```bash
# Synchronous crawl
curl -X POST http://localhost:11235/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com"],
    "crawler_config": {
      "type": "CrawlerRunConfig",
      "params": {"cache_mode": "ENABLED"}
    }
  }'
```

### 2. Async Job Queue

```bash
# Submit job
JOB_RESPONSE=$(curl -X POST http://localhost:11235/crawl/job \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"]}')

# Extract task_id
TASK_ID=$(echo $JOB_RESPONSE | jq -r '.task_id')

# Check status
curl http://localhost:11235/job/$TASK_ID
```

### 3. Webhook Integration

```json
{
  "urls": ["https://example.com"],
  "webhook_config": {
    "url": "https://your-server.com/webhook",
    "headers": {
      "Authorization": "Bearer your-token"
    },
    "delivery_mode": "immediate"
  }
}
```

## Comparison with Alternatives

### Crawl4AI vs Firecrawl

**Crawl4AI Advantages:**
- Open-source (no forced API keys)
- Self-hostable
- More extraction strategies
- Better LLM integration

**Firecrawl Advantages:**
- Managed cloud service
- Static OpenAPI specification
- Enterprise support

### Crawl4AI vs Traditional Scrapers

**Crawl4AI Advantages:**
- LLM-optimized output
- Markdown generation
- AI-powered extraction
- Modern async architecture

## Best Practices

### 1. Use Async Endpoints for Long Operations

```json
// Instead of /crawl for multiple URLs
// Use /crawl/job with webhooks
{
  "urls": ["url1", "url2", "url3"],
  "webhook_config": {...}
}
```

### 2. Implement Proper Error Handling

- Check job status periodically
- Handle webhook delivery failures
- Implement retry logic

### 3. Optimize Cache Strategy

```json
{
  "crawler_config": {
    "type": "CrawlerRunConfig",
    "params": {
      "cache_mode": "ENABLED",  // or "DISABLED", "READ_ONLY"
      "bypass_cache": false
    }
  }
}
```

### 4. Use Appropriate Extraction Strategy

- Simple content: CSS/XPath selectors
- Complex patterns: Repeated pattern parsing
- Intelligent extraction: LLM-based strategies

## Future Development

### Roadmap Considerations

Based on the project's active development:
- Continued FastAPI enhancements
- More MCP protocol features
- Enhanced monitoring capabilities
- Additional extraction strategies

### Community Contributions

- 5.3k+ forks indicate active community
- Regular releases and updates
- Discord community for support

## Resources & References

### Official Documentation
- **Main Documentation:** https://docs.crawl4ai.com/
- **Docker Deployment Guide:** https://docs.crawl4ai.com/core/docker-deployment/
- **Quick Start:** https://docs.crawl4ai.com/core/quickstart/
- **Code Examples:** https://docs.crawl4ai.com/core/examples/

### GitHub
- **Repository:** https://github.com/unclecode/crawl4ai
- **Releases:** https://github.com/unclecode/crawl4ai/releases
- **Issues:** https://github.com/unclecode/crawl4ai/issues
- **Discussions:** https://github.com/unclecode/crawl4ai/discussions

### Community
- **Discord:** https://discord.gg/jP8KfhDhyN
- **PyPI:** https://pypi.org/project/Crawl4AI/

### Tutorials & Guides
- [Crawl4AI Tutorial: A Beginner's Guide](https://apidog.com/blog/crawl4ai-tutorial/)
- [Crawl4AI Tutorial: Docker Deployment](https://www.pondhouse-data.com/blog/webcrawling-with-crawl4ai)
- [Crawl4AI - ScrapingBee Guide](https://www.scrapingbee.com/blog/crawl4ai/)

### Third-Party Integrations
- **Apify Platform:** https://apify.com/janbuchar/crawl4ai/api
- **MCP Integration:** https://github.com/coleam00/mcp-crawl4ai-rag
- **Community REST API:** https://github.com/romeoman/crawl4ai-rest-api

## Conclusion

### Summary

**OpenAPI Specification Status:** ✅ **Available** (dynamically generated by FastAPI)

**Access Methods:**
1. ⭐ **Recommended:** Run Docker container and access `/openapi.json`
2. Use Apify platform wrappers
3. Community Postman collections

**Coverage:** Comprehensive REST API with:
- Core crawling endpoints (sync & async)
- Specialized processing (HTML, PDF, screenshots)
- Job management and monitoring
- MCP protocol support
- Interactive playground

**Production Readiness:** ✅ High
- Docker-based deployment
- Redis task management
- Prometheus metrics
- JWT authentication
- Webhook support

### Feasibility of Generating OpenAPI Spec

**Verdict:** ✅ **Easy** - Already auto-generated by FastAPI

**Steps:**
1. Run Crawl4AI Docker container
2. Download from `http://localhost:11235/openapi.json`
3. Optionally validate and enhance with additional documentation

**No manual spec creation needed** - FastAPI handles this automatically based on:
- Pydantic models in `schemas.py`
- Endpoint decorators in `server.py`
- Type hints and docstrings

### Recommendation

For integration projects:
1. Use the Docker deployment to get the current OpenAPI spec
2. Save the spec file to your project's API documentation
3. Update periodically to track API changes
4. Consider using OpenAPI code generators for client libraries

For production use:
1. Deploy Crawl4AI using Docker
2. Implement async job queue pattern
3. Use webhooks for event-driven architectures
4. Monitor via Prometheus metrics endpoint

---

**Research completed:** 2025-11-22
**Next steps:** Extract and validate OpenAPI specification from running instance
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall