yacosta738/yacosta738.github.io / .agents/skills/cloudflare
cloudflare
FlaggedComprehensive Cloudflare platform skill covering Workers, Pages, storage (KV, D1, R2), AI (Workers AI, Vectorize, Agents SDK), networking (Tunnel, Spectrum), security (WAF, DDoS), and infrastructure-as-code (Terraform, Pulumi). Use for any Cloudflare development task. Biases towards retrieval from Cloudflare docs over pre-trained knowledge.
references/pipelines/README.md
View on GitHub# Cloudflare Pipelines
ETL streaming platform for ingesting, transforming, and loading data into R2 with SQL transformations.
## Overview
Pipelines provides:
- **Streams**: Durable event buffers (HTTP/Workers ingestion)
- **Pipelines**: SQL-based transformations
- **Sinks**: R2 destinations (Iceberg tables or Parquet/JSON files)
**Status**: Open beta (Workers Paid plan)
**Pricing**: No charge beyond standard R2 storage/operations
## Architecture
```
Data Sources → Streams → Pipelines (SQL) → Sinks → R2
↑ ↓ ↓
HTTP/Workers Transform Iceberg/Parquet
```
| Component | Purpose | Key Feature |
|-----------|---------|-------------|
| Streams | Event ingestion | Structured (validated) or unstructured |
| Pipelines | Transform with SQL | Immutable after creation |
| Sinks | Write to R2 | Exactly-once delivery |
## Quick Start
```bash
# Interactive setup (recommended)
npx wrangler pipelines setup
```
**Minimal Worker example:**
```typescript
interface Env {
STREAM: Pipeline;
}
export default {
async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
const event = { user_id: "123", event_type: "purchase", amount: 29.99 };
// Fire-and-forget pattern
ctx.waitUntil(env.STREAM.send([event]));
return new Response('OK');
}
} satisfies ExportedHandler<Env>;
```
## Which Sink Type?
```
Need SQL queries on data?
→ R2 Data Catalog (Iceberg)
✅ ACID transactions, time-travel, schema evolution
❌ More setup complexity (namespace, table, catalog token)
Just file storage/archival?
→ R2 Storage (Parquet)
✅ Simple, direct file access
❌ No built-in SQL queries
Using external tools (Spark/Athena)?
→ R2 Storage (Parquet with partitioning)
✅ Standard format, partition pruning for performance
❌ Must manage schema compatibility yourself
```
## Common Use Cases
- **Analytics pipelines**: Clickstream, telemetry, server logs
- **Data warehousing**: ETL into queryable Iceberg tables
- **Event processing**: Mobile/IoT with enrichment
- **Ecommerce analytics**: User events, purchases, views
## Reading Order
**New to Pipelines?** Start here:
1. [configuration.md](./configuration.md) - Setup streams, sinks, pipelines
2. [api.md](./api.md) - Send events, TypeScript types, SQL functions
3. [patterns.md](./patterns.md) - Best practices, integrations, complete example
4. [gotchas.md](./gotchas.md) - Critical warnings, troubleshooting
**Task-based routing:**
- Setup pipeline → [configuration.md](./configuration.md)
- Send/query data → [api.md](./api.md)
- Implement pattern → [patterns.md](./patterns.md)
- Debug issue → [gotchas.md](./gotchas.md)
## In This Reference
- [configuration.md](./configuration.md) - wrangler.jsonc bindings, schema definition, sink options, CLI commands
- [api.md](./api.md) - Pipeline binding interface, send() method, HTTP ingest, SQL function reference
- [patterns.md](./patterns.md) - Fire-and-forget, schema validation with Zod, integrations, performance tuning
- [gotchas.md](./gotchas.md) - Silent validation failures, immutable pipelines, latency expectations, limits
## See Also
- [r2](../r2/) - R2 storage backend for sinks
- [queues](../queues/) - Compare with Queues for async processing
- [workers](../workers/) - Worker runtime for event ingestion
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |