Skip to content

lidge-jun/codexclaw / plugins/codexclaw/skills/dev-data

cxc-dev-data

Verified

Use for analytics and data pipelines. Triggers: ETL, ELT, data quality, SQL optimization, schema drift, backfill, 데이터 파이프라인, 데이터 품질, 백필.

references/streaming.md

View on GitHub
# Streaming & Event-Driven Data Patterns

**Last reviewed**: 2026-07-02

Kafka patterns, stream processing selection, and CDC integration.

---

## Kafka Configuration Essentials

### Topic Design
- Partition count = expected peak throughput ÷ single-partition throughput
- Avoid >50 partitions per topic unless proven necessary
- Use Schema Registry (Avro/Protobuf) for backwards-compatible schema evolution
- Naming: `{domain}.{entity}.{event}` (e.g., `commerce.orders.created`)

### Delivery Semantics

| Semantic | Config | Use Case |
|----------|--------|----------|
| At-most-once | `acks=0` | Metrics, logs (loss acceptable) |
| At-least-once | `acks=all` + idempotent consumer | Default for most pipelines |
| Exactly-once | `enable.idempotence=true` + transactional API | Financial, billing, inventory |

### Producer Tuning
- `batch.size`: 16-64KB for throughput (default 16KB)
- `linger.ms`: 5-50ms to allow batching
- `compression.type`: `lz4` (speed) or `zstd` (ratio)
- Always enable `enable.idempotence=true`

### Consumer Patterns
- Prefer cooperative rebalancing (`partition.assignment.strategy=cooperative-sticky`)
- Commit offsets after processing, not before
- Monitor consumer lag via Prometheus metrics

---

## Change Data Capture (CDC)

Use **Debezium** for database → Kafka streaming:

```
PostgreSQL (WAL) → Debezium Connector → Kafka Topic → Consumer/Flink/Spark
```

- Captures INSERT, UPDATE, DELETE as events with before/after state
- No application code changes needed — reads from database WAL
- Supports PostgreSQL, MySQL, MongoDB, SQL Server

---

## Stream Processing Selection

| Need | Choose |
|------|--------|
| Ultra-low latency (<100ms) + complex state | Apache Flink |
| Existing Spark infrastructure + can tolerate >1s | Spark Structured Streaming |
| Simple transforms, data stays in Kafka | Kafka Streams (no extra cluster) |
| Latency >1min acceptable | Batch with frequent scheduling |

---

## Windowing Patterns

| Window Type | Use Case | Example |
|-------------|----------|---------|
| **Tumbling** | Fixed, non-overlapping intervals | 1-minute aggregation buckets |
| **Sliding** | Overlapping intervals | 5-min window sliding every 1 min |
| **Session** | Activity-based, gap-defined | User session grouped by inactivity gap |

Handle **late data** with watermarks: define maximum acceptable lateness, drop or route late events to dead-letter.

---

## Batch vs Streaming Decision Refinement

```
Is real-time insight required?
├── Latency <100ms + complex joins/aggregations → Flink
├── Latency <1s + existing Spark cluster → Spark Structured Streaming
├── Latency <1s + Kafka-only transforms → Kafka Streams
├── Latency 1-60s + simple enrichment → Kafka consumer with micro-batch
└── Latency >60s → Batch (Airflow/Dagster scheduled jobs)
```

**Default to batch.** Only add streaming when latency requirements genuinely demand it. Streaming adds complexity in:
- State management (checkpointing, recovery)
- Error handling (dead-letter queues, poison pills)
- Testing (hard to reproduce timing-dependent bugs)
- Monitoring (consumer lag, backpressure)
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall