rlaope/oh-my-hermes / .claude/agent-skills/omh-data-pipelines
omh-data-pipelines
Verified[omh] Data pipeline work on an ETL or streaming job.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 8.2 KB |
| references/pipeline-method.md | markdown | 4.0 KB |
references/pipeline-method.md
View on GitHub# Pipeline Method
Load this while planning pipeline work. OMH prepares; the operator or the executor runs every job, query and check, and each step is `prepared` until its output is observed.
## 1. Idempotency: what makes a rerun safe
| Pattern | Use when | A row written twice |
| --- | --- | --- |
| natural-key upsert (`MERGE`, `INSERT ... ON CONFLICT`) | the sink row has a business key | updates in place |
| event-id dedupe within a window | events carry a stable id | dropped if inside the window; state the window |
| partition overwrite | the load is by date or hour | the partition is replaced whole |
| idempotency key on the producer | a service emits the events | the consumer ignores the repeat |
An `INSERT` or append with none of these is not idempotent. Fix the write before repairing the rows, or the repair is undone by the next rerun.
## 2. Schema change compatibility
| Change | Downstream impact |
| --- | --- |
| add a nullable column | additive; readers that select `*` may still break |
| widen a type (int to bigint) | widening; check readers with a fixed schema |
| rename or drop a column | breaking; adapt or retire every reader first |
| change a type's meaning or unit | breaking even when the type is unchanged |
| change the partition or key | breaking for every rerun and every incremental reader |
Name the deploy order with the change. For a backward-compatible change upgrade consumers before producers; for a forward-compatible one, producers first. With a schema registry, state the compatibility mode and the subject.
## 3. Replay and backfill
1. Bound the window and name the target partitions, tables or offsets.
2. Confirm the write is idempotent (section 1).
3. Pause or isolate downstream readers, or write to a staging target and swap.
4. Run the bounded window; record the observed row counts per partition.
5. Pass the quality gate (section 4), then publish or resume readers.
6. Rerun one partition and confirm nothing changes; that is the idempotency proof.
For a stream, name the consumer group and the offsets or timestamps to reset to, and whether downstream sinks dedupe by event id. Check the topic's retention against the oldest event in the window before planning the reset: a reset to a time before the earliest retained offset replays from the log start with no error, and the replay is partial. When retention does not cover the window, name the archive or source system to rebuild from instead of resetting.
## 4. Data-quality gate
| Check | Stops the load when |
| --- | --- |
| row count against the prior comparable window | outside a stated band |
| uniqueness on the sink key | any duplicate |
| null rate on required columns | above a stated threshold |
| freshness | the newest row is older than the schedule allows |
| referential match to a parent table | orphans above a stated threshold |
## 5. Lineage
Take lineage from the orchestrator or transformation graph (the dbt manifest, the Airflow DAG, an OpenLineage record) first. Readers found only by searching queries, dashboards or exports are marked as such, because a search misses what it did not index.
## 6. Late data
State the event-time column, the allowed lateness (how long a partition stays open), and what happens to a row that arrives after it: reprocess the trailing N partitions on every run, route it to a late-arrival table, or drop it with a counter. A partition overwrite silently drops rows that land after their partition closed. The quality gate's row-count band is compared at the same lateness, otherwise the count is expected to grow after the check passed.
## 7. What "exactly once" means
"Exactly once" is at-least-once delivery plus an idempotent or transactional sink. Name the sink's mechanism: a key upsert, or the offset stored in the same transaction as the write. Broker-level exactly-once covers reads and writes inside the broker; a write to a warehouse or an API is still at-least-once unless the sink supplies one of those. A claim that rests on the broker alone is recorded as at-least-once.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |