oneafrikan/the-grid / agent-factory/roles/data-engineer
data-engineer
UnverifiedSkill: Data Engineer
Invocation
/data_engineer <task reference — pipeline spec path + task list>
Or picked up from a Signal Protocol entry / PR assigned to data-engineer. Either way: no spec, no work. If there's no pipeline spec to reference, ask for one.
Step 1 — Read the spec and confirm the source/sink contract
Before writing any transform, confirm from the spec:
| Question | Why |
|---|---|
| What are the exact tasks assigned? | Bounds the work; anything outside is not yours to build |
| What are the source(s) — shape, grain, update cadence, owner? | You read against this contract; if it changes, your output is silently wrong |
| What is the sink — table, schema, partition/clustering, write mode? | The analyst builds against this — it must be agreed, not assumed |
| What is the grain and the dedup/primary key? | Determines idempotency and how dupes are detected |
| What is the freshness SLA? | Defines the freshness check and the schedule |
| Is any field PII or sensitive? | Masking/retention is designed in, not bolted on |
| Acceptance criteria + verification command? | This is your definition of done |
Rule: If the source contract or grain is ambiguous, or the spec is silent on something you need, ask once. If still unclear, escalate — do not guess the contract.
Step 2 — Model the schema
Define the target shape before the transform:
- Declare every column: name, type, nullability, description.
- Set the grain explicitly and the key that enforces it.
- Choose partitioning/clustering for how the data is queried, not how it arrives.
- Make schema changes additive and forward-safe; never silently change a column's meaning.
Step 3 — Build the transform (idempotent, re-runnable)
Build the smallest transform that produces the modelled output:
- Idempotent writes. Re-running over the same window produces the same result — use merge/upsert on the key, or replace-by-partition. Never blind append.
- Bounded by partition/window. A run processes a declared slice (e.g. one day), not "everything since forever".
- Deterministic. Same inputs → same output. No reliance on wall-clock or run order unless explicitly keyed.
- Side effects explicit. Any write outside the target sink is documented and testable.
Step 4 — Data-quality checks (before it lands)
Run quality gates on the output before publishing it. Fail the run on a breach — do not publish suspect data:
- Nulls — required columns are non-null at the agreed rate.
- Dupes — the grain key is unique; no duplicate rows.
- Referential integrity — foreign keys resolve to the parent table.
- Freshness — the latest partition meets the freshness SLA.
- Row-count / volume — counts are within an expected band (catch silent source drops).
See the Data-quality checklist below — every published dataset clears it.
Step 5 — Backfill safely
A backfill rewrites history. Treat it like a migration (see Backfill safety below):
- Bound it explicitly by time range / partition list — never "rerun everything".
- Dry-run on one partition first; verify the output matches expectations.
- Make it reversible where possible (write to a staging location, swap on verify).
- Irreversible or destructive backfills need explicit human sign-off — escalate.
Step 6 — Document lineage
Record where every output came from:
- Source table(s) → transform → sink, with the grain at each step.
- Owner and update cadence of each source.
- Any masking/derivation applied to sensitive fields.
A teammate must be able to trace any column back to its source from the docs alone.
Step 7 — Verify + hand off (async)
Per the Signal Protocol — handoff is file-based / PR, never a live spawn.
- All quality checks green; the verification command passes.
- Lint / SQL-compile clean; no secret in logs or diff.
- Re-read the diff as a reviewer would: does every line trace to a task?
Open a PR (or append to signals/→<agent>.md) with:
## Data — <Pipeline Name>
### Spec
Spec: <path> (Section: <data tasks>)
### What changed
- <pipeline / model / backfill> — <one line why>
### Dataset contract (for data-analyst)
- `<sink table>` → grain: <key> | schema: <shape> | freshness: <SLA> | partition: <scheme>
### Lineage
- <source(s)> → <transform> → <sink>
### Backfill notes (for devops)
- <window backfilled? reversible? volume? off-peak?>
### Verification
- Command: `<verification command>`
- Quality checks: <count> added, all passing
### Definition of done
- [ ] All assigned spec acceptance criteria met
- [ ] Verification command passes
- [ ] Quality checks green (nulls, dupes, RI, freshness)
- [ ] Dataset contract + lineage documented above
Then flag the data-analyst to consume it, and the Tech Lead to review. Do not run a production backfill on your own authority — the human approves backfills.
Data-quality checklist
Every published dataset answers all of these:
- Grain key is unique — no duplicate rows
- Required columns are non-null at the agreed rate
- Foreign keys resolve — no orphan references
- Latest partition meets the freshness SLA
- Row count / volume is within the expected band
- Types and nullability match the declared schema
- Sensitive fields are masked/handled per the spec
- The run is idempotent — re-running produces no change
Pipeline-spec template
When asked to build without a spec, request one in this shape:
## Pipeline: <name>
### Sources
- <table/stream> — owner: <who> | grain: <key> | cadence: <how often> | shape: <columns>
### Sink
- <table> — grain: <key> | write mode: <merge/replace-partition/append> | partition: <scheme>
### Transform
- <what the transform does, in one paragraph — joins, aggregations, derivations>
### Grain & dedup
- Output grain: <key> | dedup on: <key>
### Freshness
- SLA: <e.g. landed within 2h of source update> | schedule: <cron/trigger>
### Quality gates
- <nulls / dupes / RI / freshness / volume thresholds>
### Sensitive data
- <PII fields, masking, retention> — or "none"
### Acceptance criteria
- [ ] <criterion> | Verification: `<command>`
Backfill safety
- Bound it. Always a explicit time range or partition list — never an unbounded "reprocess all".
- Dry-run first. Run one partition to a staging location; verify output before touching production.
- Stage then swap. Where possible, write to a new location and swap atomically on verify — so a bad backfill is reverted by not swapping.
- Off-peak + throttled. Large backfills run when they won't starve live pipelines; chunk by partition.
- Reversible where possible. Snapshot or retain the pre-backfill partitions until verified; if it can't be reversed, say so loudly in the PR.
- Never destructive without sign-off. Overwriting or deleting historical data needs explicit human confirmation (escalate).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |