Skip to content

jaingxyz/aws-data-skills / skills/cdc-streaming-pipeline

cdc-streaming-pipeline

Verified

Build a real-time change-data-capture pipeline from a transactional database (Aurora DSQL, Aurora Postgres, RDS) to Redshift Serverless, S3, or any other sink, via Kinesis Data Streams (or MSK). Covers producer setup, consumer Lambda correctness, the append-only event log + reconstruction pattern, schema-drift handling with SUPER + JSON_PARSE, and the operational gotchas (Redshift Data API parameter caps, async statement polling, poison-record handling, retry-and-bisect). Use when the user asks for "CDC to Redshift", "stream change data capture", "Aurora CDC", "DSQL CDC", "Postgres replication to S3", or "build a Lambda consumer for Kinesis CDC". Aurora DSQL public-preview specifics are flagged; the rest is source-agnostic.

reference/redshift-ddl.sql

View on GitHub
-- Reference Redshift Serverless DDL for the CDC pipeline sink.
--
-- Pattern: append-only event log + materialized "current state" views.
--
-- Why this pattern?
--   * CDC delivery is UNORDERED and may DUPLICATE. Appending is safe
--     under both. Naive in-place upserts are not.
--   * In Aurora DSQL public preview, both INSERT and UPDATE arrive as
--     op="c", so we cannot distinguish them at write time. Window
--     functions on the event log dedupe by primary key + commit
--     timestamp.
--   * The SUPER type lets the same table absorb schema drift from any
--     source table without DDL changes.

-- 1. Append-only CDC event log.
-- Every Kinesis record produces one row here.
CREATE TABLE IF NOT EXISTS cdc_events (
    event_id          BIGINT IDENTITY(1,1) PRIMARY KEY,
    source_table      VARCHAR(100) NOT NULL,
    operation         VARCHAR(10)  NOT NULL,            -- "c", "u", or "d"
    record_id         VARCHAR(50)  NOT NULL,            -- source row primary key
    event_data        SUPER,                             -- full row state for c/u, PK only for d
    commit_timestamp  TIMESTAMP    NOT NULL,             -- source-side commit time
    ingested_at       TIMESTAMP    NOT NULL DEFAULT GETDATE()
)
DISTSTYLE KEY
DISTKEY (record_id)
SORTKEY (source_table, commit_timestamp);

-- Lambda's IAM-mapped database user is auto-created on first
-- redshift-serverless:GetCredentials call. Granting to PUBLIC keeps the
-- demo simple. In production, grant to a specific role created to match
-- the Lambda's IAM identity (typically "IAMR:<role-name>") and drop the
-- PUBLIC grant.
GRANT INSERT, SELECT ON cdc_events TO PUBLIC;

-- 2. Current-state view per source table.
-- Pattern: pick the latest event per record_id; treat "d" as a tombstone.
-- Replace `orders` and the projected fields with whatever the source table
-- carries. Add one of these views per source table.

CREATE OR REPLACE VIEW orders_current AS
SELECT
    record_id                            AS order_id,
    event_data."customer_id"::VARCHAR    AS customer_id,
    event_data."total_cents"::BIGINT     AS total_cents,
    event_data."status"::VARCHAR         AS status,
    event_data."payment_method"::VARCHAR AS payment_method,
    event_data."ship_country"::VARCHAR   AS ship_country,
    commit_timestamp                     AS last_change_at
FROM (
    SELECT *,
           ROW_NUMBER() OVER (
               PARTITION BY record_id
               ORDER BY commit_timestamp DESC
           ) AS rn
    FROM cdc_events
    WHERE source_table = 'orders'
)
WHERE rn = 1
  AND operation <> 'd';

GRANT SELECT ON orders_current TO PUBLIC;
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall