jaingxyz/aws-data-skills / skills/dsql-to-analytics-pipeline
dsql-to-analytics-pipeline
VerifiedBuild a real-time analytics pipeline downstream of Aurora DSQL using the DSQL CDC stream. Covers consumer Lambda correctness, the append-only event log + ROW_NUMBER reconstruction pattern that absorbs unordered/duplicate delivery, Redshift Data API parameter-cap chunking, async statement polling, BisectBatchOnFunctionError, MaximumRecordAgeInSeconds, DLQ wiring, and the DSQL-preview gotcha that both INSERT and UPDATE arrive as op='c'. Triggers on phrases like: DSQL to analytics, DSQL to Redshift, analyze DSQL data, real-time DSQL analytics, DSQL CDC pipeline, Aurora DSQL change data capture, stream DSQL changes, replicate DSQL, DSQL to S3, DSQL to Iceberg, DSQL Kinesis Lambda, DSQL data warehouse.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.3 KB |
| evals/evals.json | config | 12.5 KB |
| references/append-only-pattern.md | markdown | 7.0 KB |
| references/cdc-stream-setup.md | markdown | 7.1 KB |
| references/event-source-mapping.yaml | config | 4.5 KB |
| references/lambda-consumer.md | markdown | 10.9 KB |
| references/sink-redshift.md | markdown | 6.6 KB |
| references/sink-s3-iceberg.md | markdown | 5.8 KB |
references/event-source-mapping.yaml
View on GitHub# Reference: Kinesis -> Lambda EventSourceMapping for the DSQL CDC consumer.
#
# Targets Redshift Serverless via the Redshift Data API. Every parameter
# below has a reason; comments call out what each one prevents.
#
# Drop this into the Resources section of a CloudFormation template that
# also defines the Kinesis stream (CdcStream), the Lambda function
# (CdcProcessorFunction), and the Lambda execution role (LambdaExecRole).
#
# The DSQL CDC stream itself is NOT in this snippet because there is no
# CloudFormation resource type for it yet (public preview). Create it
# with `aws dsql create-stream` after this stack deploys; see
# cdc-stream-setup.md.
Resources:
CdcProcessorFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: !Sub "${ProjectName}-cdc-processor"
Runtime: python3.11
Handler: cdc_processor.lambda_handler
Role: !GetAtt LambdaExecRole.Arn
# Headroom for synchronous Redshift describe-statement polling.
# Worst case: ceil(BatchSize / ROWS_PER_CHUNK) * STATEMENT_POLL_TIMEOUT_S
# plus boto3 RTT overhead. With BatchSize=100, ROWS_PER_CHUNK=40,
# STATEMENT_POLL_TIMEOUT_S=25, that is 3 * 25 = 75s + overhead.
Timeout: 120
MemorySize: 256
Environment:
Variables:
REDSHIFT_WORKGROUP: !Ref RedshiftWorkgroup
REDSHIFT_DATABASE: dev
# 200-param Data API cap / 5 params per row = 40 rows max.
ROWS_PER_CHUNK: "40"
STATEMENT_POLL_TIMEOUT_S: "25"
Code:
ZipFile: |
# PLACEHOLDER. Real code is deployed by a follow-up script.
# Raise (not return). If Kinesis ever invokes this placeholder,
# returning 200 would silently advance the iterator past CDC
# events that never made it to Redshift. Raising forces the
# EventSourceMapping to retry and surfaces the misconfiguration
# in CloudWatch within seconds.
def lambda_handler(event, context):
raise RuntimeError(
"Placeholder Lambda code is running. "
"Deploy the real handler before enabling the event source."
)
CdcEventSource:
Type: AWS::Lambda::EventSourceMapping
Properties:
EventSourceArn: !GetAtt CdcStream.Arn
FunctionName: !Ref CdcProcessorFunction
# LATEST: start at tail. Use TRIM_HORIZON for backfill-from-stream
# scenarios. AT_TIMESTAMP if you need a specific resume point.
StartingPosition: LATEST
# BatchSize: max records delivered per invocation. The Lambda
# chunks internally to stay within Redshift Data API's
# 200-parameter limit (40 records * 5 params each). Values up to
# ~400 work but increase tail latency on partial batches.
BatchSize: 100
MaximumBatchingWindowInSeconds: 5
# Operational resilience. Bound retries on poison records so
# they do not block shard processing forever (default would
# retry until the record expires from the stream after up to
# 24h of retention).
MaximumRetryAttempts: 5
# Cap how long a record can be retried before it is moved to the
# DLQ (or dropped, if no DLQ). Without this, a record that keeps
# failing for transient reasons (Redshift unavailability, etc.)
# can hold up shard progress for the full Kinesis retention
# window (up to 24h on a Standard stream). 6h is generous for a
# CDC pipeline; tighten if you have stricter freshness SLAs.
MaximumRecordAgeInSeconds: 21600
# On error, retry with each half of the batch separately.
# Isolates poison records quickly without re-processing the
# whole batch through MaximumRetryAttempts retries.
BisectBatchOnFunctionError: true
# Send permanently-failed batches to an SQS DLQ. Each DLQ
# message is a *pointer* (stream ARN + shard ID + sequence
# numbers), not the original records. Replay requires reading
# the stream again at the recorded position. The Lambda
# execution role does not need any DLQ permission; the Lambda
# service writes to the DLQ.
DestinationConfig:
OnFailure:
Destination: !GetAtt CdcDlq.Arn
Enabled: true
# SQS DLQ for permanently-failed batches. SQS is the simplest target
# for OnFailure; SNS is also supported if you want fan-out alerting.
# Long retention so an operator can investigate at their own pace.
CdcDlq:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub "${ProjectName}-cdc-dlq"
MessageRetentionPeriod: 1209600 # 14 days, the SQS max
VisibilityTimeout: 60
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |