Skip to content

jaingxyz/aws-data-skills / skills/dsql-to-analytics-pipeline

dsql-to-analytics-pipeline

Verified

Build a real-time analytics pipeline downstream of Aurora DSQL using the DSQL CDC stream. Covers consumer Lambda correctness, the append-only event log + ROW_NUMBER reconstruction pattern that absorbs unordered/duplicate delivery, Redshift Data API parameter-cap chunking, async statement polling, BisectBatchOnFunctionError, MaximumRecordAgeInSeconds, DLQ wiring, and the DSQL-preview gotcha that both INSERT and UPDATE arrive as op='c'. Triggers on phrases like: DSQL to analytics, DSQL to Redshift, analyze DSQL data, real-time DSQL analytics, DSQL CDC pipeline, Aurora DSQL change data capture, stream DSQL changes, replicate DSQL, DSQL to S3, DSQL to Iceberg, DSQL Kinesis Lambda, DSQL data warehouse.

references/event-source-mapping.yaml

View on GitHub
# Reference: Kinesis -> Lambda EventSourceMapping for the DSQL CDC consumer.
#
# Targets Redshift Serverless via the Redshift Data API. Every parameter
# below has a reason; comments call out what each one prevents.
#
# Drop this into the Resources section of a CloudFormation template that
# also defines the Kinesis stream (CdcStream), the Lambda function
# (CdcProcessorFunction), and the Lambda execution role (LambdaExecRole).
#
# The DSQL CDC stream itself is NOT in this snippet because there is no
# CloudFormation resource type for it yet (public preview). Create it
# with `aws dsql create-stream` after this stack deploys; see
# cdc-stream-setup.md.

Resources:

  CdcProcessorFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub "${ProjectName}-cdc-processor"
      Runtime: python3.11
      Handler: cdc_processor.lambda_handler
      Role: !GetAtt LambdaExecRole.Arn
      # Headroom for synchronous Redshift describe-statement polling.
      # Worst case: ceil(BatchSize / ROWS_PER_CHUNK) * STATEMENT_POLL_TIMEOUT_S
      # plus boto3 RTT overhead. With BatchSize=100, ROWS_PER_CHUNK=40,
      # STATEMENT_POLL_TIMEOUT_S=25, that is 3 * 25 = 75s + overhead.
      Timeout: 120
      MemorySize: 256
      Environment:
        Variables:
          REDSHIFT_WORKGROUP: !Ref RedshiftWorkgroup
          REDSHIFT_DATABASE: dev
          # 200-param Data API cap / 5 params per row = 40 rows max.
          ROWS_PER_CHUNK: "40"
          STATEMENT_POLL_TIMEOUT_S: "25"
      Code:
        ZipFile: |
          # PLACEHOLDER. Real code is deployed by a follow-up script.
          # Raise (not return). If Kinesis ever invokes this placeholder,
          # returning 200 would silently advance the iterator past CDC
          # events that never made it to Redshift. Raising forces the
          # EventSourceMapping to retry and surfaces the misconfiguration
          # in CloudWatch within seconds.
          def lambda_handler(event, context):
              raise RuntimeError(
                  "Placeholder Lambda code is running. "
                  "Deploy the real handler before enabling the event source."
              )

  CdcEventSource:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      EventSourceArn: !GetAtt CdcStream.Arn
      FunctionName: !Ref CdcProcessorFunction

      # LATEST: start at tail. Use TRIM_HORIZON for backfill-from-stream
      # scenarios. AT_TIMESTAMP if you need a specific resume point.
      StartingPosition: LATEST

      # BatchSize: max records delivered per invocation. The Lambda
      # chunks internally to stay within Redshift Data API's
      # 200-parameter limit (40 records * 5 params each). Values up to
      # ~400 work but increase tail latency on partial batches.
      BatchSize: 100
      MaximumBatchingWindowInSeconds: 5

      # Operational resilience. Bound retries on poison records so
      # they do not block shard processing forever (default would
      # retry until the record expires from the stream after up to
      # 24h of retention).
      MaximumRetryAttempts: 5

      # Cap how long a record can be retried before it is moved to the
      # DLQ (or dropped, if no DLQ). Without this, a record that keeps
      # failing for transient reasons (Redshift unavailability, etc.)
      # can hold up shard progress for the full Kinesis retention
      # window (up to 24h on a Standard stream). 6h is generous for a
      # CDC pipeline; tighten if you have stricter freshness SLAs.
      MaximumRecordAgeInSeconds: 21600

      # On error, retry with each half of the batch separately.
      # Isolates poison records quickly without re-processing the
      # whole batch through MaximumRetryAttempts retries.
      BisectBatchOnFunctionError: true

      # Send permanently-failed batches to an SQS DLQ. Each DLQ
      # message is a *pointer* (stream ARN + shard ID + sequence
      # numbers), not the original records. Replay requires reading
      # the stream again at the recorded position. The Lambda
      # execution role does not need any DLQ permission; the Lambda
      # service writes to the DLQ.
      DestinationConfig:
        OnFailure:
          Destination: !GetAtt CdcDlq.Arn

      Enabled: true

  # SQS DLQ for permanently-failed batches. SQS is the simplest target
  # for OnFailure; SNS is also supported if you want fan-out alerting.
  # Long retention so an operator can investigate at their own pace.
  CdcDlq:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${ProjectName}-cdc-dlq"
      MessageRetentionPeriod: 1209600  # 14 days, the SQS max
      VisibilityTimeout: 60
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall