jaingxyz/aws-data-skills / skills/streaming-into-data-lake
streaming-into-data-lake
VerifiedStream rows continuously into Apache Iceberg tables on S3 Tables (or standard Iceberg on a general purpose bucket) using Amazon Data Firehose with IcebergDestinationConfiguration. Covers the Firehose stream, the optional record-shaping Lambda (ProcessingConfiguration), the Lake Formation grants required for Firehose to write through Glue, and the three-phase CloudFormation deploy that gates the stream on grants existing. Triggers on: stream into data lake, Firehose to S3 Tables, Firehose to Iceberg, real-time ingestion to Iceberg, Firehose IcebergDestinationConfiguration, stream Kinesis to Iceberg, near real-time CDC to data lake, continuous append to Iceberg. Do NOT use for: batch file imports (use ingesting-into-data-lake), creating the destination Iceberg table itself (use creating-data-lake-table), the one-time Lake Formation onboarding (use setting-up-lake-formation), finding tables by fuzzy name (use finding-data-lake-assets), running queries (use querying-data-lake), Glue connections (Firehose does not use them; do not invoke connecting-to-data-source), Kinesis to Lambda to Redshift Data API direct writes (not Iceberg; recommend a different workflow), or producer-side Kafka or MSK setup.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.0 KB |
| evals/baseline.json | config | 85 B |
| evals/evals.json | config | 15.0 KB |
| references/error-bucket-decoding.md | markdown | 6.2 KB |
| references/firehose-iceberg-config.md | markdown | 5.1 KB |
| references/lake-formation-grants.md | markdown | 5.4 KB |
| references/three-phase-deploy.md | markdown | 5.6 KB |
| references/transform-lambda.md | markdown | 6.4 KB |
references/firehose-iceberg-config.md
View on GitHub# Firehose IcebergDestinationConfiguration
The `IcebergDestinationConfiguration` block on `AWS::KinesisFirehose::DeliveryStream` is the contract between Firehose and the Iceberg table. Every field below is load-bearing.
## Minimum viable block
```yaml
IcebergDestinationConfiguration:
RoleARN: !GetAtt FirehoseRole.Arn
CatalogConfiguration:
CatalogARN: !Sub "arn:${AWS::Partition}:glue:${AWS::Region}:${AWS::AccountId}:catalog/s3tablescatalog/${TableBucketName}"
DestinationTableConfigurationList:
- DestinationDatabaseName: !Ref Namespace
DestinationTableName: !Ref TableName
UniqueKeys: [record_id]
BufferingHints:
SizeInMBs: 64
IntervalInSeconds: 60
S3Configuration:
BucketARN: !GetAtt ErrorBucket.Arn
RoleARN: !GetAtt FirehoseRole.Arn
Prefix: !Sub "errors/${StreamName}/"
ErrorOutputPrefix: !Sub "errors/${StreamName}/iceberg-failed/"
ProcessingConfiguration:
Enabled: true
Processors:
- Type: Lambda
Parameters:
- ParameterName: LambdaArn
ParameterValue: !GetAtt TransformLambda.Arn
- ParameterName: BufferSizeInMBs
ParameterValue: "1"
- ParameterName: BufferIntervalInSeconds
ParameterValue: "60"
```
## CatalogConfiguration
| Target | `CatalogARN` value |
|---|---|
| S3 Tables (federated catalog) | `arn:aws:glue:<region>:<account>:catalog/s3tablescatalog/<table-bucket-name>` |
| Standard Iceberg on a general purpose bucket | `arn:aws:glue:<region>:<account>:catalog` (the default Glue catalog) |
The federated catalog ARN MUST exist as a registered catalog before Firehose creates the stream. Verify by listing databases inside it: `aws glue get-databases --catalog-id "<account-id>:s3tablescatalog/<bucket>"` (use `aws sts get-caller-identity --query Account --output text` to source the account-id prefix). The call returns `EntityNotFoundException` if the catalog is unregistered. Otherwise the stream creation fails with `Iceberg.GlueTableNotFound` even though the underlying table exists. (`aws glue get-catalog` is not a subcommand on most installed AWS CLI versions; `get-databases` is the reliable existence probe.)
## DestinationTableConfigurationList
- `DestinationDatabaseName` is the Glue database (the S3 Tables namespace, for S3 Tables targets).
- `DestinationTableName` is the table name. Lowercase only.
- `UniqueKeys` is required when records carry `operation = update` or `operation = delete`. List the column or columns that uniquely identify a row. For pure `insert` workloads, `UniqueKeys` may be omitted.
You can list multiple `DestinationTableConfigurationList` entries to fan one stream into multiple tables. Firehose routes each record to the entry whose `DestinationDatabaseName` and `DestinationTableName` match the per-record metadata fields the transform Lambda emits. Cover this in [`transform-lambda.md`](transform-lambda.md) under per-record routing.
## S3Configuration (error bucket)
Mandatory. Even with no Iceberg failures expected, Firehose requires an S3 fallback. `ErrorOutputPrefix` controls where failed records land: `errors/<stream-name>/iceberg-failed/`. Decoding the contents is covered in [`error-bucket-decoding.md`](error-bucket-decoding.md).
The Firehose role MUST hold `s3:PutObject`, `s3:GetBucketLocation`, and `s3:ListBucket` on the error bucket.
## ProcessingConfiguration
Set `Enabled: true` only when a transform Lambda is needed (see SKILL.md Phase 2). The Lambda ARN goes in the `LambdaArn` parameter. `BufferSizeInMBs` and `BufferIntervalInSeconds` apply to the Lambda invocation buffer, not the Iceberg buffer. Keep both small (1 MB, 60 s) to limit blast radius when the Lambda is misbehaving.
## lambda:InvokeFunction grant
The Firehose role MUST hold `lambda:InvokeFunction` on the transform Lambda ARN. Add this to the role inline policy alongside the Iceberg, S3, and Lake Formation grants:
```yaml
- Effect: Allow
Action: lambda:InvokeFunction
Resource: !GetAtt TransformLambda.Arn
```
Without this, Firehose fails the entire batch with `Lambda.InvokeAccessDenied` and CloudWatch reports `ExecuteProcessingFailure.Records` matching the batch size.
## BufferingHints
- `SizeInMBs`: 1 to 128. Default 64 is fine for most workloads.
- `IntervalInSeconds`: 60 to 900. Minimum is 60 for Iceberg destinations. The stream will accept smaller values at create time but enforce 60 at runtime, leading to records that never appear during a 30-second test.
## Source-side configuration
`IcebergDestinationConfiguration` works with all three Firehose source types:
- Direct PUT: no `KinesisStreamSourceConfiguration` block.
- Kinesis Data Streams: add `KinesisStreamSourceConfiguration` with `KinesisStreamARN` and a role that holds `kinesis:DescribeStream`, `kinesis:GetShardIterator`, `kinesis:GetRecords`, `kinesis:ListShards`.
- MSK: add `MSKSourceConfiguration` with the cluster ARN and topic. The Firehose role needs `kafka-cluster:Connect`, `kafka-cluster:DescribeCluster`, `kafka-cluster:DescribeClusterDynamicConfiguration`, `kafka-cluster:DescribeTopic`, `kafka-cluster:DescribeGroup`, `kafka-cluster:ReadData` on the topic, plus `kafka:GetBootstrapBrokers` on the cluster.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |