jaingxyz/aws-data-skills / skills/streaming-into-data-lake
streaming-into-data-lake
VerifiedStream rows continuously into Apache Iceberg tables on S3 Tables (or standard Iceberg on a general purpose bucket) using Amazon Data Firehose with IcebergDestinationConfiguration. Covers the Firehose stream, the optional record-shaping Lambda (ProcessingConfiguration), the Lake Formation grants required for Firehose to write through Glue, and the three-phase CloudFormation deploy that gates the stream on grants existing. Triggers on: stream into data lake, Firehose to S3 Tables, Firehose to Iceberg, real-time ingestion to Iceberg, Firehose IcebergDestinationConfiguration, stream Kinesis to Iceberg, near real-time CDC to data lake, continuous append to Iceberg. Do NOT use for: batch file imports (use ingesting-into-data-lake), creating the destination Iceberg table itself (use creating-data-lake-table), the one-time Lake Formation onboarding (use setting-up-lake-formation), finding tables by fuzzy name (use finding-data-lake-assets), running queries (use querying-data-lake), Glue connections (Firehose does not use them; do not invoke connecting-to-data-source), Kinesis to Lambda to Redshift Data API direct writes (not Iceberg; recommend a different workflow), or producer-side Kafka or MSK setup.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.0 KB |
| evals/baseline.json | config | 85 B |
| evals/evals.json | config | 15.0 KB |
| references/error-bucket-decoding.md | markdown | 6.2 KB |
| references/firehose-iceberg-config.md | markdown | 5.1 KB |
| references/lake-formation-grants.md | markdown | 5.4 KB |
| references/three-phase-deploy.md | markdown | 5.6 KB |
| references/transform-lambda.md | markdown | 6.4 KB |
references/lake-formation-grants.md
View on GitHub# Lake Formation Grants for Firehose
Firehose writes to Iceberg tables through the Glue catalog. Lake Formation gates every catalog access. The Firehose role MUST hold three grants. Missing any one produces an opaque `AccessDenied` from Firehose with no indication of which grant is missing.
## The grants (depends on stream mode)
The exact permissions depend on whether the stream is `AppendOnly: true` (the default for CDC archive use cases) or in MERGE mode (`AppendOnly: false` with per-record `update`/`delete`).
For `AppendOnly: true`:
| Resource | Permissions | What it lets Firehose do |
|---|---|---|
| Catalog | `DESCRIBE` | List databases under the catalog |
| Database | `DESCRIBE`, `CREATE_TABLE`, `ALTER` | Read metadata, evolve schema. `CREATE_TABLE` is needed for some Glue-side compaction operations Iceberg performs. |
| Table | `DESCRIBE`, `SELECT`, `INSERT`, `ALTER` | Read manifest, append rows, evolve schema. `DESCRIBE` is required even for insert-only streams (Lake Formation gates manifest reads under `DESCRIBE` + `SELECT`). |
For MERGE mode add `DELETE` to the Table grant.
The companion repo's working set (`07-deploy-iceberg.sh`) uses the AppendOnly grants and is the validated reference. Do not add `DELETE` until you switch the stream's `AppendOnly` flag to `false`; under append-only, granting unused `DELETE` is harmless but inconsistent with the principle of least privilege.
## CloudFormation form
```yaml
LFGrantOnCatalog:
Type: AWS::LakeFormation::PrincipalPermissions
Condition: CreateFirehoseRole
Properties:
Principal:
DataLakePrincipalIdentifier: !GetAtt FirehoseRole.Arn
Resource:
Catalog: {}
Permissions: [DESCRIBE]
PermissionsWithGrantOption: []
LFGrantOnDatabase:
Type: AWS::LakeFormation::PrincipalPermissions
Condition: CreateFirehoseRole
Properties:
Principal:
DataLakePrincipalIdentifier: !GetAtt FirehoseRole.Arn
Resource:
Database:
CatalogId: !Ref AWS::AccountId
Name: !Ref Namespace
Permissions: [DESCRIBE, CREATE_TABLE, ALTER]
LFGrantOnTable:
Type: AWS::LakeFormation::PrincipalPermissions
Condition: CreateFirehoseRole
Properties:
Principal:
DataLakePrincipalIdentifier: !GetAtt FirehoseRole.Arn
Resource:
Table:
CatalogId: !Ref AWS::AccountId
DatabaseName: !Ref Namespace
Name: !Ref TableName
Permissions: [DESCRIBE, SELECT, INSERT, ALTER]
# For MERGE mode (AppendOnly: false), append DELETE to Permissions.
```
For S3 Tables targets, the catalog ID for the database and table grants is the federated catalog identifier `<account-id>:s3tablescatalog/<bucket>`, not the account id alone.
## Shell helper for incremental grants
CloudFormation's `AWS::LakeFormation::PrincipalPermissions` does not handle the case where the grant already exists. A re-run produces `ResourceAlreadyExistsException` and fails the deploy. For deploys that may be re-run (CI, idempotent scripts), use a shell helper that aborts on real errors and treats already-granted as success:
```bash
lf_grant() {
local principal="$1"
local resource_json="$2"
local permissions="$3"
local out
out=$(aws lakeformation grant-permissions \
--principal "DataLakePrincipalIdentifier=${principal}" \
--resource "${resource_json}" \
--permissions ${permissions} 2>&1) || {
if echo "${out}" | grep -q "AlreadyExists\|already granted"; then
echo " already granted, skipping"
return 0
fi
echo "ERROR: lf_grant failed:"
echo "${out}"
return 1
}
echo " granted"
}
# Catalog grant
lf_grant "${FIREHOSE_ROLE_ARN}" \
'{"Catalog":{}}' \
"DESCRIBE"
# Database grant
lf_grant "${FIREHOSE_ROLE_ARN}" \
"{\"Database\":{\"CatalogId\":\"${CATALOG_ID}\",\"Name\":\"${DB_NAME}\"}}" \
"DESCRIBE"
# Table grant
lf_grant "${FIREHOSE_ROLE_ARN}" \
"{\"Table\":{\"CatalogId\":\"${CATALOG_ID}\",\"DatabaseName\":\"${DB_NAME}\",\"Name\":\"${TABLE_NAME}\"}}" \
"SELECT INSERT DELETE"
```
The function aborts on any error other than already-granted. The deploy script that calls it MUST run with `set -euo pipefail` so a failed grant stops the rest of the deploy.
## Data Lake Admin requirement
The identity running the Lake Formation grant calls MUST be a Data Lake Admin in the region. Without admin status, every grant call returns:
```
AccessDeniedException: Insufficient Lake Formation permission(s) on <resource>
```
This happens even when the IAM identity has `lakeformation:GrantPermissions` on `*`. Lake Formation enforces admin status separately from IAM. Verify with:
```bash
aws lakeformation get-data-lake-settings --region <region> \
--query "DataLakeSettings.DataLakeAdmins[].DataLakePrincipalIdentifier" \
--output text
```
If the calling ARN is not in the list, either run the deploy as a different identity or have an existing admin add the deploy identity. Adding a Data Lake Admin requires existing admin status and is not in scope for this skill; see `setting-up-lake-formation`.
## Verifying grants applied
```bash
aws lakeformation list-permissions \
--principal DataLakePrincipalIdentifier=<firehose-role-arn> \
--region <region>
```
Output should show three entries: one with `Catalog`, one with the database, one with the table. If any is missing, re-run the corresponding `lf_grant` call. Lake Formation grant propagation is usually under 5 seconds but can take up to 30 seconds.
`aws iam simulate-principal-policy` does NOT see Lake Formation grants. Do not use it to verify Firehose Iceberg permissions.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |