Skip to content

jaingxyz/aws-data-skills / skills/firehose-iceberg-pipeline

firehose-iceberg-pipeline

Verified

Build an Amazon Data Firehose delivery stream that writes to Apache Iceberg tables on S3 Tables via IcebergDestinationConfiguration. Covers the column-shape footgun (records land in the error bucket with Iceberg.MissingColumnWithinRecord unless a transform Lambda reshapes them), the microsecond timestamp requirement, the three-phase deploy pattern that works around Firehose's synchronous glue:GetTable validation at create time, parameter-inheritance traps in `aws cloudformation deploy`, Lake Formation grant ordering, and how to decode the error bucket. Use when you see "Firehose to Iceberg", "Firehose to S3 Tables", "IcebergDestinationConfiguration", "Firehose data transformation Lambda", "Firehose error bucket", "Iceberg.MissingColumnWithinRecord", "MissingColumnWithinRecord", or any time someone is wiring a Kinesis Data Stream / Direct PUT source to Iceberg tables and producer record shape does not match the Iceberg column layout.

FileKindSize
SKILL.mdskill md22.4 KB
reference/firehose-role-policies.mdmarkdown5.1 KB
reference/iceberg-table-create.mdmarkdown4.7 KB

reference/firehose-role-policies.md

View on GitHub
# Firehose role: complete IAM policy document

The Firehose role for `IcebergDestinationConfiguration` writing to S3
Tables needs five distinct permission groups. Missing any one of them
results in a synchronous stream-create failure or, worse, a successful
stream that silently drops records.

This file is the full inline-policies template. Replace `<account-id>`,
`<region>`, and the resource names to fit your stack.

## Trust policy

```yaml
AssumeRolePolicyDocument:
  Version: "2012-10-17"
  Statement:
    - Effect: Allow
      Principal:
        Service: firehose.amazonaws.com
      Action: sts:AssumeRole
```

## Inline policy 1: Kinesis read (skip if source is Direct PUT)

```yaml
- PolicyName: KinesisRead
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - kinesis:DescribeStream
          - kinesis:GetShardIterator
          - kinesis:GetRecords
          - kinesis:ListShards
        Resource: !ImportValue my-kinesis-stream-arn
```

## Inline policy 2: Glue access through the federated catalog

The bucket-nested `s3tablescatalog/<bucket>` catalog has THREE ARN
layers that Firehose walks: the parent catalog, the bucket-nested
child, and the database/table under each. List all of them:

```yaml
- PolicyName: GlueIceberg
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - glue:GetTable
          - glue:GetTables
          - glue:GetDatabase
          - glue:GetDatabases
          - glue:CreateTable
          - glue:UpdateTable
        Resource:
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog/s3tablescatalog"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:catalog/s3tablescatalog/*"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/s3tablescatalog/*"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:database/*/s3tablescatalog/*"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:table/s3tablescatalog/*/*"
          - !Sub "arn:aws:glue:${AWS::Region}:${AWS::AccountId}:table/*/s3tablescatalog/*/*"
      # Lake Formation is the gatekeeper for the federated catalog.
      # Grant the API-level call here; the actual data perms are
      # granted via lakeformation grant-permissions, not IAM.
      - Effect: Allow
        Action:
          - lakeformation:GetDataAccess
        Resource: "*"
```

## Inline policy 3: S3 Tables data I/O

Firehose talks to S3 Tables directly for the actual writes (Glue is
only the metadata path):

```yaml
- PolicyName: S3TablesIO
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - s3tables:GetTableBucket
          - s3tables:GetNamespace
          - s3tables:GetTable
          - s3tables:GetTableMetadataLocation
          - s3tables:UpdateTableMetadataLocation
          - s3tables:PutTableData
          - s3tables:GetTableData
        Resource:
          - !GetAtt TableBucket.TableBucketARN
          - !Sub "${TableBucket.TableBucketARN}/*"
```

## Inline policy 4: error bucket I/O

```yaml
- PolicyName: ErrorBucketIO
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - s3:PutObject
          - s3:GetBucketLocation
          - s3:ListBucket
        Resource:
          - !GetAtt FirehoseErrorBucket.Arn
          - !Sub "${FirehoseErrorBucket.Arn}/*"
```

## Inline policy 5: CloudWatch Logs

```yaml
- PolicyName: Logs
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - logs:CreateLogStream
          - logs:PutLogEvents
        # CreateLogStream and PutLogEvents are stream-level actions; the
        # canonical resource ARN is the log-group ARN with a
        # :log-stream:* suffix. The bare LogGroup.Arn alone is the
        # log-group resource type and CloudWatch Logs may reject it
        # under strict-mode IAM evaluation.
        Resource: !Sub "${FirehoseLogGroup.Arn}:log-stream:*"
```

## Inline policy 6: invoke transform Lambda

Easy to forget. The FIREHOSE role (not the Lambda execution role) needs
this for `ProcessingConfiguration` to work:

```yaml
- PolicyName: InvokeTransform
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - lambda:InvokeFunction
          - lambda:GetFunctionConfiguration
        Resource: !GetAtt TransformLambda.Arn
```

## What is NOT in IAM: Lake Formation grants

The IAM `glue:GetTable` permission alone is not enough. Lake Formation
also needs to grant DESCRIBE on the catalog, DESCRIBE / CREATE_TABLE /
ALTER on the database, and ALL on the table to the role. These are
applied via `aws lakeformation grant-permissions` (NOT in CFN; see
SKILL.md "Lake Formation grants for the Firehose role").

If the IAM is correct but LF grants are missing, the symptom is the
same as missing IAM:

```
Role <arn> is not authorized to perform: glue:GetTable for the given
table or the table does not exist
```

This error is misleading; both IAM and LF have to be present.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 323kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall