Apache Iceberg (Generic)

Set up an Apache Iceberg destination in Streamkap using a REST or Hive catalog, with S3 bucket and IAM configuration for append or upsert ingestion.

Prerequisites

An AWS user with sufficient privileges to create and configure IAM users, roles and S3 buckets.

Iceberg Data Lake Setup

1. Create S3 Bucket

For better isolation, security and resource management, it is recommended to create a new S3 bucket, specifically for your Streamkap-to-Iceberg pipelines.

Go to the Amazon S3 Console.
Click Create bucket.
Set:
- Bucket name: e.g., iceberg-bucket.
- Region: Match your compute region (e.g., us-west-2).
Keep Object Ownership set to ACLs disabled.
Uncheck “Block all public access” only if you’re managing access via bucket policies (optional).
Click Create Bucket.

Optionally, create a base folder to further organize Iceberg tables created by the connector.

Open the new bucket.
Click Create folder.
Enter a name: e.g., warehouse/streamkap/.

2. Create S3 User

It is recommended to create a separate IAM user and role with minimum necessary access.

Go to the Amazon IAM Console.
Click Users in the sidebar.
Click Add users.
Enter a User name: e.g., streamkap_user
Set the Access type:
- ✅ Check “Access key – Programmatic access”
- ❌ Uncheck “Console access” (not needed)

Click Next.

Choose Attach policies directly and create a custom policy:
- Click Create policy (opens in new tab).
- Go to the JSON tab and paste this policy, ensuring you replace any <...> placeholders as required:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStreamkapAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<account_id>:user/streamkap_user"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket_name>",
        "arn:aws:s3:::<bucket_name>/*"
      ]
    }
  ]
}

Streamkap Setup

1. Create the Destination

Navigate to Add Connectors.
Choose Iceberg.
Name: Enter a name for your connector.

2. Connection Settings

Catalog Type: The type of Iceberg catalog.
- RESTorHIVE:
  - Catalog Name: Iceberg catalog name.
  - Catalog URI: The Iceberg catalog URI.
  - AWS Access Key: The AWS Access Key ID used to connect to S3.
  - AWS Secret Access Key: The AWS Secret Access Key used to connect to S3.
Region: The AWS region to be used.
S3 Bucket Path: Path to the storage location for the Iceberg tables.
Schema: An Iceberg table name prefix—equivalent to a database schema (e.g., public, sales, analytics).

3. Ingestion Settings

Ingestion Mode: Specifies the strategy used to insert events into the Iceberg tables.

Changing ingestion mode
append and upsert modes use different, incompatible methods for loading data into the Iceberg tables. If - for whatever reason - you want to change modes for an existing Iceberg Connector, please create a new Iceberg Destination instead i.e. a separate destination for insert, and for upsert.
- upsertmode:
  - Primary key fields: Optional. A comma-separated list of field names to use as record identifiers when a primary key’s not present.

Iceberg partitions a table by the values in one or more of its columns, so a query that filters on those columns reads only the matching files. Streamkap applies the spec you set here when it creates a table.

Partition By: Optional. A comma-separated partition spec applied to every table this Connector creates, for example day(created_at), bucket(id, 16). Each entry is one of:
- a column name on its own, to partition by that column’s value;
- year(col), month(col), day(col) or hour(col), to partition a date or timestamp column by that period;
- bucket(col, N), to hash the column into N buckets;
- truncate(col, N), to partition a string by its first N characters, or a number into ranges N wide.
Write the transform name in lower case: day(created_at) is accepted, DAY(created_at) is not. year, month, day and hour also accept the plural (years, months, days, hours). For bucket and truncate, N must be a whole number above zero.

Column names, unlike transform names, are case-sensitive and must match the column as it arrives at the Destination.
Per-table Partitioning: Optional, advanced setting. Partition specs for named tables, as a JSON object keyed by the bare table name. It requires Schema to be set, and adds the Schema to each key for you, so leave it out of the key. A table listed here uses its own spec instead of Partition By, and an empty string leaves that table unpartitioned. Each key is a bare table name with no dots or spaces, and table names are matched without case, so Orders and orders count as the same table and cannot both appear.
```
{
  "orders": "day(order_date), bucket(customer_id, 16)",
  "events": "hour(event_time)",
  "audit": ""
}
```

Click Save.