AWS Glue

Set up an AWS Glue destination in Streamkap using the Glue Data Catalog, with IAM role and trust policy configuration for append or upsert ingestion.

Prerequisites

An AWS user with sufficient privileges to create and configure IAM users and roles.

AWS Glue Setup

1. Create IAM User

It is recommended to create a separate IAM user and role with minimum necessary access.

Go to the Amazon IAM Console.
Click Users in the sidebar.
Click Add users.
Enter a User name: e.g., streamkap_user
Set the Access type:
- ✅ Check “Access key – Programmatic access”
- ❌ Uncheck “Console access” (not needed)

Click Next.

Choose Attach policies directly and create a custom policy:
- Click Create policy (opens in new tab).
- Go to the JSON tab and paste this policy, ensuring you replace any <...> placeholders as required:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:CreateTable",
                "glue:GetTable",
                "glue:GetTables",
                "glue:UpdateTable",
                "glue:DeleteTable",
                "glue:CreatePartition",
                "glue:GetPartition",
                "glue:GetPartitions",
                "glue:BatchCreatePartition",
                "glue:UpdatePartition",
                "glue:DeletePartition",
                "glue:BatchDeletePartition"
            ],
            "Resource": [
                "arn:aws:glue:<region>:<account_id>:catalog",
                "arn:aws:glue:<region>:<account_id>:database/<glue_database_name>",
                "arn:aws:glue:<region>:<account_id>:table/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]
}

For the Connector to access AWS Glue, it needs to be able to assume the user’s role. Please grant the following Trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::300973880807:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Streamkap Setup

1. Create the Destination

Navigate to Add Connectors.
Choose Iceberg.
Name: Enter a name for your connector.

2. Connection Settings

Catalog Type: The type of Iceberg catalog.
- GLUE:
  - AWS IAM Role: AWS IAM role (e.g., arn:aws:iam:::role/)
Region: The AWS region to be used.
S3 Bucket Path: Path to the storage location for the Iceberg tables.
Schema: An Iceberg table name prefix—equivalent to a database schema (e.g., public, sales, analytics).

3. Ingestion Settings

Ingestion Mode: Specifies the strategy used to insert events into the Iceberg tables.

Changing ingestion mode
append and upsert modes use different, incompatible methods for loading data into the Iceberg tables. If - for whatever reason - you want to change modes for an existing Iceberg Connector, please create a new Iceberg Destination instead i.e. a separate destination for insert, and for upsert.
- upsert mode:
  - Primary key fields: Optional. A comma-separated list of field names to use as record identifiers when a primary key’s not present.

Iceberg partitions a table by the values in one or more of its columns, so a query that filters on those columns reads only the matching files. Streamkap applies the spec you set here when it creates a table.

Partition By: Optional. A comma-separated partition spec applied to every table this Connector creates, for example day(created_at), bucket(id, 16). Each entry is one of:
- a column name on its own, to partition by that column’s value;
- year(col), month(col), day(col) or hour(col), to partition a date or timestamp column by that period;
- bucket(col, N), to hash the column into N buckets;
- truncate(col, N), to partition a string by its first N characters, or a number into ranges N wide.
Write the transform name in lower case: day(created_at) is accepted, DAY(created_at) is not. year, month, day and hour also accept the plural (years, months, days, hours). For bucket and truncate, N must be a whole number above zero.

Column names, unlike transform names, are case-sensitive and must match the column as it arrives at the Destination.
Per-table Partitioning: Optional, advanced setting. Partition specs for named tables, as a JSON object keyed by the bare table name. It requires Schema to be set, and adds the Schema to each key for you, so leave it out of the key. A table listed here uses its own spec instead of Partition By, and an empty string leaves that table unpartitioned. Each key is a bare table name with no dots or spaces, and table names are matched without case, so Orders and orders count as the same table and cannot both appear.
```
{
  "orders": "day(order_date), bucket(customer_id, 16)",
  "events": "hour(event_time)",
  "audit": ""
}
```

Click Save.