Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Amazon S3

Stream change data capture records into Amazon S3 from Streamkap, with JSON Lines, JSON Array, or Parquet formats and configurable file rotation.

Requirements

  • AWS Access Key and Secret Access Key with the following permissions to the destination bucket:

    • s3:GetObject
    • s3:PutObject
    • s3:AbortMultipartUpload
    • s3:ListMultipartUploadParts
    • s3:ListBucketMultipartUploads

Configure Amazon S3 Connector

  • Name: A descriptive name for the connector
  • AWS Access Key: An Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • AWS Secret Access Key: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • Region: Name of the region for bucket to which Streamkap will load data
  • Bucket Name: The name of the bucket to which Streamkap will load data
  • Format: The format of the file. The following options are available: JSON Lines, JSON Array and Parquet
  • Filename Template: The format of the filename. A file extension is appended automatically based on the selected Format and Compression Type — do not include one in the template. See below for more information about formatting options.
  • Compression Type: Compression type for output files. Supported algorithms are gzip, snappy, zstd and none. Defaults to gzip. Only shown for JSON Lines and JSON Array — hidden for Parquet, which compresses internally and doesn’t use this setting.
  • Envelope Output (Advanced, default: enabled): Wraps each output record in an envelope with Kafka metadata (key, offset, timestamp, headers) alongside the value. Disable to write only the record’s own value. For Parquet, only applies when the value is a record or map.
  • Output Fields (Advanced): List of fields to include in output. Available options are: key, offset, timestamp, value and headers. Defaults to value.
  • Max Records Per File (Advanced, default: 0): Maximum number of records buffered into a single file before it’s rotated. This caps how large a file can grow, but rotation only makes a new file eligible for upload — files are still written to S3 on the connector’s normal flush interval, not immediately when the cap is hit. Set to 0 for unlimited (one file per topic-partition per flush interval).
  • S3 Multipart Upload Part Size (bytes) (Advanced, default: 5242880 / 5MB): Size of each part in a multipart upload to S3. This controls how a single file is chunked for upload to S3, not how many records go into a file — see Max Records Per File for that. Larger parts mean fewer S3 API calls for big files but more memory used per upload. AWS requires multipart parts to be at least 5MB (except the last part of a file).

Filename template (default: {{topic}}-{{partition}}-{{start_offset}}): The format of the filename, excluding the extension. You can combine any of the elements below using other text or characters, including dashes (-) and underscores (_). Static text placed at the start of the template (e.g. dir1/dir2/{{topic}}-{{partition}}-{{start_offset}}) acts as a directory prefix.

Element Description
{{topic}} The Streamkap topic name. For example, a PostgreSQL Source table web.salesorders topic’s name would be salesorders
{{partition:padding=true|false}} The partition number of the records in the file, typically 0. Streamkap topics and their data can be partitioned for better performance in certain scenarios. For example, a topic salesorders has 10 partitions, 0 through to 9. If padding set to true it will set leading zeroes for offset, the default value is false;
{{start_offset:padding=true|false}} The offset number of the first record in the file. Every record streamed has an incrementing offset number. For example, a topic salesorders has 1000 records, offsets 0 through to 999. Note that in the case of a multi-partitioned topic, offset numbers are not unique across partitions. If padding set to true it will set leading zeroes for offset, the default value is false;
{{timestamp:unit=yyyy|MM|dd|HH}} The timestamp for when the file was created by the Connector. For example, the template {{topic}}{{timestamp:unit=yyyy}}-{{timestamp:unit=MM}} and timestamp of 2024-01-01 20:24 would create a file named salesorders2024-01
{{key}} The Kafka key

Currently, supported combinations of variables are:

  • topic, partition, start_offset, and timestamp - grouping by the topic, partition, and timestamp;
  • key - grouping by the key.
  • key, topic, partition - grouping by the topic, partition, and key.