Skip to main content

​
Requirements

  • AWS Access Key and Secret Access Key with the following permissions to the destination bucket:
    • s3:GetObject
    • s3:PutObject
    • s3:AbortMultipartUpload
    • s3:ListMultipartUploadParts
    • s3:ListBucketMultipartUploads

​
Configure Amazon S3 Connector

  • Name: A descriptive name for the connector
  • AWS Access Key: An Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • AWS Secret Access Key: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • Region: Name of the region for bucket to which Streamkap will load data
  • Bucket Name: The name of the bucket to which Streamkap will load data
  • Format: The format of the file. The following options are available: JSON Lines, JSON Array and Parquet
CSV output is not currently supported. Writing CSV requires reading the source data as raw, unparsed bytes (a ByteArrayConverter) instead of as structured records, and Streamkap doesn’t currently offer that as a serialization option. If you have a use case that needs CSV output, contact Streamkap support to discuss options.
  • Filename Template: The format of the filename. A file extension is appended automatically based on the selected Format and Compression Type — do not include one in the template. See below for more information about formatting options.
  • Compression Type: Compression type for output files. Supported algorithms are gzip, snappy, zstd and none. Defaults to gzip. Only shown for JSON Lines and JSON Array — hidden for Parquet, which compresses internally and doesn’t use this setting.
  • Envelope Output (Advanced, default: enabled): Wraps each output record in an envelope with Kafka metadata (key, offset, timestamp, headers) alongside the value. Disable to write only the record’s own value. For Parquet, only applies when the value is a record or map.
  • Output Fields (Advanced): List of fields to include in output. Available options are: key, offset, timestamp, value and headers. Defaults to value.
  • Max Records Per File (Advanced, default: 0): Maximum number of records buffered into a single file before it’s rotated. This caps how large a file can grow, but rotation only makes a new file eligible for upload — files are still written to S3 on the connector’s normal flush interval, not immediately when the cap is hit. Set to 0 for unlimited (one file per topic-partition per flush interval).
  • S3 Multipart Upload Part Size (bytes) (Advanced, default: 5242880 / 5MB): Size of each part in a multipart upload to S3. This controls how a single file is chunked for upload to S3, not how many records go into a file — see Max Records Per File for that. Larger parts mean fewer S3 API calls for big files but more memory used per upload. AWS requires multipart parts to be at least 5MB (except the last part of a file).
Filename template (default: {{topic}}-{{partition}}-{{start_offset}}): The format of the filename, excluding the extension. You can combine any of the elements below using other text or characters, including dashes (-) and underscores (_). Static text placed at the start of the template (e.g. dir1/dir2/{{topic}}-{{partition}}-{{start_offset}}) acts as a directory prefix. Currently, supported combinations of variables are:
  • topic, partition, start_offset, and timestamp - grouping by the topic, partition, and timestamp;
  • key - grouping by the key.
  • key, topic, partition - grouping by the topic, partition, and key.