Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Google Cloud Storage

Stream change data capture records into a Google Cloud Storage bucket from Streamkap, with JSON Lines, JSON Array, or Parquet formats.

Requirements

  • A Google Cloud Storage bucket for Streamkap to write to.

  • A Google Cloud service account with a JSON key. The service account needs these permissions on the bucket:

    • storage.objects.create
    • storage.objects.delete (needed to overwrite files)

    The predefined Storage Object User role (roles/storage.objectUser) includes both. You can also create a custom role with only these two permissions.

Configure Google Cloud Storage Connector

  • Name: A descriptive name for the connector
  • GCS Credentials JSON Service Account Key: The service account’s JSON key. Paste it as a single line, with every double quote escaped as \" and every \n inside private_key escaped as \\n. For example: {\"type\": \"service_account\", \"project_id\": \"my-project\", ...}
  • Bucket Name: The name of the bucket to which Streamkap will load data
  • Format: The format of the file. Choose JSON Lines, JSON Array or Parquet
  • Filename Template: The format of the filename. A file extension is appended automatically based on the selected Format and Compression Type, so do not include one in the template. See below for more information about formatting options.
  • Filename Prefix: Text added to the start of every filename. Use it to write files into a folder in the bucket, for example dir1/dir2/.
  • Compression Type: Compression type for output files. Supported algorithms are gzip, snappy, zstd and none. Defaults to gzip.
  • Output Fields: List of fields to include in output. Available options are: key, offset, timestamp, value and headers. Defaults to value.

Streamkap adds a _streamkap_offset field to each record’s value. It holds the record’s position in its Streamkap topic.

Filename template (default: {{topic}}-{{partition}}-{{start_offset}}): The format of the filename, excluding the extension. You can combine any of the elements below using other text or characters, including dashes (-) and underscores (_).

Element Description
{{topic}} The Streamkap topic name. For example, a PostgreSQL Source table web.salesorders topic’s name would be salesorders
{{partition:padding=true|false}} The partition number of the records in the file, typically 0. Streamkap topics and their data can be partitioned for better performance in certain scenarios. For example, a topic salesorders has 10 partitions, 0 through to 9. If padding is set to true, the number has leading zeroes. Defaults to false
{{start_offset:padding=true|false}} The offset number of the first record in the file. Every record streamed has an incrementing offset number. Offset numbers are not unique across partitions of the same topic. If padding is set to true, the number has leading zeroes. Defaults to false
{{timestamp:unit=yyyy|MM|dd|HH}} The timestamp for when the file was created by the connector. For example, the template {{topic}}{{timestamp:unit=yyyy}}-{{timestamp:unit=MM}} and a timestamp of 2024-01-01 20:24 would create a file named salesorders2024-01
{{key}} The Kafka key

Currently, supported combinations of variables are:

  • topic, partition, start_offset, and timestamp - grouping by the topic, partition, and timestamp;
  • key - grouping by the key.
  • key, topic, partition - grouping by the topic, partition, and key.