Cloudflare R2

Stream change data capture records into Cloudflare R2 from Streamkap, with configurable file formats, compression, and filename templates.

Change Data Capture using Cloudflare R2 with Streamkap

Requirements

Cloudflare R2 Access Key and Secret Access Key with the following permissions to the destination bucket:
- Object Read & Write : Allows the ability to read, write, and list objects in specific buckets.

Configure Cloudflare R2 Connector

Name: A descriptive name for the connector
Access Key: An Access Key with the appropriate permissions for the bucket to which Streamkap will load data
Secret Access Key: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load data
Account ID: Cloudflare Account ID. This is required to access the R2 bucket.
Bucket Name: The name of the bucket to which Streamkap will load data
Format: The format of the file. The following options are available: CSV, JSONL, JSON and Parquet
Filename Template: The format of the filename. See below for more information about formatting options.
Compression Type: Compression type for output files. Supported algorithms are gzip, snappy, zstd and none. Defaults to gzip
Output Fields: List of fields to include in output. Available options are: key, offset, timestamp, value and headers. Defaults to value.

Filename template (default: {{topic}}--{{partition}}--{{start_offset}}): The format of the filename. You can combine any of the elements below using other text or characters, including dashes (-) and underscores (_)

Element	Description
`{{topic}}`	The Streamkap topic name. For example, a PostgreSQL Source table `web.salesorders` topic’s name would be `salesorders`
`{{partition:padding=true\|false}}`	The partition number of the records in the file, typically `0`. Streamkap topics and their data can be partitioned for better performance in certain scenarios. For example, a topic `salesorders` has 10 partitions, 0 through to 9. If padding set to true it will set leading zeroes for offset, the default value is false;
`{{start_offset:padding=true\|false}}`	The offset number of the first record in the file. Every record streamed has an incrementing offset number. For example, a topic `salesorders` has 1000 records, offsets 0 through to 999. Note that in the case of a multi-partitioned topic, offset numbers are not unique across partitions. If padding set to true it will set leading zeroes for offset, the default value is false;
`{{timestamp:unit=yyyy\|MM\|dd\|HH}}`	The timestamp for when the file was created by the Connector. For example, the template `{{topic}}{{timestamp:unit=yyyy}}-{{timestamp:unit=MM}}` and timestamp of `2024-01-01 20:24` would create a file named `salesorders2024-01`
`{{key}}`	The Kafka key

Currently, supported combinations of variables are:

topic, partition, start_offset, and timestamp - grouping by the topic, partition, and timestamp;
key - grouping by the key.
key, topic, partition - grouping by the topic, partition, and key.