Skip to main content
Change Data Capture using Cloudflare R2 with Streamkap

​
Requirements

  • Cloudflare R2 Access Key and Secret Access Key with the following permissions to the destination bucket:
    • Object Read & Write : Allows the ability to read, write, and list objects in specific buckets.

​
Configure Cloudflare R2 Connector

  • Name: A descriptive name for the connector
  • Access Key: An Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • Secret Access Key: The Secret Access Key with the appropriate permissions for the bucket to which Streamkap will load data
  • Account ID: Cloudflare Account ID. This is required to access the R2 bucket.
  • Bucket Name: The name of the bucket to which Streamkap will load data
  • Format: The format of the file. The following options are available: CSV, JSONL, JSON and Parquet
  • Filename Template: The format of the filename. See below for more information about formatting options.
  • Compression Type: Compression type for output files. Supported algorithms are gzip, snappy, zstd and none. Defaults to gzip
  • Output Fields: List of fields to include in output. Available options are: key, offset, timestamp, value and headers. Defaults to value.
Filename template (default: {{topic}}--{{partition}}--{{start_offset}}): The format of the filename. You can combine any of the elements below using other text or characters, including dashes (-) and underscores (_) Currently, supported combinations of variables are:
  • topic, partition, start_offset, and timestamp - grouping by the topic, partition, and timestamp;
  • key - grouping by the key.
  • key, topic, partition - grouping by the topic, partition, and key.