Skip to main content
Stream change data into Google BigQuery. Streamkap writes using the BigQuery Storage Write API — the lowest-cost, high-throughput ingestion path.

​
Prerequisites

  • A Google Cloud project.
  • A BigQuery dataset that already exists — Streamkap writes tables into it but does not create the dataset for you.
  • A Google service account with a JSON key (P12 keys are not supported).
  • The service account granted the role below.

​
Required permissions

Grant the service account access on the target dataset (least privilege), not the whole project:
  • BigQuery Data Editor (roles/bigquery.dataEditor) on the dataset — covers everything Streamkap needs: writing rows, creating tables, and evolving table schemas.
Prefer a custom role? The minimum permissions are bigquery.datasets.get, bigquery.tables.get, bigquery.tables.list, bigquery.tables.updateData, plus bigquery.tables.create (auto-create tables) and bigquery.tables.update (schema evolution).

​
Delivery guarantees

Streamkap provides at-least-once delivery and appends rows via the Storage Write API, so retries can write the same change more than once — a table can contain duplicate rows for the same key. Query the latest version per key (or schedule a cleanup) as shown in Best Practices.

​
Google Cloud setup

Replace the { ... } placeholders. You can use the Cloud Console or the gcloud/bq CLI: 
# 1. Create the dataset (if it doesn't exist yet)
bq --location={REGION} mk --dataset {PROJECT_ID}:{DATASET}

# 2. Create a service account
gcloud iam service-accounts create streamkap-bigquery \
  --display-name="Streamkap BigQuery" --project={PROJECT_ID}

# 3. Grant BigQuery Data Editor on the dataset (least privilege).
#    Easiest via Console: BigQuery > {DATASET} > Sharing > Permissions >
#    Add principal = the service account email, role = BigQuery Data Editor.

# 4. Create a JSON key
gcloud iam service-accounts keys create key.json \
  --iam-account=streamkap-bigquery@{PROJECT_ID}.iam.gserviceaccount.com
The dataset must exist before you create the destination. Streamkap creates and evolves tables inside it, but never creates the dataset.

​
Streamkap setup

  1. Go to Destinations and choose BigQuery.
  2. Enter the configuration:
    • Name — a unique, memorable name for this destination.
    • JSON key file — upload the service-account JSON key you created.
    • Dataset Name — the existing dataset to write into. The GCP project is read automatically from the key file.
    • Time Partitioning — partition granularity for auto-created tables (DAY, HOUR, MONTH, YEAR). Default DAY. Choose NONE to create non-partitioned tables.
    • Partition Field (optional) — a record field to partition by. Leave blank for ingestion-time partitioning.
    • Clustering Fields (optional) — comma-separated fields to cluster by (max 4).
    • Partition Expiration in Days (optional) — automatically delete partitions older than this many days from auto-created tables. Leave blank to keep all partitions. Ignored when Time Partitioning is NONE.
    • Auto-create Tables — create BigQuery tables for new topics automatically (on by default).
    • Allow New Fields / Allow Required Field Relaxation — let the table schema evolve as the source schema changes (both on by default).
    • Tasks — number of parallel tasks.
  3. Click Save.

​
Schema evolution

With Allow New Fields and Allow Required Field Relaxation enabled (the defaults), Streamkap keeps BigQuery tables in step with the source as it changes:
Source changeBigQuery result
Add a columnAdded as a NULLABLE column; existing rows are NULL.
Make a NOT NULL column nullableColumn relaxed REQUIRED → NULLABLE.
Widen a type (e.g. INT→BIGINT, longer VARCHAR)No change — both map to the same BigQuery type.
BigQuery cannot drop or rename columns, and cannot add a REQUIRED column to a populated table. A dropped source column is retained in BigQuery and filled with NULLs going forward; a renamed column appears as a new column alongside the old one. Avoid incompatible type changes (e.g. VARCHAR→INT), which the connector cannot apply.
Source column names that aren’t valid BigQuery column names (for example names with leading digits or unsupported characters) are adjusted automatically so the data still loads — the BigQuery column may therefore differ slightly from the source column name.

​
Data type mapping

Mappings are best-effort to the nearest BigQuery equivalent.
StreamkapBigQuery
INT8, INT16, INT32, INT64INTEGER
FLOAT32, FLOAT64FLOAT
BOOLEANBOOLEAN
STRINGSTRING
BYTESBYTES
org.apache.kafka.connect.data.DecimalNUMERIC / BIGNUMERIC
org.apache.kafka.connect.data.DateDATE
org.apache.kafka.connect.data.TimeTIME
org.apache.kafka.connect.data.Timestamp, io.debezium.time.ZonedTimestampTIMESTAMP
io.debezium.data.JsonJSON
STRUCTRECORD
ARRAYrepeated field

​
Network access

BigQuery is reached over Google Cloud APIs, so there is no IP allowlist to configure — access is controlled entirely by the service account’s IAM permissions.