BigQuery overview

Configure Google BigQuery as a Streamkap CDC destination: prerequisites, IAM permissions, service account JSON key, Storage Write API, and schema evolution.

Stream change data into Google BigQuery. Streamkap writes using the BigQuery Storage Write API — the lowest-cost, high-throughput ingestion path.

Prerequisites

• A Google Cloud project.
• A BigQuery dataset that already exists — Streamkap writes tables into it but does not create the dataset for you.
• A Google service account with a JSON key (P12 keys are not supported).
• The service account granted the role below.

Required permissions

Grant the service account access on the target dataset (least privilege), not the whole project:

• BigQuery Data Editor (roles/bigquery.dataEditor) on the dataset — covers everything Streamkap needs: writing rows, creating tables, and evolving table schemas.

Delivery guarantees

Streamkap provides at-least-once delivery and appends rows via the Storage Write API, so retries can write the same change more than once — a table can contain duplicate rows for the same key. Query the latest version per key (or schedule a cleanup) as shown in Best Practices.

Google Cloud setup

Replace the { ... } placeholders. You can use the Cloud Console or the gcloud/bq CLI:

# 1. Create the dataset (if it doesn't exist yet)
bq --location={REGION} mk --dataset {PROJECT_ID}:{DATASET}

# 2. Create a service account
gcloud iam service-accounts create streamkap-bigquery \
  --display-name="Streamkap BigQuery" --project={PROJECT_ID}

# 3. Grant BigQuery Data Editor on the dataset (least privilege).
#    Easiest via Console: BigQuery > {DATASET} > Sharing > Permissions >
#    Add principal = the service account email, role = BigQuery Data Editor.

# 4. Create a JSON key
gcloud iam service-accounts keys create key.json \
  --iam-account=streamkap-bigquery@{PROJECT_ID}.iam.gserviceaccount.com

Streamkap setup

1. Go to Destinations and choose BigQuery.
2. Enter the configuration:
   • Name — a unique, memorable name for this destination.
   • JSON key file — upload the service-account JSON key you created.
   • Dataset Name — the existing dataset to write into. The GCP project is read automatically from the key file.
   • Time Partitioning — partition granularity for auto-created tables (DAY, HOUR, MONTH, YEAR). Default DAY. Choose NONE to create non-partitioned tables.
   • Partition Field (optional) — a record field to partition by. Leave blank for ingestion-time partitioning.
   • Clustering Fields (optional) — comma-separated fields to cluster by (max 4).
   • Partition Expiration in Days (optional) — automatically delete partitions older than this many days from auto-created tables. Leave blank to keep all partitions. Ignored when Time Partitioning is NONE.
   • Auto-create Tables — create BigQuery tables for new topics automatically (on by default).
   • Allow New Fields / Allow Required Field Relaxation — let the table schema evolve as the source schema changes (both on by default).
   • Tasks — number of parallel tasks.
3. Click Save.

Schema evolution

With Allow New Fields and Allow Required Field Relaxation enabled (the defaults), Streamkap keeps BigQuery tables in step with the source as it changes:

Data type mapping

Mappings are best-effort to the nearest BigQuery equivalent.

Network access

BigQuery is reached over Google Cloud APIs, so there is no IP allowlist to configure — access is controlled entirely by the service account’s IAM permissions.

Related documentation

• BigQuery Best Practices — partitioning, clustering, and deduplicating appended data
• Error Reference — common error codes and resolution steps