---
title: "BigQuery overview"
description: "Configure Google BigQuery as a Streamkap CDC destination: prerequisites, IAM permissions, service account JSON key, Storage Write API, and schema evolution."
sidebar:
  label: "Overview"
---

Stream change data into Google BigQuery. Streamkap writes using the
[BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api) — the
lowest-cost, high-throughput ingestion path.

## Prerequisites

* A Google Cloud project.
* A BigQuery **dataset** that already exists — Streamkap writes tables into it but does **not** create the dataset for you.
* A Google **service account** with a **JSON** key (P12 keys are not supported).
* The service account granted the role below.

## Required permissions

Grant the service account access **on the target dataset** (least privilege), not the whole project:

* **BigQuery Data Editor** (`roles/bigquery.dataEditor`) on the dataset — covers everything Streamkap needs: writing rows, creating tables, and evolving table schemas.

:::info
Prefer a custom role? The minimum permissions are `bigquery.datasets.get`,
`bigquery.tables.get`, `bigquery.tables.list`, `bigquery.tables.updateData`, plus
`bigquery.tables.create` (auto-create tables) and `bigquery.tables.update` (schema
evolution).
:::

## Delivery guarantees

Streamkap provides **at-least-once** delivery and appends rows via the Storage Write API, so retries can write the same change more than once — a table can contain **duplicate rows for the same key**. Query the latest version per key (or schedule a cleanup) as shown in [Best Practices](/google-bigquery-best-practices).

## Google Cloud setup

Replace the `{ ... }` placeholders. You can use the Cloud Console or the `gcloud`/`bq` CLI:

```bash gcloud
# 1. Create the dataset (if it doesn't exist yet)
bq --location={REGION} mk --dataset {PROJECT_ID}:{DATASET}

# 2. Create a service account
gcloud iam service-accounts create streamkap-bigquery \
  --display-name="Streamkap BigQuery" --project={PROJECT_ID}

# 3. Grant BigQuery Data Editor on the dataset (least privilege).
#    Easiest via Console: BigQuery > {DATASET} > Sharing > Permissions >
#    Add principal = the service account email, role = BigQuery Data Editor.

# 4. Create a JSON key
gcloud iam service-accounts keys create key.json \
  --iam-account=streamkap-bigquery@{PROJECT_ID}.iam.gserviceaccount.com
```

:::warning
The dataset must exist before you create the destination. Streamkap creates and
evolves **tables** inside it, but never creates the dataset.
:::

## Streamkap setup

1. Go to **Destinations** and choose **BigQuery**.
2. Enter the configuration:
   * **Name** — a unique, memorable name for this destination.
   * **JSON key file** — upload the service-account JSON key you created.
   * **Dataset Name** — the existing dataset to write into. The GCP project is read automatically from the key file.
   * **Time Partitioning** — partition granularity for auto-created tables (`DAY`, `HOUR`, `MONTH`, `YEAR`). Default `DAY`. Choose `NONE` to create non-partitioned tables.
   * **Partition Field** *(optional)* — a record field to partition by. Leave blank for ingestion-time partitioning.
   * **Clustering Fields** *(optional)* — comma-separated fields to cluster by (max 4).
   * **Partition Expiration in Days** *(optional)* — automatically delete partitions older than this many days from auto-created tables. Leave blank to keep all partitions. Ignored when Time Partitioning is `NONE`.
   * **Auto-create Tables** — create BigQuery tables for new topics automatically (on by default).
   * **Allow New Fields** / **Allow Required Field Relaxation** — let the table schema evolve as the source schema changes (both on by default).
   * **Tasks** — number of parallel tasks.
3. Click **Save**.

## Schema evolution

With **Allow New Fields** and **Allow Required Field Relaxation** enabled (the defaults), Streamkap keeps BigQuery tables in step with the source as it changes:

| Source change | BigQuery result |
| --- | --- |
| Add a column | Added as a `NULLABLE` column; existing rows are `NULL`. |
| Make a `NOT NULL` column nullable | Column relaxed `REQUIRED` → `NULLABLE`. |
| Widen a type (e.g. `INT`→`BIGINT`, longer `VARCHAR`) | No change — both map to the same BigQuery type. |

:::warning
BigQuery cannot **drop** or **rename** columns, and cannot add a `REQUIRED`
column to a populated table. A dropped source column is retained in BigQuery and
filled with `NULL`s going forward; a renamed column appears as a **new** column
alongside the old one. Avoid **incompatible type changes** (e.g. `VARCHAR`→`INT`),
which the connector cannot apply.
:::

:::info
Source column names that aren't valid BigQuery column names (for example names
with leading digits or unsupported characters) are adjusted automatically so the
data still loads — the BigQuery column may therefore differ slightly from the
source column name.
:::

## Data type mapping

Mappings are best-effort to the nearest BigQuery equivalent.

| Streamkap | BigQuery |
| --- | --- |
| `INT8`, `INT16`, `INT32`, `INT64` | `INTEGER` |
| `FLOAT32`, `FLOAT64` | `FLOAT` |
| `BOOLEAN` | `BOOLEAN` |
| `STRING` | `STRING` |
| `BYTES` | `BYTES` |
| `org.apache.kafka.connect.data.Decimal` | `NUMERIC` / `BIGNUMERIC` |
| `org.apache.kafka.connect.data.Date` | `DATE` |
| `org.apache.kafka.connect.data.Time` | `TIME` |
| `org.apache.kafka.connect.data.Timestamp`, `io.debezium.time.ZonedTimestamp` | `TIMESTAMP` |
| `io.debezium.data.Json` | `JSON` |
| `STRUCT` | `RECORD` |
| `ARRAY` | repeated field |

## Network access

BigQuery is reached over Google Cloud APIs, so there is no IP allowlist to configure — access is controlled entirely by the service account's IAM permissions.

## Related documentation

- [BigQuery Best Practices](/google-bigquery-best-practices) — partitioning, clustering, and deduplicating appended data
- [Error Reference](/error-reference) — common error codes and resolution steps