> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamkap.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Configure Google BigQuery as a Streamkap CDC destination: prerequisites, IAM permissions, service account JSON key, Storage Write API, and schema evolution.

Stream change data into Google BigQuery. Streamkap writes using the
[BigQuery Storage Write API](https://cloud.google.com/bigquery/docs/write-api) — the
lowest-cost, high-throughput ingestion path.

## Prerequisites

* A Google Cloud project.
* A BigQuery **dataset** that already exists — Streamkap writes tables into it but does **not** create the dataset for you.
* A Google **service account** with a **JSON** key (P12 keys are not supported).
* The service account granted the role below.

## Required permissions

Grant the service account access **on the target dataset** (least privilege), not the whole project:

* **BigQuery Data Editor** (`roles/bigquery.dataEditor`) on the dataset — covers everything Streamkap needs: writing rows, creating tables, and evolving table schemas.

<Info>
  Prefer a custom role? The minimum permissions are `bigquery.datasets.get`,
  `bigquery.tables.get`, `bigquery.tables.list`, `bigquery.tables.updateData`, plus
  `bigquery.tables.create` (auto-create tables) and `bigquery.tables.update` (schema
  evolution).
</Info>

## Delivery guarantees

Streamkap provides **at-least-once** delivery and appends rows via the Storage Write API, so retries can write the same change more than once — a table can contain **duplicate rows for the same key**. Query the latest version per key (or schedule a cleanup) as shown in [Best Practices](/google-bigquery-best-practices).

## Google Cloud setup

Replace the `{ ... }` placeholders. You can use the Cloud Console or the `gcloud`/`bq` CLI:

<CodeGroup>
  ```bash gcloud theme={null}
  # 1. Create the dataset (if it doesn't exist yet)
  bq --location={REGION} mk --dataset {PROJECT_ID}:{DATASET}

  # 2. Create a service account
  gcloud iam service-accounts create streamkap-bigquery \
    --display-name="Streamkap BigQuery" --project={PROJECT_ID}

  # 3. Grant BigQuery Data Editor on the dataset (least privilege).
  #    Easiest via Console: BigQuery > {DATASET} > Sharing > Permissions >
  #    Add principal = the service account email, role = BigQuery Data Editor.

  # 4. Create a JSON key
  gcloud iam service-accounts keys create key.json \
    --iam-account=streamkap-bigquery@{PROJECT_ID}.iam.gserviceaccount.com
  ```
</CodeGroup>

<Warning>
  The dataset must exist before you create the destination. Streamkap creates and
  evolves **tables** inside it, but never creates the dataset.
</Warning>

## Streamkap setup

1. Go to **Destinations** and choose **BigQuery**.
2. Enter the configuration:
   * **Name** — a unique, memorable name for this destination.
   * **JSON key file** — upload the service-account JSON key you created.
   * **Dataset Name** — the existing dataset to write into. The GCP project is read automatically from the key file.
   * **Time Partitioning** — partition granularity for auto-created tables (`DAY`, `HOUR`, `MONTH`, `YEAR`). Default `DAY`. Choose `NONE` to create non-partitioned tables.
   * **Partition Field** *(optional)* — a record field to partition by. Leave blank for ingestion-time partitioning.
   * **Clustering Fields** *(optional)* — comma-separated fields to cluster by (max 4).
   * **Partition Expiration in Days** *(optional)* — automatically delete partitions older than this many days from auto-created tables. Leave blank to keep all partitions. Ignored when Time Partitioning is `NONE`.
   * **Auto-create Tables** — create BigQuery tables for new topics automatically (on by default).
   * **Allow New Fields** / **Allow Required Field Relaxation** — let the table schema evolve as the source schema changes (both on by default).
   * **Tasks** — number of parallel tasks.
3. Click **Save**.

## Schema evolution

With **Allow New Fields** and **Allow Required Field Relaxation** enabled (the defaults), Streamkap keeps BigQuery tables in step with the source as it changes:

| Source change                                        | BigQuery result                                         |
| ---------------------------------------------------- | ------------------------------------------------------- |
| Add a column                                         | Added as a `NULLABLE` column; existing rows are `NULL`. |
| Make a `NOT NULL` column nullable                    | Column relaxed `REQUIRED` → `NULLABLE`.                 |
| Widen a type (e.g. `INT`→`BIGINT`, longer `VARCHAR`) | No change — both map to the same BigQuery type.         |

<Warning>
  BigQuery cannot **drop** or **rename** columns, and cannot add a `REQUIRED`
  column to a populated table. A dropped source column is retained in BigQuery and
  filled with `NULL`s going forward; a renamed column appears as a **new** column
  alongside the old one. Avoid **incompatible type changes** (e.g. `VARCHAR`→`INT`),
  which the connector cannot apply.
</Warning>

<Info>
  Source column names that aren't valid BigQuery column names (for example names
  with leading digits or unsupported characters) are adjusted automatically so the
  data still loads — the BigQuery column may therefore differ slightly from the
  source column name.
</Info>

## Data type mapping

Mappings are best-effort to the nearest BigQuery equivalent.

| Streamkap                                                                    | BigQuery                 |
| ---------------------------------------------------------------------------- | ------------------------ |
| `INT8`, `INT16`, `INT32`, `INT64`                                            | `INTEGER`                |
| `FLOAT32`, `FLOAT64`                                                         | `FLOAT`                  |
| `BOOLEAN`                                                                    | `BOOLEAN`                |
| `STRING`                                                                     | `STRING`                 |
| `BYTES`                                                                      | `BYTES`                  |
| `org.apache.kafka.connect.data.Decimal`                                      | `NUMERIC` / `BIGNUMERIC` |
| `org.apache.kafka.connect.data.Date`                                         | `DATE`                   |
| `org.apache.kafka.connect.data.Time`                                         | `TIME`                   |
| `org.apache.kafka.connect.data.Timestamp`, `io.debezium.time.ZonedTimestamp` | `TIMESTAMP`              |
| `io.debezium.data.Json`                                                      | `JSON`                   |
| `STRUCT`                                                                     | `RECORD`                 |
| `ARRAY`                                                                      | repeated field           |

## Network access

BigQuery is reached over Google Cloud APIs, so there is no IP allowlist to configure — access is controlled entirely by the service account's IAM permissions.

## Related documentation

* [BigQuery Best Practices](/google-bigquery-best-practices) — partitioning, clustering, and deduplicating appended data
* [Error Reference](/error-reference) — common error codes and resolution steps