Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Databricks Delta Lake

Stream change data capture records into Databricks Delta Lake from Streamkap via JDBC, with upsert merge and append insert ingestion modes.

Streamkap Setup

To set up the Connector, you will need to gather connection details and configure your DataBricks cluster. Log in to your Databricks Cloud Account and then follow the steps below.

Get connection details

Streamkap connects to Databricks via a JDBC URL. You can use either an All-Purpose Compute or a SQL Warehouse as the compute resource.

Option A: All-Purpose Compute

  1. Open the Compute page from the sidebar and choose your cluster
  2. Click on Advanced Options
  3. Open the JDBC/ODBC tab
  4. Copy the JDBC Connection URL

Option B: SQL Warehouse

A SQL Warehouse can automatically scale across multiple Spark clusters to handle concurrent workloads, but is generally more expensive than an All-Purpose Cluster.

To get the JDBC URL for a SQL Warehouse:

  1. Open the SQL Warehouses page from the sidebar
  2. Select your warehouse
  3. Open the Connection Details tab
  4. Copy the JDBC URL

Generate an access token

For setting the Streamkap DataBricks’ Token:

  1. Open Settings page from the sidebar and then User Settings
  2. Open the Personal Access Tokens tab
  3. Click + Generate New Token
  4. (Optional) Enter a comment and change the token lifetime
  5. Click Generate
  6. Copy the access token

Create a staging location

Streamkap writes each batch to a Parquet file and uploads it to a staging location before loading it into your table. Pick one of the two options below.

  1. Create a volume in the catalog and schema you are loading into, for example main.streamkap.stage

  2. Grant the identity that owns your access token the following on that volume:

    GRANT USE CATALOG ON CATALOG main TO `<your principal>`;
GRANT USE SCHEMA ON SCHEMA main.streamkap TO `<your principal>`;
GRANT READ VOLUME ON VOLUME main.streamkap.stage TO `<your principal>`;
GRANT WRITE VOLUME ON VOLUME main.streamkap.stage TO `<your principal>`;

  3. Set Staging Volume Path on the Streamkap destination to the volume path, for example /Volumes/main/streamkap/stage/

Option B: DBFS root (legacy workspaces only)

  1. Create a tmp directory on the Databricks File System (DBFS)
  2. Leave Staging Volume Path blank

How it works

As data’s streamed from the source in to topics (think of them as partitioned tables), the Databricks Sink connector will:

  • Check whether tables for the topics exist in Databricks, if not, it creates them

  • Automatically handle schema evolution when the source schema changes (e.g. new columns, data type changes)

  • Stream change data into Parquet files and upload them to your staging location (a Unity Catalog volume, or the tmp directory on DBFS) and:

    • Load data to the target table using SQL bulk import COPY
    • Clean up the Parquet files

Ingestion Modes

Streamkap supports two ingestion modes for writing data to Databricks Delta Lake: Upsert and Append.

Upsert

Upsert mode uses a MERGE INTO statement to insert new records and update existing ones based on the primary key columns from the source table.

  • New records (no matching primary key in the target) are inserted
  • Existing records (matching primary key) are updated with the latest values
  • Deleted records (when hard delete is enabled) are physically removed from the target table
  • Out-of-order protection: Streamkap tracks record timestamps and offsets to ensure older records never overwrite newer data

Upsert is the recommended mode for most use cases, as it keeps your target table in sync with the source and handles updates and deletes automatically.

Append

Append mode uses a simple INSERT INTO statement to add all incoming records as new rows.

  • Every record is inserted regardless of whether a row with the same key already exists
  • No deduplication or update logic is applied
  • Deletes from the source are not reflected in the target

Append is useful for event logs, audit trails, or any scenario where you want to preserve every change as a separate row rather than maintaining a current-state replica.