MariaDB (Generic)

Connect a self-hosted or generic MariaDB server to Streamkap for change data capture, including binary log configuration, user grants, and heartbeats.

Prerequisites

MariaDB version ≥ 11.4.3
MariaDB binlog enabled on the primary server
Connection details
Streamkap user and role

Quick Reference: Minimum Required Permissions

The following table summarizes all permissions the Streamkap user needs. The setup steps on each source connector page walk through granting each one.

Core permissions (always required):

Permission	Scope	Purpose
`REPLICATION CLIENT`	`.`	Read binlog metadata and positions
`REPLICATION SLAVE`	`.`	Read binlog events for CDC streaming
`RELOAD`	`.`	Flush operations required for consistent snapshots
`SHOW DATABASES`	`.`	Discover available databases and schemas
`SELECT`	`{schema}.*`	Read table data during initial and incremental snapshots

Snapshot signal table (required if GTID is disabled):

Permission	Scope	Purpose
`SELECT`	`streamkap.streamkap_signal`	Read signal table state
`INSERT`	`streamkap.streamkap_signal`	Trigger snapshot signals
`UPDATE`	`streamkap.streamkap_signal`	Update signal table state
`DELETE`	`streamkap.streamkap_signal`	Clean up processed signals

Heartbeat table (required if heartbeats are enabled):

Permission	Scope	Purpose
`SELECT`	`streamkap.streamkap_heartbeat`	Read heartbeat state
`INSERT`	`streamkap.streamkap_heartbeat`	Write heartbeat records
`UPDATE`	`streamkap.streamkap_heartbeat`	Update heartbeat timestamps
`DELETE`	`streamkap.streamkap_heartbeat`	Clean up old heartbeat records

Combined GRANT statements:

-- Core permissions (always required)
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO 'streamkap_user'@'%';
GRANT SELECT ON {schema}.* TO 'streamkap_user'@'%';

-- Signal table (if GTID is disabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_signal TO 'streamkap_user'@'%';

-- Heartbeat table (if heartbeats are enabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

Granting Privileges

It’s recommended to create a separate user and role for Streamkap to access your MariaDB database. Below is an example script that does that.

-- Replace { ... } placeholders as required

-- Create user
CREATE USER streamkap_user@'%' IDENTIFIED BY 'password';

-- Grant permissions
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO streamkap_user@'%';

-- Grant select on all schemas needed
GRANT SELECT ON {schema}.* TO 'streamkap_user'@'%';

Enable Snapshots

You can perform ad-hoc snapshots of all or some of your tables in the Streamkap app. See Snapshots & Backfilling for more information.

To enable this feature, there are 2 methods available for MariaDB databases.

Method 1: Enable GTID (default)

Global transaction identifiers (GTIDs) uniquely identify transactions that occur on a server within a cluster. Though not required, using GTIDs simplifies replication and enables you to more easily confirm if primary and replica servers are consistent as well as carry out incremental snapshots.

For MariaDB, this is enabled by default, no additional setup is necessary.

Method 2: Create a table in the source database

If for some reason you have disabled GTIDs and cannot enable them, you will need to create the table and give permissions to the streamkap_user. Streamkap will use this collection for managing snapshots.

-- Create the schema
CREATE SCHEMA streamkap;

CREATE TABLE streamkap.streamkap_signal (
  id VARCHAR(255) PRIMARY KEY, 
  type VARCHAR(32) NOT NULL, 
  data VARCHAR(2000) NULL
);

GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_signal TO 'streamkap_user'@'%';

Heartbeats

Connectors use “offsets”—like bookmarks—to track their position in the database’s log or change stream. When no changes occur for long periods, these offsets may become outdated, and the Connector might lose its place or stop capturing changes.

Heartbeats ensure the Connector stays active and continues capturing changes.

There are two layers of heartbeat protection:

Layer 1: Connector heartbeats (enabled by default)

The Connector periodically emits heartbeat messages to an internal topic, even when no actual data changes are detected. This keeps offsets fresh and prevents staleness.

No configuration is necessary for this layer; it is automatically enabled. We recommend keeping this layer enabled for all deployments.

Layer 2: Source database heartbeats (recommended)

You can configure regular updates to a dedicated heartbeat table in the source database. This simulates activity, ensuring change events are generated consistently, maintaining log progress and providing additional resilience.

How this layer is configured depends on the connection type (if supported by the Source):

Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.

This layer requires you to set up a heartbeat table—and for read-only connections, a scheduled job (e.g., pg_cron for PostgreSQL, event_scheduler for MySQL)—on your source database.

For read-write connections (when Read only is No during Streamkap Setup), the Connector writes to the heartbeat table directly.

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table with id, text, and last_update fields
CREATE TABLE streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

-- Grant permission to the Streamkap user
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

-- Insert the first row into the heartbeat table
INSERT INTO streamkap_heartbeat (text) VALUES ('test_heartbeat');

For read-only connections (when Read only is Yes during Streamkap Setup), the Connector cannot write to the heartbeat table directly. Instead, you must configure a scheduled job on the primary database to generate artificial traffic. These changes will replicate to the read replica, which the Connector then consumes.

Enable the event scheduler

The MariaDB Event Scheduler flag must be enabled on your database (event_scheduler=ON). See your provider’s documentation:

Amazon RDS

Modifying parameters in a DB parameter group in Amazon RDS

MariaDB Documentation

Event Scheduler documentation

Check if the event scheduler is enabled:

SHOW VARIABLES WHERE VARIABLE_NAME = 'event_scheduler';

Create the heartbeat table

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table
CREATE TABLE streamkap.streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Insert the initial row
INSERT INTO streamkap.streamkap_heartbeat (text) VALUES ('test_heartbeat');

Create the scheduled event

CREATE EVENT streamkap.streamkap_heartbeat_event
ON SCHEDULE EVERY 1 MINUTE
DO
  UPDATE streamkap.streamkap_heartbeat
  SET text = 'updated_heartbeat',
      last_update = CURRENT_TIMESTAMP
  WHERE id = 1;

Grant permissions

Whichever database user is used to create and run the event scheduler (often the MariaDB root user or a dedicated event scheduler user) needs appropriate permissions on the heartbeat table. Additionally, the Streamkap user also needs permissions to monitor the heartbeat table.

GRANT EVENT ON streamkap.* TO {event scheduler user};
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO {event scheduler user};

-- Grant permissions to the Streamkap user for monitoring and diagnostics
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

Useful event scheduler commands

-- View all scheduled events in the streamkap schema
SHOW EVENTS IN streamkap;

-- View event details
SELECT * FROM information_schema.EVENTS WHERE EVENT_SCHEMA = 'streamkap';

-- Disable an event temporarily
ALTER EVENT streamkap.streamkap_heartbeat_event DISABLE;

-- Enable an event
ALTER EVENT streamkap.streamkap_heartbeat_event ENABLE;

-- Drop an event
DROP EVENT IF EXISTS streamkap.streamkap_heartbeat_event;

Update Server Configuration File

Open a connection to your MariaDB database’s server. Access your MariaDB server configuration file (usually /etc/my.cnf). These lines enable ROW format binary log replication which Streamkap needs to perform incremental updates.

Enable binary logging
Set a unique server-id number if not set already. If your configuration already has a server-id entry, you don’t need to change it. Otherwise, choose a number between 1 and 4294967295 as the server-id.
Set a minimum 3 days for log expiry (default is 30 days)

binlog_format=ROW
binlog_row_image=FULL
log_bin=mariadb-binlog  
server-id=123456789
binlog_expire_logs_seconds=259200

Restart your MariaDB server for these changes to take effect

Validate binlog row value options

To enable the connector to consume UPDATE events, this variable must be set to a value other than PARTIAL_JSON.

Check current variable value: show global variables where variable_name = 'binlog_row_value_options';
If the value of the variable is set to PARTIAL_JSON, run the following command to unset it: set @@global.binlog_row_value_options="";

Verify binary logs are enabled

You can either:

Run the following SQL query on the DB instance SHOW VARIABLES LIKE '%log_bin%';. Result should be ON
Run SHOW BINARY LOGS

Consider Access Restrictions

Visit Connection Options to ensure Streamkap can reach your database

Setup MariaDB Connector in Streamkap

Go to Sources and click Create New
Input
- Name for your Connector
- Hostname
- Port (Default 3306)
- Username (Username you chose earlier, our scripts use streamkap_user)
- Password
- Read only
  - Whether or not to use a read-only connection. MariaDB has GTID enabled by default, so no additional configuration is necessary. See Enable GTID for more information.
  - If you have disabled GTID mode and cannot enable it, set Read only to No and create the signal table as described here.
    - Signal Table: Full path to the signal table including database and table name (e.g., streamkap.streamkap_signal). This table is used for incremental snapshotting. See Enable Snapshots for setup instructions.
- Heartbeats: Enabled by default.
  - For read-write connections, configure a heartbeat table in the source database and set Heartbeat Table Database. See Heartbeats for setup instructions.
  - For read-only connections, configure a scheduled heartbeat event on the primary database using the MariaDB Event Scheduler, and include the heartbeat table in Add Schemas/Tables. See Heartbeats for setup instructions.
- Connection Timezone - The timezone of your database
- Timezone conversion
  MariaDB converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. By default, the current time zone for each connection is the database server’s time zone but this option allows you to override that.
  As long as the time zones remain the same, you get back the same value you store.
  We recommend using the default SERVER option which attempts to detect the session time zone from the values configured on the MariaDB server session variables time_zone or system_time_zone. It also reduces the chance of problems with daylight savings adjustment ‘fall back’ and ‘spring forward’.
  If either time zones change, an ad-hoc snapshot is recommended so your source and destination timestamps are consistent.
Connect via SSH Tunnel. See SSH Tunnel
Advanced Parameters
- Represent Binary Data As (Default bytes)
- Capture Only Captured Databases DDL (Default false) - Used to control whether the connector records schema structures from all databases defined in the server (the default) or only those databases for which you’ve explicitly configured the connector. Specify true to capture schema history only for the specific databases you’ve configured. This is particularly valuable when databases are large, to reduce the volume of DDL stored in the schema history topic. It also improves startup times when the connector restarts or recovers from failures. See Schema History Optimization for details.
- Capture Only Captured Tables DDL (Default false) - Used to control whether the connector records the schema structure for all tables in the configured databases (the default) or only the tables whose changes the connector captures. Specify true to capture schema history only for the specific tables you’ve configured. This is particularly valuable when tables are large, to reduce the volume of DDL statements stored in the schema history topic. It also improves startup times when the connector restarts or recovers from failures. See Schema History Optimization for details.
Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.

VIRTUAL generated columns are not streamed

MariaDB VIRTUAL generated columns are computed when rows are read and are not written to the binary log, so live streaming cannot capture or update their values. (PERSISTENT/stored generated columns are written to the binary log and stream normally.)

Snapshots read the table directly, so they do capture a value for VIRTUAL columns — but because streaming cannot keep it current, that value will become stale. We recommend excluding VIRTUAL generated columns from capture: in the Source’s Settings → Advanced section, set Column Selection Mode to Exclusion and add the column to the Column Exclusion List (fully-qualified, e.g. database.table.column). If you need the value at your destination, calculate it there — or with a transform — from the columns it is derived from.

Click Save The connector will take approximately 1 minute to start processing data.