MySQL (Generic)

Connect a self-hosted or generic MySQL server to Streamkap for binlog-based change data capture, including server configuration, user grants, and heartbeats.

Prerequisites

MySQL version ≥ 5.7
MySQL binlog enabled on the primary server
Connection details including Server ID
A database user with sufficient privileges to configure the database, including enabling binary logging and creating users

Quick Reference: Minimum Required Permissions

The following table summarizes all permissions the Streamkap user needs. The setup steps on each source connector page walk through granting each one.

Core permissions (always required):

Permission	Scope	Purpose
`REPLICATION CLIENT`	`.`	Read binlog metadata and positions
`REPLICATION SLAVE`	`.`	Read binlog events for CDC streaming
`RELOAD`	`.`	Flush operations required for consistent snapshots
`SHOW DATABASES`	`.`	Discover available databases and schemas
`SELECT`	`{schema}.*`	Read table data during initial and incremental snapshots

Snapshot signal table (required if GTID is not enabled):

Permission	Scope	Purpose
`SELECT`	`streamkap.streamkap_signal`	Read signal table state
`INSERT`	`streamkap.streamkap_signal`	Trigger snapshot signals
`UPDATE`	`streamkap.streamkap_signal`	Update signal table state
`DELETE`	`streamkap.streamkap_signal`	Clean up processed signals

Heartbeat table (required if heartbeats are enabled):

Permission	Scope	Purpose
`SELECT`	`streamkap.streamkap_heartbeat`	Read heartbeat state
`INSERT`	`streamkap.streamkap_heartbeat`	Write heartbeat records
`UPDATE`	`streamkap.streamkap_heartbeat`	Update heartbeat timestamps
`DELETE`	`streamkap.streamkap_heartbeat`	Clean up old heartbeat records

Combined GRANT statements:

-- Core permissions (always required)
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO 'streamkap_user'@'%';
GRANT SELECT ON {schema}.* TO 'streamkap_user'@'%';

-- Signal table (if GTID is not enabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_signal TO 'streamkap_user'@'%';

-- Heartbeat table (if heartbeats are enabled)
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

MySQL Setup

1. Grant Database Access

Configure one of the Connection Options to ensure Streamkap can reach your database.

2. Configure Binary Logging

Binary logging records all changes to your database tables. The Connector relies on MySQL’s implementation of this.

Open a connection to your MySQL database’s server. Access your MySQL server configuration file (usually /etc/my.cnf). Ensure that this file has the following lines in the [mysqld] section. These lines enable ROW format binary log replication, which the Connector needs to perform incremental updates.

Enable binlog
Set Server-ID if not set already
Set minimum 3 days for log expiry

[mysqld]
binlog-format=ROW
log-bin=mysql-binlog
server-id=123456789
expire-logs-days=3
log-slave-updates=1

If your configuration already has a server-id entry, you don’t need to change it. Otherwise, choose any number between 1 and 4294967295 as the server-id.

Restart your MySQL server to effect these changes.

3. Verify Binary Logs Are Enabled

You can verify using any of these methods:

Run the following SQL query on the DB instance SHOW VARIABLES LIKE '%log_bin%';. Result should be ON
Run SHOW BINARY LOGS

4. Create Database User

It’s recommended to create a separate user and role for the Connector to access your MySQL database. Below is an example script that does that.

-- Replace { ... } placeholders as required

-- Create user
CREATE USER 'streamkap_user'@'%' IDENTIFIED WITH mysql_native_password BY '{password}';

-- Grant Permissions
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO 'streamkap_user'@'%';

-- Grant Select on all schemas needed
GRANT SELECT ON {schema}.* TO 'streamkap_user'@'%';

5. Enable Snapshots

To backfill your data, the Connector needs to be able to perform snapshots. See Snapshots & Backfilling for more information.

To enable this feature, there are 2 methods available:

Method 1: Enable GTID (Recommended)

Global transaction identifiers (GTIDs) uniquely identify transactions that occur on a server within a cluster. Though not required, using GTIDs simplifies replication and enables you to more easily confirm if primary and replica servers are consistent as well as carry out incremental snapshots.

Set up following these instructions. Ensure you follow the guide for your version Replication GTIDs - How To and that GTID mode is ON.

Method 2: Create a table in the source database

If you cannot enable GTID mode, you will need to create the table and give permissions to the streamkap_user. The Connector will use this collection for managing snapshots.

-- Create the schema
CREATE SCHEMA streamkap;

CREATE TABLE streamkap_signal (
  id VARCHAR(255) PRIMARY KEY,
  type VARCHAR(32) NOT NULL,
  data VARCHAR(2000) NULL
);

GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_signal TO 'streamkap_user'@'%';

6. Heartbeats

Connectors use “offsets”—like bookmarks—to track their position in the database’s log or change stream. When no changes occur for long periods, these offsets may become outdated, and the Connector might lose its place or stop capturing changes.

Heartbeats ensure the Connector stays active and continues capturing changes.

There are two layers of heartbeat protection:

Layer 1: Connector heartbeats (enabled by default)

The Connector periodically emits heartbeat messages to an internal topic, even when no actual data changes are detected. This keeps offsets fresh and prevents staleness.

No configuration is necessary for this layer; it is automatically enabled. We recommend keeping this layer enabled for all deployments.

Layer 2: Source database heartbeats (recommended)

You can configure regular updates to a dedicated heartbeat table in the source database. This simulates activity, ensuring change events are generated consistently, maintaining log progress and providing additional resilience.

How this layer is configured depends on the connection type (if supported by the Source):

Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.

This layer requires you to set up a heartbeat table—and for read-only connections, a scheduled job (e.g., pg_cron for PostgreSQL, event_scheduler for MySQL)—on your source database.

For read-write connections (when Read only is No during Streamkap Setup), the Connector writes to the heartbeat table directly.

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table with id, text, and last_update fields
CREATE TABLE streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Grant permission to the Streamkap user
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

-- Insert the first row into the heartbeat table
INSERT INTO streamkap_heartbeat (text) VALUES ('test_heartbeat');

For read-only connections (when Read only is Yes during Streamkap Setup), the Connector cannot write to the heartbeat table directly. Instead, you must configure a scheduled job on the primary database to generate artificial traffic. These changes will replicate to the read replica, which the Connector then consumes.

Enable the event scheduler

The MySQL Event Scheduler flag must be enabled on your database (event_scheduler=ON). See your provider’s documentation:

Amazon RDS

Modifying parameters in a DB parameter group in Amazon RDS

Amazon Aurora

Aurora MySQL configuration parameters reference

Google Cloud SQL

Configuring database flags

Azure Database

Server parameters

Check if the event scheduler is enabled:

SHOW VARIABLES WHERE VARIABLE_NAME = 'event_scheduler';

Create the heartbeat table

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Create the heartbeat table
CREATE TABLE streamkap.streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Insert the initial row
INSERT INTO streamkap.streamkap_heartbeat (text) VALUES ('test_heartbeat');

Create the scheduled event

CREATE EVENT streamkap.streamkap_heartbeat_event
ON SCHEDULE EVERY 1 MINUTE
DO
  UPDATE streamkap.streamkap_heartbeat
  SET text = 'updated_heartbeat',
      last_update = CURRENT_TIMESTAMP
  WHERE id = 1;

Grant permissions

Whichever database user is used to create and run the event scheduler (often the MySQL root user or a dedicated event scheduler user) needs appropriate permissions on the heartbeat table. Additionally, the Streamkap user also needs permissions to monitor the heartbeat table.

GRANT EVENT ON streamkap.* TO {event scheduler user};
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO {event scheduler user};

-- Grant permissions to the Streamkap user for monitoring and diagnostics
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

Useful event scheduler commands

-- View all scheduled events in the streamkap schema
SHOW EVENTS IN streamkap;

-- View event details
SELECT * FROM information_schema.EVENTS WHERE EVENT_SCHEMA = 'streamkap';

-- Disable an event temporarily
ALTER EVENT streamkap.streamkap_heartbeat_event DISABLE;

-- Enable an event
ALTER EVENT streamkap.streamkap_heartbeat_event ENABLE;

-- Drop an event
DROP EVENT IF EXISTS streamkap.streamkap_heartbeat_event;

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Source

Navigate to Add Connectors.
Choose MySQL.

2. Connection Settings

Name: Enter a name for your connector.
Hostname: Specify the hostname.
Port: Default is 3306.
Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your database. This is necessary if the Connector cannot connect directly to your database.
- See SSH Tunnel for setup instructions.
Username: Username to access the database. By default, Streamkap scripts use streamkap_user.
Password: Password to access the database.
Read only: Whether or not to use a read-only connection. Requires GTID to be enabled on the source database. See Enable GTID for more information.
Heartbeats: Enabled by default.
- For read-write connections, configure a heartbeat table in the source database and set Heartbeat Table Database. See Heartbeats for setup instructions.
- For read-only connections, configure a scheduled heartbeat event on the primary database using the MySQL Event Scheduler, and include the heartbeat table in Schema and Table Capture. See Heartbeats for setup instructions.
Connection Timezone: The timezone of your database.

3. Snapshot Settings

Signal Table: Full path to the signal table including database and table name (e.g., streamkap.streamkap_signal). This table is used for incremental snapshotting. See Enable Snapshots for setup instructions.

4. Advanced Parameters

Represent binary data as: Specifies how the data for binary columns e.g. blob, binary, varbinary should be interpreted. Your destination for this data can impact which option you choose. Default is bytes.
Capture Only Captured Databases DDL: Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. Default is false. See Schema History Optimization for details.
Capture Only Captured Tables DDL: Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. Default is false. See Schema History Optimization for details.

Click Next.

5. Schema and Table Capture

Add Schemas/Tables: Specify the schema(s) and table(s) for capture.
- You can bulk upload here. The format is a simple list of schemas and tables, with each entry on a new row. Save as a .csv file without a header.

VIRTUAL generated columns are not streamed

MySQL VIRTUAL generated columns are computed when rows are read and are not written to the binary log, so live streaming cannot capture or update their values. (STORED generated columns are written to the binary log and stream normally.)

Snapshots read the table directly, so they do capture a value for VIRTUAL columns — but because streaming cannot keep it current, that value will become stale. We recommend excluding VIRTUAL generated columns from capture: in the Source’s Settings → Advanced section, set Column Selection Mode to Exclusion and add the column to the Column Exclusion List (fully-qualified, e.g. database.table.column). If you need the value at your destination, calculate it there — or with a transform — from the columns it is derived from.

Click Save.