​

Prerequisites

• MariaDB version ≥ 11.4.3
• binlog configured
• Streamkap user and role

​

Granting Privileges

-- Replace { ... } placeholders as required;
CREATE USER streamkap_user@'%' IDENTIFIED BY '{password}';

-- Grant Permissions
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO streamkap_user@'%';

-- Grant Select on all schemas needed
GRANT SELECT ON {schema}.* TO 'streamkap_user';

​

Enable Snapshots

​

Method 1: Enable GTID (default)

​

Method 2: Create a table in the source database

-- Create the schema
CREATE SCHEMA streamkap;

CREATE TABLE streamkap.streamkap_signal (
  id VARCHAR(255) PRIMARY KEY, 
  type VARCHAR(32) NOT NULL, 
  data VARCHAR(2000) NULL
);

GRANT SELECT, UPDATE, INSERT ON streamkap.streamkap_signal TO streamkap_user;

​

Heartbeats

​

Layer 1: Connector heartbeats (enabled by default)

​

Layer 2: Source database heartbeats (recommended)

• Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
• Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table with id, text, and last_update fields
CREATE TABLE streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

-- Grant permission to the Streamkap user
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO STREAMKAP_USER;

-- Insert the first row into the heartbeat table
INSERT INTO streamkap_heartbeat (text) VALUES ('test_heartbeat');

SHOW VARIABLES WHERE VARIABLE_NAME = 'event_scheduler';

-- Create the streamkap schema
CREATE SCHEMA IF NOT EXISTS streamkap;

-- Switch to the streamkap schema
USE streamkap;

-- Create the heartbeat table
CREATE TABLE streamkap.streamkap_heartbeat (
    id INT AUTO_INCREMENT PRIMARY KEY,
    text TEXT,
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Insert the initial row
INSERT INTO streamkap.streamkap_heartbeat (text) VALUES ('test_heartbeat');

CREATE EVENT streamkap.streamkap_heartbeat_event
ON SCHEDULE EVERY 1 MINUTE
DO
  UPDATE streamkap.streamkap_heartbeat
  SET text = 'updated_heartbeat',
      last_update = CURRENT_TIMESTAMP
  WHERE id = 1;

GRANT EVENT ON streamkap.* TO {event scheduler user};
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO {event scheduler user};

-- Grant permissions to the Streamkap user for monitoring and diagnostics
GRANT SELECT, UPDATE, INSERT, DELETE ON streamkap.streamkap_heartbeat TO 'streamkap_user'@'%';

-- View all scheduled events in the streamkap schema
SHOW EVENTS IN streamkap;

-- View event details
SELECT * FROM information_schema.EVENTS WHERE EVENT_SCHEMA = 'streamkap';

-- Disable an event temporarily
ALTER EVENT streamkap.streamkap_heartbeat_event DISABLE;

-- Enable an event
ALTER EVENT streamkap.streamkap_heartbeat_event ENABLE;

-- Drop an event
DROP EVENT IF EXISTS streamkap.streamkap_heartbeat_event;

​

Enable binary logging

1. Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/
2. In the navigation pane, choose Databases, and then choose the DB instance or Multi-AZ DB cluster that you want to modify
3. Choose Modify
   1. For Backup retention period, choose a positive nonzero value, for example 3 days
4. Choose Continue
5. Choose Apply immediately
6. Choose Modify DB instance or Modify cluster to save your changes and enable automated backups

​

Configure binary logging

1. Open the Amazon RDS console at https://console.aws.amazon.com/rds/
2. In the navigation pane, choose Parameter groups
3. Choose the parameter group used by the DB instance you want to modify
4. You can’t modify a default parameter group. If the DB instance is using a default parameter group, create a new parameter group and associate it with the DB instance
5. From Parameter group actions, choose Edit
6. Set the binlog_format parameter to the binary logging format of ROW
7. Set the binlog_row_imageparameter to FULL
8. Choose Save changes to save the updates to the DB parameter group

​

Verify binary logs are enabled

• Check the parameter group for the DB instance and that log_bin parameter is ON
• Run the following SQL query on the DB instance SHOW VARIABLES LIKE '%log_bin%';. Result should be ON
• Run SHOW BINARY LOGS

​

Consider Access Restrictions

• Visit Connection Options to ensure Streamkap can reach your database

​

Setup MariaDB Connector in Streamkap

• Go to Sources and click Create New
• Input
  • Name for your Connector
  • Hostname
  • Port (Default 3306)
  • Username (Username you chose earlier, our scripts use streamkap_user)
  • Password
  • Read only
    • Whether or not to use a read-only connection. MariaDB has GTID enabled by default, so no additional configuration is necessary. See Enable GTID for more information.
    • If you have disabled GTID mode and cannot enable it, set Read only to No and create the signal table as described here.
      • Signal Table Database: Streamkap will use a table in this database to manage snapshots e.g. streamkap. See Enable Snapshots for more information.
  • Heartbeats: Enabled by default.
    • For read-write connections, configure a heartbeat table in the source database and set Heartbeat Table Database. See Heartbeats for setup instructions.
    • For read-only connections, configure a scheduled heartbeat event on the primary database using the MariaDB Event Scheduler, and include the heartbeat table in Add Schemas/Tables. See Heartbeats for setup instructions.
  • Connection Timezone - The timezone of your database
  • Timezone conversionMariaDB converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. By default, the current time zone for each connection is the database server’s time zone but this option allows you to override that.As long as the time zones remain the same, you get back the same value you store.We recommend using the default SERVER option which attempts to detect the session time zone from the values configured on the MariaDB server session variables ‘time_zone’ or ‘system_time_zone’. It also reduces the chance of problems with daylight savings adjustment ‘fall back’ and ‘spring forward’.If either time zones change, an ad-hoc snapshot is recommended so your source and destination timestamps are consistent.
  • Connect via SSH Tunnel. See SSH Tunnel
• Advanced Parameters
  • Represent Binary Data As (Default bytes)
  • Capture Only Captured Databases DDL (Default false) - Specifies whether the connector records schema structures from all databases defined in the server (the default) or only those databases for which you’ve explicitly configured the connector. Specify true to capture schema history only for the specific databases you’ve configured. This is particularly valuable when databases are large, to reduce the volume of DDL stored in the schema history topic. It also improves startup times when the connector restarts or recovers from failures. See Schema History Optimization for details.
  • Capture Only Captured Tables DDL (Default false) - Specifies whether the connector records the schema structure for all tables in the configured databases (the default) or only the tables whose changes the connector captures. Specify true to capture schema history only for the specific tables you’ve configured. This is particularly valuable when tables are large, to reduce the volume of DDL statements stored in the schema history topic. It also improves startup times when the connector restarts or recovers from failures. See Schema History Optimization for details.
• Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.

• Click Save The connector will take approximately 1 minute to start processing data.