Prerequisites

• MySQL version ≥ 5.7
• MySQL binlog enabled
• Streamkap user and role

Granting Privileges

It's recommended to create a separate user and role for Streamkap to access your MySQL database. Below is an example script that does that.

-- Replace { ... } placeholders as required

-- Identify version
SHOW VARIABLES LIKE 'VERSION';

--On MySQL version 5.6 to 8.0
CREATE USER streamkap_user@'%' IDENTIFIED BY '{password}';

--On MySQL version 8.0+
CREATE USER streamkap_user@'%' IDENTIFIED WITH mysql_native_password BY '{password}';

--Grant Permissions
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO streamkap_user@'%';

--Grant Select on all schemas needed
GRANT SELECT ON {schema}.* TO 'streamkap_user';

Enable Snapshots

You can perform ad-hoc snapshots of all or some of your tables in the Streamkap app. See Snapshots & Backfilling for more information.

This feature is available without any additional configuration because 'GTID-based replication' is enabled by default - and cannot be disabled - for MySQL Cloud SQL instances. See About replication in Cloud SQL for more information.

Configure binary logging

📘

If you are using a read replica, you must enable binary logging on the read replica

1. In the Google Cloud console, go to the Cloud SQL Instances page.
2. Go to Cloud SQL Instances
3. Open the more actions menu for the instance you want to enable point-in-time recovery on and click Edit.
4. Under Customize your instance, expand the Data Protection section.
5. Select the Enable point-in-time recovery checkbox.
6. Expand Advanced options.
7. Enter the number of days to retain logs, from 3-7. We recommend 7 days,
8. Click Save.

Consider Access Restrictions

• Visit Connection Options to ensure Streamkap can reach your database

Setup MySQL Connector in Streamkap

• Go to Sources and click Create New
• Input
  • Name for your Connector
  • Hostname
  • Port (Default 3306)
  • Username (Username you chose earlier, our scripts use streamkap_user)
  • Password
  • Heartbeat - Required for low volume connectors. See MySQL Heartbeats
  • Connection Timezone - The timezone of your database

  • 📘

    Timezone conversion

    MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. By default, the current time zone for each connection is the database server's time zone but this option allows you to override that.

    As long as the time zones remain the same, you get back the same value you store.

    We recommend using the default SERVER option which attempts to detect the session time zone from the values configured on the MySQL server session variables 'time_zone' or 'system_time_zone'. It also reduces the chance of problems with daylight savings adjustment 'fall back' and 'spring forward'.

    If either time zones change, an ad-hoc snapshot is recommended so your source and destination timestamps are consistent.

• Connect via SSH Tunnel. See SSH Tunnel
• Advanced Parameters
  • Snapshot Mode (Default When Needed) See MySQL Snapshot Modes for more information
  • Represent Binary Data As (Default bytes)
  • Snapshot Chunk Size (Default 1024) - This is the number of rows read at a time when snapshotting. This is a low safe value. As a guide, if you have 100m + rows of data you may want to move this to 5120. If you have 1bn then a higher number still will allow you to backfill faster.
  • Max Batch Size (Default 2048) - A value that specifies the maximum size of each batch of events that the connector processes. Only increase if experiencing lag
• Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.
• Click Save
  The connector will take approximately 1 minute to start processing data.