Google Cloud MySQL
MySQL Change Data Capture Setup on Google Cloud SQL with Streamkap
Prerequisites
- MySQL version ≥ 5.7
- MySQL binlog enabled
- Streamkap user and role
Granting Privileges
It's recommended to create a separate user and role for Streamkap to access your MySQL database. Below is an example script that does that.
-- Replace { ... } placeholders as required
-- Identify version
SHOW VARIABLES LIKE 'VERSION';
--On MySQL version 5.6 to 8.0
CREATE USER streamkap_user@'%' IDENTIFIED BY '{password}';
--On MySQL version 8.0+
CREATE USER streamkap_user@'%' IDENTIFIED WITH mysql_native_password BY '{password}';
--Grant Permissions
GRANT REPLICATION CLIENT, RELOAD, SHOW DATABASES, REPLICATION SLAVE ON *.* TO streamkap_user@'%';
--Grant Select on all schemas needed
GRANT SELECT ON {schema}.* TO 'streamkap_user';
Enable Snapshots
You can perform ad-hoc snapshots of all or some of your tables in the Streamkap app. See Snapshots & Backfilling for more information.
This feature is available without any additional configuration because 'GTID-based replication' is enabled by default - and cannot be disabled - for MySQL Cloud SQL instances. See About replication in Cloud SQL for more information.
Configure binary logging
If you are using a read replica, you must enable binary logging on the read replica
- In the Google Cloud console, go to the Cloud SQL Instances page.
- Go to Cloud SQL Instances
- Open the more actions menu for the instance you want to enable point-in-time recovery on and click Edit.
- Under Customize your instance, expand the Data Protection section.
- Select the Enable point-in-time recovery checkbox.
- Expand Advanced options.
- Enter the number of days to retain logs, from 3-7. We recommend 7 days,
- Click Save.
Consider Access Restrictions
- Visit Connection Options to ensure Streamkap can reach your database
Setup MySQL Connector in Streamkap
- Go to Sources and click Create New
- Input
- Name for your Connector
- Hostname
- Port (Default
3306
) - Username (Username you chose earlier, our scripts use
streamkap_user
) - Password
- Timezone - The timezone of your database
- Advanced Parameters
- Snapshot Mode (Default
When Needed
) See MySQL Snapshot Modes for more information - Represent Binary Data As (Default
bytes
) - Snapshot Chunk Size (Default
1024
) - This is the number of rows read at a time when snapshotting. This is a low safe value. As a guide, if you have 100m + rows of data you may want to move this to5120
. If you have 1bn then a higher number still will allow you to backfill faster. - Max Batch Size (Default
2048
) - A value that specifies the maximum size of each batch of events that the connector processes. Only increase if experiencing lag
- Snapshot Mode (Default
- Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.
- Click Save
The connector will take approximately 1 minute to start processing data.
Updated 17 days ago