Database Upgrade Guide

Minor Version Upgrades (e.g., 14.8 to 14.12)

Minor version upgrades are generally safe for CDC pipelines:

• Replication slots are preserved across minor upgrades
• The pipeline reconnects automatically after the database restarts
• No snapshot is typically required

Verify that wal_level = logical remains set after the upgrade, as minor upgrades should not change this.

Major Version Upgrades (e.g., 14 to 16)

PostgreSQL removes replication slots during major upgrades and does not restore them. When the connector restarts, it requests the last known offset, but PostgreSQL cannot return it from the new slot. A new replication slot only tracks changes from its creation point, so the connector would skip older change events and resume from the latest log position.

Procedure:

SHOW wal_level;

SELECT pg_create_logical_replication_slot('streamkap_pgoutput_slot', 'pgoutput');

-- Check if publication exists
SELECT * FROM pg_publication WHERE pubname = 'streamkap_pub';

-- Recreate if needed (all tables)
CREATE PUBLICATION streamkap_pub FOR ALL TABLES;

-- Or recreate with specific tables
CREATE PUBLICATION streamkap_pub FOR TABLE table1, table2, table3;

RDS / Aurora PostgreSQL Specifics

The same general procedure applies for Amazon RDS and Aurora PostgreSQL managed upgrades. Key considerations:

• Use the RDS console or CLI to perform the version upgrade
• After the upgrade, verify rds.logical_replication is still set to 1 in your parameter group
• Recreate the replication slot and verify the publication as described above
• For Aurora, ensure you are upgrading the primary instance (Aurora read replicas only support physical replication)

For full PostgreSQL source setup and troubleshooting details, see:

• PostgreSQL (self-hosted)
• Amazon RDS PostgreSQL
• Amazon RDS Aurora PostgreSQL
• PostgreSQL Source FAQ

Minor Version Upgrades (e.g., 8.0.30 to 8.0.35)

Minor version upgrades are generally seamless for MySQL CDC pipelines:

• If GTID is enabled, the pipeline resumes from the GTID position automatically
• Binary log positions are typically preserved
• No snapshot is required in most cases

Major Version Upgrades (e.g., 5.7 to 8.0)

Major MySQL upgrades require verifying that all CDC-related configuration remains intact.

Pre-upgrade verification:

Confirm these settings are properly configured in your target version:

Upgrade behavior depends on GTID configuration:

MySQL 5.7 to 8.0 specific notes:

• MySQL 8.0 changes the default authentication plugin to caching_sha2_password. If your Streamkap user was created with mysql_native_password, verify it still authenticates correctly after the upgrade
• Review binlog retention settings after upgrade: CALL mysql.rds_show_configuration; (RDS) or SHOW VARIABLES LIKE 'binlog_expire_logs_seconds'; (self-hosted)

For full MySQL source setup details, see:

• Amazon RDS MySQL
• MySQL (Generic)
• MySQL Source FAQ

General Upgrade Considerations

Streamkap’s Oracle connector uses LogMiner to read redo and archive logs for CDC. After upgrading your Oracle database, verify the following:

Post-upgrade verification checklist:

1. Supplemental logging is enabled – Oracle upgrades may reset supplemental logging settings

   -- Verify database-level supplemental logging
   SELECT NAME, SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE;
   -- SUPPLEMENTAL_LOG_DATA_MIN should be YES

2. Table-level supplemental logging is intact – Verify for each captured table

   -- Check table-level supplemental logging
   SELECT * FROM ALL_LOG_GROUPS WHERE TABLE_NAME = '{TABLE_NAME}';

3. ARCHIVELOG mode is still enabled

   archive log list;
   -- Database log mode should be: Archive Mode

4. Redo log configuration is unchanged – Verify redo log sizes and groups

   SELECT GROUP#, BYTES/1024/1024 SIZE_MB, STATUS FROM V$LOG ORDER BY 1;

5. Streamkap user privileges are intact – Verify the STREAMKAP_USER (or C##STREAMKAP_USER for CDB) retains all necessary grants, including LOGMINING, FLASHBACK ANY TABLE, and access to V$ views

Upgrade procedure:

1. Stop writes to the database
2. Allow the connector to capture all remaining events
3. Stop the Source in Streamkap
4. Perform the Oracle upgrade using your standard procedure
5. Verify all items in the checklist above
6. Re-enable supplemental logging if needed (ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;)
7. Resume the Source in Streamkap
8. Trigger a snapshot if you suspect any events were missed

For full Oracle source setup details, see:

• Oracle (Generic)
• Oracle on Amazon RDS
• Oracle Source FAQ

General Upgrade Considerations

SQL Server CDC relies on change tracking tables and the SQL Server Agent service. Upgrades can disrupt both of these.

Post-upgrade verification checklist:

1. CDC is still enabled on the database

   SELECT name, is_cdc_enabled FROM sys.databases WHERE name = '{database}';
   -- is_cdc_enabled should be 1

2. CDC capture jobs are running – SQL Server CDC depends on capture and cleanup jobs

   EXEC sys.sp_cdc_help_jobs;

3. SQL Server Agent is running – The Agent must be active for CDC capture jobs to execute. If it is not running, start it through SQL Server Management Studio or via service management

4. CDC is enabled on all captured tables – Verify each table still has CDC tracking enabled

   EXEC sys.sp_cdc_help_change_data_capture;

5. Streamkap user permissions are intact – Verify the streamkap_user retains SELECT privileges on the cdc schema and source tables

Upgrade procedure:

1. Stop writes to the database
2. Allow the connector to capture all remaining events
3. Stop the Source in Streamkap
4. Perform the SQL Server upgrade
5. Verify all items in the checklist above
6. Re-enable CDC on the database and tables if needed
7. Ensure SQL Server Agent is running
8. Resume the Source in Streamkap
9. Trigger a snapshot if you suspect any events were missed

For full SQL Server source setup details, see:

• SQL Server (Generic)
• SQL Server on Amazon RDS
• SQL Server on Google Cloud SQL
• Azure SQL Database
• SQL Server Source FAQ

General Upgrade Considerations

Streamkap’s MongoDB connector uses change streams (backed by the oplog) for CDC. MongoDB upgrade behavior depends on your deployment type and the version jump.

Replica set rolling upgrades are CDC-friendly:

• Rolling upgrades (upgrading secondary nodes first, then stepping down the primary) allow the connector to continue reading change streams with minimal interruption
• The connector automatically reconnects to the new primary after a stepdown

Key risks during major version upgrades:

• Change stream resume tokens may not survive major version upgrades if the oplog format changes
• If the resume token is invalidated, the connector cannot resume from its last position

Post-upgrade verification checklist:

1. Oplog size is sufficient – Verify the oplog window is large enough to cover the upgrade duration plus a buffer

2. Replica set is healthy – All members should be in a healthy state

   rs.status()

3. Streamkap user privileges are intact – Verify the streamkap_user retains readAnyDatabase and read on the local database

4. Change streams are functional – Verify the connector can open a change stream on the target collections

Upgrade procedure:

1. Verify your oplog window is large enough to cover the expected upgrade duration
2. If performing a rolling upgrade, allow the connector to continue running during secondary node upgrades
3. Before stepping down the primary, allow the connector to capture all pending events
4. Stop the Source in Streamkap if the resume token may be invalidated (major version upgrade)
5. Complete the upgrade
6. Verify all items in the checklist above
7. Resume the Source in Streamkap
8. Trigger a snapshot if the resume token was invalidated or you suspect data gaps

For full MongoDB source setup details, see:

• MongoDB Atlas
• MongoDB (Generic)
• MongoDB Source FAQ