Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

Schema History Optimization

MySQL, MariaDB, Oracle, SQL Server, and Db2 sources record DDL in a schema history topic, which you can narrow to only the databases and tables you capture.

Overview

Some Streamkap Sources (MySQL, MariaDB, Oracle, SQL Server, Db2) maintain a schema history topic to track Data Definition Language (DDL) changes over time. This is necessary because their transaction logs don’t include complete schema metadata—These Sources need to track schema history separately to properly decode change events.

By default, the Source records schema structures from all databases and tables in your instance, even those you’re not capturing. For large database instances with many databases or tables, this can impact performance and potentially cause timeouts during startup or schema changes.

The Problem

When you have large database instances, schema history can cause several issues:

  • Slow Source startup and restarts - The Source must read and process the entire schema history topic on startup
  • Large topic sizes - Recording DDL for hundreds or thousands of uncaptured tables unnecessarily grows the topic
  • Timeout issues - Processing extensive schema history can cause timeouts

Optimization Settings

Two optional settings help optimize schema history performance for large database instances:

Capture Only Captured Databases DDL

When to enable: Your database instance contains many databases, but you’re only capturing a subset.

What it does: Limits schema history recording to only the databases you’ve selected for capture, ignoring all others.

Default: Disabled (records DDL from all databases)

Example use case: You have a SQL Server instance with 50 databases but only capture 3 of them. Enable this to record schema history for only those 3 databases.

Capture Only Captured Tables DDL

When to enable: Your captured databases contain many tables, but you’re only capturing specific tables.

What it does: Limits schema history recording to only the tables you’ve selected for capture, ignoring all others in the captured databases.

Default: Disabled (records DDL from all tables in captured databases)

Example use case: Your database has 500 tables but you only capture 20 tables for your pipeline. Enable this to record schema history for only those 20 tables.

How to Configure

These settings are available in the Advanced section when creating or editing your Source:

  1. Navigate to your Source configuration
  2. Expand the Advanced section
  3. Find the Capture Only Captured Databases DDL setting
  4. Find the Capture Only Captured Tables DDL setting
  5. Enable either or both as needed

Important Considerations

When these settings are enabled:

  • Schema changes to captured databases/tables are still tracked normally
  • Schema changes to non-captured databases/tables are ignored. When a new table is included for capture, its schema will attempt to be recorded from that point onward
  • The Source continues to handle schema history for all captured objects

Backward Compatibility

If you enable these settings on an existing Source:

  • The Source will continue working normally
  • Future schema changes will follow the new filtering rules-see above
  • Historical schema history data remains unchanged in the topic