Regex Support for >100k Tables

Use regex patterns in Source schema configuration to merge related tables into logical topics for multi-tenant, sharded, or cell-based architectures.

There are various use cases where a group of related tables must be considered one logical table when streamed to one or more destinations:

• Multi-tenancy implemented with “DB per tenant” or “Schema per tenant”
• Application-level sharding
• Cell-based architecture

In such cases, use the Regex support in the Schema configuration for a Source connector. All tables matching one regex in the schema will be considered one logical topic and streamed to one logical destination table.

Schema Configuration

Regex can be applied at the schema or database level, at the table level, or both.

In the Source connector’s Schema tab, add a schema using a regex pattern (e.g. shard[0-9]*). All matching schemas will be merged, and the tables within them will be treated as a single logical set.

Advanced Settings

To enable regex support, go to the Source connector’s Settings tab and scroll to the Advanced section:

1. Toggle Enable regex support on
2. Configure the Replacement string for matching regex snippets
3. Set the Regex support key field template — an extra key field is needed to ensure unique records across merged sources (e.g. {{database}}.{{table}})
4. Set the metadata field name to store source information and ensure uniqueness (e.g. _streamkap_source_metadata)