Regex Support for >100k Tables

There are various use cases where a group of related tables must be considered one logical table when streamed to one or more destinations:

• Multi-tenancy implemented with “DB per tenant” or “Schema per tenant”
• Application-level sharding
• Cell-based architecture

In such cases, use the Regex support in the Schema configuration for a Source connector. All tables matching one regex in the schema will be considered one logical topic and streamed to one logical destination table.

​

Schema Configuration

Regex can be applied at the schema or database level, at the table level, or both. In the Source connector’s Schema tab, add a schema using a regex pattern (e.g. shard[0-9]*). All matching schemas will be merged, and the tables within them will be treated as a single logical set.

​

Advanced Settings

To enable regex support, go to the Source connector’s Settings tab and scroll to the Advanced section:

1. Toggle Enable regex support on
2. Configure the Replacement string for matching regex snippets
3. Set the Regex support key field template — an extra key field is needed to ensure unique records across merged sources (e.g. {{database}}.{{table}})
4. Set the metadata field name to store source information and ensure uniqueness (e.g. _streamkap_source_metadata)