Search pipelines (POST)

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

PostgreSQL Hostname.

PostgreSQL Port. For example, 5432

Username to access the database

Password to access the database

Database from which to stream data

When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots

Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On).

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2)

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

The name of the replication slot for the connector to use.

The name of the publication for the connector to use.

Schemas to include.

Source tables to sync.

Controls what happens when the re-select post processor fails to fetch a column. 'Fail' stops the connector; 'Warn' logs a warning and continues.

When enabled, TOAST columns that cannot be read from the WAL are re-fetched from the source database at event time.

Whether to use an encrypted connection to the PostgreSQL server

Changes the format of topics to 'DatabaseName_TopicName'

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Regex pattern to match topics for enrichment.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

IP address or hostname of the Db2 database server

Port number of the Db2 database server

Name of the Db2 database user for connecting to the DB2 database server

Password to use when connecting to the Db2 database server

The name of the Db2 database from which to stream the changes

Source schemas to sync.

Source tables to sync.

Path to the signal table as schema.table (e.g., 'MYSCHEMA.STREAMKAP_SIGNAL'). The database name will be added automatically. This table is used for incremental snapshotting.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

DocumentDB Connection String. See DocumentDB documentation for further details.

How to encode arrays. 'Array' encodes them as Array objects but requires all values in the array to be of the same type. 'Array_String' encodes them as JSON Strings and should be used if arrays have mixed types

How to encode nested documents. 'Document' encodes them as JSON Objects, 'String' encodes them as JSON Strings

Source databases to sync.

Source collections to sync.

Full path to the signal collection including database and collection name (e.g., 'mydb.streamkap_signal'). This collection is used for incremental snapshotting. Follow the documentation for creating this collection.

The strategy used to handle change events for documents exceeding specified BSON size.

The maximum allowed size in bytes of the stored document for which change events are processed. This includes both, the size before and after database operation, more specifically this limits the size of fullDocument and fullDocumentBeforeChange filed of MongoDB change events.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

Public key to add to SSH server

Number of rows read per snapshot chunk. Lower values throttle snapshot throughput to prevent backlog on the destination

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

AWS Region

AWS Access Key ID

AWS Secret Key

used for backfill (snapshot)

Source tables to sync.

Dynamodb Service Endpoint (optional)

The maximum number of active tasks

If > 0, snapshot will run in parallel with streaming and snapshot records _streamkap_ts_ms will be set back in time by this amount to prioritize CDC events downstream

Poll Timeout (ms)

Delay in milliseconds between reading snapshot chunks. Higher values throttle snapshot throughput to prevent backlog on the destination

Incremental snapshot max threads

Full Export Expiration Time (ms)

Signal Kafka Poll Timeout (ms)

Force nested lists as JSON string

Force nested maps as JSON string

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

ElasticSearch host. Optionally it is possible to specify many hosts using ; as separator (host1;host2;host3)

ElasticSearch protocol (http/https)

Port for ElasticSearch HTTP/HTTPS REST API. For example, 443 or 9200

Type of Authentication

Elasticsearch username

Elasticsearch password

The name of ElasticSearch indices

The name of datetime field to use to detect new records.

The day and time to start consuming records from. Leave it empty if you want to consume records as of now.

The maximum number of active tasks

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

IP address or hostname of the Informix database server.

Informix SQLI listener port. For example, 9088.

Username to access the Informix database.

Password to access the Informix database.

The name of the Informix database from which to stream the changes.

Schemas to include.

Source tables to sync.

Path to the signal table as schema.table (e.g., 'myschema.streamkap_signal'). The database name will be added automatically. This table is used for incremental snapshotting.

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Specifies how the data for binary columns e.g. blob, byte, text should be represented. This setting depends on what the destination is. See the documentation for more details.

Size of the read buffer for receiving CDC events from the Informix server, in bytes.

Timeout (seconds) the CDC engine will wait before breaking out of a blocking read to check for shutdown signals.

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

Executor publish cadence (ms) — how often the parallel-snapshot executor publishes SourceSnapshotState (rows_scanned, status transitions) to the streamkap_state topic. Lower values surface progress faster at the cost of state-topic traffic.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Public key to add to SSH server

Prefix for the topic

Comma-separated topic names to stream from this Kafka cluster.

The serialised format of the data written to the Kafka topic

If untoggled (default), Streamkap attempts to infer schema from your data - depending on the Destination. Otherwise, Streamkap assumes the Kafka message key and value contain schema and payload structures

Enable when this topic is fed by another Streamkap pipeline and records already carry _streamkap_offset, _streamkap_ts_ms, _streamkap_source_ts_ms. Skips destination-side InsertField so upstream offsets are preserved.

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

The IP address or hostname of the MariaDB database server. For example, mariadb.something.rds.amazonaws.com

Port number of the MariaDB database server. For example, 3306

The name of the MariaDB user that the connector uses to connect to the MariaDB database server.

The password of the MariaDB user that the connector uses to connect to the MariaDB database server.

Databases to include. The connector does not capture changes in any database whose name is not included.

Source tables to sync. The connector does not capture changes in any table that is not included.

Full path to the signal table including database and table name (e.g., 'mydb.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Database' below; leave it blank for Kafka-only mode.

Optional. The database containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Set the connection timezone. If set to SERVER, the source will detect the connection time zone from the values configured on the MariaDB server session variables 'time_zone' or 'system_time_zone'

How to handle a change event for a table whose schema is unknown or unreadable (for example a regex-matched database the connector lacks privileges on). 'Automatic' uses Warn when regex support is enabled (skip the event and keep streaming) and Fail otherwise.

Whether or not to use a read-only connection. MariaDB has GTID enabled by default, so no additional configuration is necessary.

Enable regex support. Useful for merging multiple tables into the same output topic.

Replacement string for matching regex snippets.

Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

Enables SSL/TLS in a specific mode. 'disable' = unencrypted connection. 'trust' = encrypted, no certificate verification.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

Public key to add to SSH server

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

A table with an estimated size greater than max split size will trigger intra-table paralelism. Table will be split into max.split.size.bytes parts and the parts will be processed in parallel.

Snapshot progress publish candence, publishing more often can affect performance.

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Mongodb Connection String. See Mongodb documentation for further details.

The hostname(s) extracted from the MongoDB connection string.

How to encode arrays. 'Array' encodes them as Array objects but requires all values in the array to be of the same type. 'Array_String' encodes them as JSON Strings and should be used if arrays have mixed types. 'String' is deprecated, use 'Array_String' instead

How to encode nested documents. 'Document' encodes them as JSON Objects, 'String' encodes them as JSON Strings

Source databases to sync.

Source collections to sync.

Full path to the signal collection including database and collection name (e.g., 'mydb.streamkap_signal'). This collection is used for incremental snapshotting. Follow the documentation for creating this collection.

A JSON array of additional pipeline stages to apply when reading change events from MongoDB. This can be used to filter or transform change stream events before they are processed by the connector

The strategy used to handle change events for documents exceeding specified BSON size. 'Skip' ignores oversized documents, 'Split' breaks them into smaller chunks

The maximum allowed size in bytes of the stored document for which change events are processed. This includes both the size before and after the database operation, specifically this limits the size of fullDocument and fullDocumentBeforeChange fields of MongoDB change events

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Regex pattern to match topics for enrichment.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

MySQL Hostname. For example, mysqldb.something.rds.amazonaws.com

MySQL Port. For example, 3306

Username to access the database

Password to access the database

Databases to include.

Source tables to sync.

Full path to the signal table including database and table name (e.g., 'mydb.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On).

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2)

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Database' below; leave it blank for Kafka-only mode.

Optional. The database containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Set the connection timezone. If set to SERVER, the source will detect the connection time zone from the values configured on the MySQL server session variables 'time_zone' or 'system_time_zone'

How to handle a change event for a table whose schema is unknown or unreadable (for example a regex-matched database the connector lacks privileges on). 'Automatic' uses Warn when regex support is enabled (skip the event and keep streaming) and Fail otherwise.

Whether or not to use a read-only connection. Requires GTID mode to be enabled on the source database.

Enable regex support. Useful for merging multiple tables into the same output topic.

Replacement string for matching regex snippets.

Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId

Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Regex pattern to match topics for enrichment.

Public key to add to SSH server

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

A table with an estimated size greater than max split size will trigger intra-table paralelism. Table will be split into max.split.size.bytes parts and the parts will be processed in parallel.

Snapshot progress publish candence, publishing more often can affect performance.

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Oracle Hostname Or IP address

Oracle Port. For example, 1521

Username to access the database

Password to access the database

Database from which to stream data. If you're working with container databases (CDB), this should be the CDB name.

Pluggable database from which to stream data. Use this for container database (CDB) installations only.

Schemas to include.

Source tables to sync.

Full path to the signal table including schema and table name (e.g., 'MYSCHEMA.STREAMKAP_SIGNAL'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

When enabled, the connector emits a periodic heartbeat to a Kafka topic - this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table - providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

Controls how LogMiner resolves column metadata. 'online_catalog' (default) is fastest; 'redo_log_catalog' mines the data dictionary from redo logs and handles schema changes better; 'hybrid' combines both but has limitations with certain settings (e.g. LOB capture is not supported).

Enables capture of CLOB, NCLOB, and BLOB columns. Not supported with the Hybrid strategy. When combined with the Re-select Post Processor, LOB values unavailable in the redo log are re-fetched from the source at event time, producing one merged event per row change.

When enabled, columns unavailable in the redo log (e.g. LOB columns) are re-fetched from the source database at event time. For LOB capture, enable together with 'Redo Log Catalog' strategy and 'Enable LOB Column Capture'.

Controls what happens when the re-select post processor fails to fetch a column. 'Fail' stops the connector; 'Warn' logs a warning and continues.

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

IP address or hostname of the PlanetScale database server (VTGate).

Integer port number of the PlanetScale database server (VTGate).

Username of the PlanetScale database server (VTGate).

Password of the PlanetScale database server (VTGate).

The name of the keyspace from which to stream the changes.

The type of Tablet (hence MySQL) from which to stream the changes

Convert tinyInt(1) columns to boolean

Source tables to sync.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Disable TLS on the VTGate gRPC connection and use plaintext. Leave off for PlanetScale Cloud (TLS required); enable only for a local / self-hosted plaintext Vitess.

Send an HTTP-style authorization: Basic header on VTGate gRPC calls. Keep on for PlanetScale Cloud (its gateway requires it); turn off for a self-hosted vtgate started with --grpc_auth_mode static, which instead expects vitess-native StaticAuthCredentials.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

Public key to add to SSH server

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

A table with an estimated size greater than max split size will trigger intra-table paralelism. Table will be split into max.split.size.bytes parts and the parts will be processed in parallel.

Snapshot progress publish candence, publishing more often can affect performance.

Port of the VTGate MySQL wire-protocol endpoint used for parallel-snapshot chunk reads (the gRPC vstream endpoint is database.port). PlanetScale Cloud = 3306; self-hosted vtgate typically 15306.

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

PostgreSQL Hostname. For example, postgres.something.rds.amazonaws.com

PostgreSQL Port. For example, 5432

Username to access the database

Password to access the database

Database from which to stream data

When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots

Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On).

Enable regex support. Useful for merging multiple tables into the same output topic

Replacement string for matching regex snippets.

Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId

Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled.

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2)

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic - this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table - providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Use a logical-message heartbeat instead of a heartbeat table. Runs SELECT pg_logical_emit_message(true, ...) on each beat to keep the replication slot advancing - works on PG14+ primaries with a SELECT-only role and is compatible with read-only mode. No table or write grant required on the source.

The name of the replication slot for the connector to use.

The name of the publication for the connector to use.

Comma-separated schemas to capture from this database. Tables inside the listed schemas can be picked individually on the next step.

Source tables to sync.

When enabled, TOAST columns that cannot be read from the WAL are re-fetched from the source database at event time.

Controls what happens when the re-select post processor fails to fetch a column. 'Fail' stops the connector; 'Warn' logs a warning and continues.

Whether to use an encrypted connection to the PostgreSQL server

Changes the format of topics to 'DatabaseName_TopicName'

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Regex pattern to match topics for enrichment.

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

A table whose on-disk size exceeds this splits into ceil(size / threshold) disjoint ctid page-range sub-ranges so it doesn't monopolize one snapshot worker. Applies only to unfiltered ctid snapshots. Default 50 GiB.

Executor publish cadence (ms) - how often the parallel-snapshot executor publishes SourceSnapshotState (rows_scanned, status transitions) to the streamkap_state topic. Lower values surface progress faster at the cost of state-topic traffic.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

The input file format

When enabled, treat the first row of each CSV file as column names. When disabled, columns are auto-generated as column1, column2, ...

When enabled, derive the Kafka topic name per file from the S3 key. When disabled, all files go to the single default topic.

Number of leading path segments to drop from the S3 key before using folders for the topic name. Example: with key 'archive/public/users/file.csv' set 2 to drop 'archive/public'.

Number of folder segments (after the skip) to include in the topic name, joined with dots. Set 0 to skip folder-based routing entirely.

ScEL expression for building the topic suffix. When set, overrides Folder Skip and Folder Levels. The connector ID is always prepended. See the S3 Source documentation for available functions and examples.

The default topic name suffix. When Dynamic Topic Routing is disabled, all files are streamed to this single topic. When enabled, this is used as a fallback for files that do not match the routing rules.

Topics produced by this S3 source. Populated automatically as topics are discovered.

The AWS Access Key ID used to connect to S3

The AWS Secret Access Key used to connect to S3

The AWS region to be used

The S3 Bucket to use

Prefix for S3 object keys to scan (e.g. "data/2024/"). Can be used to specify a directory.

The interval in milliseconds at which to scan for new files.

The policy to use for cleaning up files after processing. Log marks files as processed without deleting. Delete removes files from S3 after processing.

The maximum number of active tasks

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

The Endpoint of the SQL Server database server. For example, sqlserverdb.abcdefgh.us-west-2.rds.amazonaws.com

SQL Server Port. For example, 1433

Use TLS encryption with the SQL Server?

Username to access the database

Password to access the database

Database to stream from.

Schemas to include.

Source tables to sync.

Path to the signal table as schema.table (e.g., 'dbo.streamkap_signal'). The database name will be added automatically. This table is used for incremental snapshotting.

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts.

Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts.

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

How many parallel chunk requests to send to the source DB.

Target byte size for one chunk SELECT. Drives LIMIT = ceil(chunk.size.bytes / avg_row_size).

A table with an estimated size greater than max split size will trigger intra-table paralelism. Table will be split into max.split.size.bytes parts and the parts will be processed in parallel.

Executor publish cadence (ms) — how often the parallel-snapshot executor publishes SourceSnapshotState (rows_scanned, status transitions) to the streamkap_state topic. Lower values surface progress faster at the cost of state-topic traffic.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Webhook URL for Apex triggers. Generated after source is created.

API Key for webhook authentication. Generated after source is created.

Salesforce instance URL (e.g., https://myorg.my.salesforce.com or https://myorg.lightning.force.com)

OAuth2 Consumer Key from your Salesforce Connected App / External Client App

OAuth2 Consumer Secret from your Salesforce Connected App / External Client App

Salesforce username (optional, needed only if client_credentials flow is not enabled)

Salesforce password + security token (optional, needed only if client_credentials flow is not enabled)

Salesforce objects to capture. Select the objects you want to receive change events for.

Enable native Salesforce CDC subscription. When enabled, the connector subscribes directly to Salesforce change events without needing Apex triggers.

How to handle events for unselected objects

Topic for events from unselected objects

Flatten nested record fields to top-level. Recommended for most destinations.

Prefix for flattened fields (empty = no prefix, Salesforce field names like 'Name', 'BillingCity' are used directly)

Include change metadata (ChangeEventHeader) in output. Disable for upsert/state-table use cases.

Enable dead letter queue for failed records

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Webhook URL to register in Shopify. Generated after source is created. Pass the API key as a query parameter, e.g. ?api_key=<API_KEY>.

API Key. Append to the Webhook URL as ?api_key=<API_KEY> when registering webhooks in Shopify. Generated after source is created.

Shopify store URL (e.g., https://yourstore.myshopify.com). Required for snapshots.

Client ID from your Shopify Dev Dashboard app (recommended). Tokens auto-refresh every 24 hours.

Client Secret from your Shopify Dev Dashboard app (recommended). Encrypted at rest.

Static access token (legacy custom apps only). Provide either Client ID + Client Secret OR this Access Token.

Shopify Admin API version used by snapshots.

Shopify app client secret used to verify the X-Shopify-Hmac-Sha256 header. Leave empty to skip verification.

Shopify resources to capture. Each resource maps to a Kafka topic. Fan-out topics are added automatically when fan-out fields are configured.

How to handle events for resources not in the Shopify Resources list

Topic for events from unselected resources (used when Unselected Resource Behavior is DEFAULT_TOPIC)

Include Shopify webhook metadata (_shop_domain, _event_id, _triggered_at, _api_version, _webhook_id) in output records. Disable for upsert / state-table use cases.

Comma-separated list of nested arrays to fan out into their own topics (e.g. orders.line_items, products.variants, customers.addresses). When empty, arrays stay inline.

Enable dead letter queue for failed records

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Webhook URL to register in Stripe. Generated after source is created. Pass the API key as a query parameter, e.g. ?api_key=<API_KEY>.

API Key. Append to the Webhook URL as ?api_key=<API_KEY> when registering webhooks in Stripe. Generated after source is created.

Stripe webhook signing secret (whsec_xxx). When set, every webhook is verified against the Stripe-Signature header. Leave empty to disable verification.

Stripe secret API key (sk_live_xxx or sk_test_xxx). Required for snapshots. Encrypted at rest.

Stripe resources to capture. Each resource maps to a Kafka topic. Fan-out topics are added automatically when fan-out fields are configured.

How to handle events for resources not in the Stripe Resources list

Topic for events from unselected resources (used when Unselected Resource Behavior is DEFAULT_TOPIC)

Include Stripe event metadata (_event_id, _event_type, _event_created, _api_version, _livemode, _previous_attributes) on the output record. Disable for upsert / state-table use cases.

Comma-separated list of nested lists to fan out into their own topics. Allowed: invoice.lines, charge.refunds, subscription.items.

Enable dead letter queue for failed records

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

PostgreSQL Hostname. For example, postgres.something.rds.amazonaws.com

PostgreSQL Port. For example, 5432

Username to access the database

Password to access the database

Database from which to stream data

When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots

Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table.

Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On).

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2)

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

When enabled, the connector emits a periodic heartbeat to a Kafka topic — this keeps the poll loop active and offsets advancing on low-traffic sources, preventing replication-slot/log lag and false-positive health alerts. To also write to a 'streamkap_heartbeat' table in the source database (keeps the source transaction log moving), set 'Heartbeat Table Schema' below; leave it blank for Kafka-only mode.

Optional. The schema containing a 'streamkap_heartbeat' table — providing this enables source-table heartbeat mode, which writes to the table on each beat to keep the source transaction log active. Leave blank for Kafka-only heartbeat (no table or write grant required). See the Streamkap documentation for table setup.

The name of the replication slot for the connector to use.

The name of the publication for the connector to use.

Schemas to include.

Source tables to sync.

Controls what happens when the re-select post processor fails to fetch a column. 'Fail' stops the connector; 'Warn' logs a warning and continues.

When enabled, TOAST columns that cannot be read from the WAL are re-fetched from the source database at event time.

Whether to use an encrypted connection to the PostgreSQL server

Changes the format of topics to 'DatabaseName_TopicName'

Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

The name of the static field to be added to the message key.

The value of the static field to be added to the message key.

The name of the static field to be added to the message value.

The value of the static field to be added to the message value.

Regex pattern to match topics for enrichment.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

IP address or hostname of the Vitess database server (VTGate).

Integer port number of the Vitess database server (VTGate).

An optional username of the Vitess database server (VTGate). If not configured, unauthenticated VTGate gRPC is used.

An optional password of the Vitess database server (VTGate). If not configured, unauthenticated VTGate gRPC is used.

The name of the keyspace from which to stream the changes.

The type of Tablet (hence MySQL) from which to stream the changes

IP address or hostname of the VTCtld server.

Integer port number of the VTCtld server.

The username of the VTCtld server.

The password of the VTCtld database server.

Source tables to sync.

<span>Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. <a href='https://docs.streamkap.com/streamkap-ip-addresses#streamkap-ip-addresses' class='docs-url' target='_blank'>View the Streamkap IP addresses to allowlist on your SSH server</a> </span>

Hostname of your SSH server

Port of your SSH server

User that allows Streamkap to connect to SSH server

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.

Public key to add to SSH server

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Webhook URL. This URL will be generated after the source is created

API. This key will be generated after the source is created

Camel Message Header contains key field details for Kafka Message, default is 'key'

Topic names to use as output

The incoming format of the data receive by webhook

Whether to add a 'delete' field to the schema when DELETE HTTP method is used

Move column(s) from value to key. Comma separated list of table columns in format 'table1.column1,table2.column2'

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Add _streamkap_topic field containing the Kafka topic name. Required for topic_router transforms to preserve end-to-end data lineage.

Webhook URL. This URL will be generated after the source is created

API Key. This key will be generated after the source is created

Behavior when an unknown Zendesk event type is encountered

Topic name for unknown event types (used when Unknown Event Behavior is DEFAULT_TOPIC)

Flatten nested detail fields to top-level with a prefix. Recommended for destinations that do not support nested objects.

Prefix for flattened detail fields to avoid name collisions (e.g. detail.id becomes detail_id)

Include the event field in output records. Enable for audit-log use cases (append mode). Disable for state-table use cases (upsert mode) where only the entity state matters.

Comma-separated list of domain-scoped fields to fan out into separate topics (e.g. ticket.tags, ticket.custom_fields, organization.tags). When empty, arrays stay inline.

Enable dead letter queue. Failed records are written to a DLQ topic instead of crashing the connector.

The maximum number of records returned in a single call to poll()

The maximum number of active tasks.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Whether to quote identifiers in SQL statements

Convert column(s) to Integer - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Float - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Decimal - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

If true, truncate decimals to a 38 precision

Convert column(s) to String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to JSON String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to JSON String? - if possible.

Convert column(s) to Binary JSON - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to Binary JSON? - if possible.

Converts all JSON String column(s) to Binary JSON - if possible.

Replaces column(s) value with a user-defined string. Comma separated list of table columns in format 'table1.column1,table2.column2'

List of regex patterns to search for. Comma separated list in format 'regex1,regex2'

List of replacement values for each regex pattern. Comma separated list in format 'regex1,regex2'

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Mask (anonymise) string column(s) in-place. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*').

Columns to exclude from masking. Comma separated list in format 'table1.column1,table2.column2'.

Masking algorithm. Hash functions are deterministic (same input -> same token) and length-preserving.

Secret salt prepended before hashing (used by the SHA256/MD5 functions). Strongly recommended: without it, low-cardinality values (phone, SSN, email) are reversible via a precomputed rainbow table.

Fill character used by the REDACT function (length is preserved).

Constant replacement used by the FIXED function.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Warning: Should only be used in conjunction with numeric conversion or other conversions. If field remained string after previous conversion, rename to field to <previous-field-name>_str. Comma separated list of table columns in format 'table1.column1,table2.column2'

Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name

JSON mapping of source to target column names. Keys should be schema.table.column, table.column or just column. Values must be valid column names (no dots or spaces).

Example: { "public.orders.amount": "Amount", "orders.quantity": "Qty", "customer_id": "CustomerId" }

Drop columns. Comma separated list of table columns to drop in format 'table1.column1,table2.column2'

When enabled, converts ALL optional columns to required (NOT NULL). The exclude list is still respected. Enabling this automatically disables 'Mark Column(s) as Optional'.

Controls how NULL values are handled when columns become required. NONE: throws an error on NULL values. TYPE_MIN: replaces NULLs with type-specific sentinel values (e.g. MIN_VALUE for integers, 'null' for strings, epoch for dates).

Mark columns as optional/nullable. Comma separated list of table columns in format 'table1.column1,table2.column2'

Comma-separated list of field mappings e.g. <code>field1:newField1,field2:newField2</code>.

Headers to columns. Comma separated list of headers using format <code><header name>:<header type>[:field name]</code> e.g. 'headerKey1:STRING,headerKey2:INT32:customKey2Name'

Upload the BigQuery service-account JSON key file.

Name of the destination BigQuery dataset. The dataset must already exist.

Time partitioning granularity used when auto-creating tables. Select NONE to create non-partitioned tables.

Optional record field to partition by. Leave blank for ingestion-time partitioning.

Optional comma-separated list of fields to cluster the table by (max 4).

Optional. Automatically delete partitions older than this many days from tables this destination creates. Leave blank to keep all partitions. Ignored when Time Partitioning is NONE.

Automatically create BigQuery tables for new topics.

Allow adding new columns to BigQuery tables when the schema evolves.

Allow changing BigQuery columns from REQUIRED to NULLABLE when the schema evolves.

The maximum number of records returned in a single call to poll()

The maximum number of active task

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Whether to quote identifiers in SQL statements

Convert column(s) to Integer - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Float - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Decimal - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

If true, truncate decimals to a 38 precision

Convert column(s) to String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to JSON String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to JSON String? - if possible.

Convert column(s) to Binary JSON - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to Binary JSON? - if possible.

Converts all JSON String column(s) to Binary JSON - if possible.

Replaces column(s) value with a user-defined string. Comma separated list of table columns in format 'table1.column1,table2.column2'

List of regex patterns to search for. Comma separated list in format 'regex1,regex2'

List of replacement values for each regex pattern. Comma separated list in format 'regex1,regex2'

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Mask (anonymise) string column(s) in-place. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*').

Columns to exclude from masking. Comma separated list in format 'table1.column1,table2.column2'.

Masking algorithm. Hash functions are deterministic (same input -> same token) and length-preserving.

Secret salt prepended before hashing (used by the SHA256/MD5 functions). Strongly recommended: without it, low-cardinality values (phone, SSN, email) are reversible via a precomputed rainbow table.

Fill character used by the REDACT function (length is preserved).

Constant replacement used by the FIXED function.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Warning: Should only be used in conjunction with numeric conversion or other conversions. If field remained string after previous conversion, rename to field to <previous-field-name>_str. Comma separated list of table columns in format 'table1.column1,table2.column2'

Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name

JSON mapping of source to target column names. Keys should be schema.table.column, table.column or just column. Values must be valid column names (no dots or spaces).

Example: { "public.orders.amount": "Amount", "orders.quantity": "Qty", "customer_id": "CustomerId" }

Drop columns. Comma separated list of table columns to drop in format 'table1.column1,table2.column2'

When enabled, converts ALL optional columns to required (NOT NULL). The exclude list is still respected. Enabling this automatically disables 'Mark Column(s) as Optional'.

Controls how NULL values are handled when columns become required. NONE: throws an error on NULL values. TYPE_MIN: replaces NULLs with type-specific sentinel values (e.g. MIN_VALUE for integers, 'null' for strings, epoch for dates).

Mark columns as optional/nullable. Comma separated list of table columns in format 'table1.column1,table2.column2'

Comma-separated list of field mappings e.g. <code>field1:newField1,field2:newField2</code>.

Headers to columns. Comma separated list of headers using format <code><header name>:<header type>[:field name]</code> e.g. 'headerKey1:STRING,headerKey2:INT32:customKey2Name'

Upsert or append modes are available

Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to upsert only)

ClickHouse Hostname Or IP address

Username to access ClickHouse

Password to access the ClickHouse

ClickHouse Port. For example, 8443

ClickHouse database

Enable TLS for network connections

Per topic configuration in JSON format

Allow JSON data type in ClickHouse, make sure the ClickHouse server supports it. If not set, the connector will use String type for JSON data.

Optional prefix prepended to every destination table name (e.g. 'prod_' produces tables like 'prod_customers'). Only letters, digits, and underscores are allowed.

Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to NONE

The maximum number of records returned in a single call to poll()

The maximum number of active tasks

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Whether to quote identifiers in SQL statements

Convert column(s) to Integer - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Float - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Decimal - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

If true, truncate decimals to a 38 precision

Convert column(s) to String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to JSON String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to JSON String? - if possible.

Convert column(s) to Binary JSON - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to Binary JSON? - if possible.

Converts all JSON String column(s) to Binary JSON - if possible.

Replaces column(s) value with a user-defined string. Comma separated list of table columns in format 'table1.column1,table2.column2'

List of regex patterns to search for. Comma separated list in format 'regex1,regex2'

List of replacement values for each regex pattern. Comma separated list in format 'regex1,regex2'

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Mask (anonymise) string column(s) in-place. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*').

Columns to exclude from masking. Comma separated list in format 'table1.column1,table2.column2'.

Masking algorithm. Hash functions are deterministic (same input -> same token) and length-preserving.

Secret salt prepended before hashing (used by the SHA256/MD5 functions). Strongly recommended: without it, low-cardinality values (phone, SSN, email) are reversible via a precomputed rainbow table.

Fill character used by the REDACT function (length is preserved).

Constant replacement used by the FIXED function.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Warning: Should only be used in conjunction with numeric conversion or other conversions. If field remained string after previous conversion, rename to field to <previous-field-name>_str. Comma separated list of table columns in format 'table1.column1,table2.column2'

Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name

JSON mapping of source to target column names. Keys should be schema.table.column, table.column or just column. Values must be valid column names (no dots or spaces).

Example: { "public.orders.amount": "Amount", "orders.quantity": "Qty", "customer_id": "CustomerId" }

Drop columns. Comma separated list of table columns to drop in format 'table1.column1,table2.column2'

When enabled, converts ALL optional columns to required (NOT NULL). The exclude list is still respected. Enabling this automatically disables 'Mark Column(s) as Optional'.

Controls how NULL values are handled when columns become required. NONE: throws an error on NULL values. TYPE_MIN: replaces NULLs with type-specific sentinel values (e.g. MIN_VALUE for integers, 'null' for strings, epoch for dates).

Mark columns as optional/nullable. Comma separated list of table columns in format 'table1.column1,table2.column2'

Comma-separated list of field mappings e.g. <code>field1:newField1,field2:newField2</code>.

Headers to columns. Comma separated list of headers using format <code><header name>:<header type>[:field name]</code> e.g. 'headerKey1:STRING,headerKey2:INT32:customKey2Name'

CockroachDB Hostname

CockroachDB Port. For example, 26257

Database name

Username to access the database

Password to access with the database

Schema for the associated table name

Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to NONE

Specifies the strategy used to insert events into the database

Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database

Specifies how the connector resolves the primary key columns from the event

Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from.

Falls back to Streamkap's default for tables where no match is found

Map source tables to specific destination tables. Input should be the format of source_table_name:destination_table_name separated by a new line

The maximum number of active tasks. NOTE: Increasing this value may increase parallelism and throughput but can also lead to higher costs on databricks side.

When enabled, preserves NULL values from the source database instead of replacing them with schema default values. Enable this if you need to distinguish between explicit NULLs and default values.

Whether to quote identifiers in SQL statements

Convert column(s) to Integer - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Float - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to Decimal - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

If true, truncate decimals to a 38 precision

Convert column(s) to String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert column(s) to JSON String - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to JSON String? - if possible.

Convert column(s) to Binary JSON - if possible. Comma separated list of table columns in format 'table1.column1,table2.column2'

Convert all Structs and Arrays to Binary JSON? - if possible.

Converts all JSON String column(s) to Binary JSON - if possible.

Replaces column(s) value with a user-defined string. Comma separated list of table columns in format 'table1.column1,table2.column2'

List of regex patterns to search for. Comma separated list in format 'regex1,regex2'

List of replacement values for each regex pattern. Comma separated list in format 'regex1,regex2'

Truncate or nullify oversized string fields. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*'). WARNING: Do not include primary key columns - truncation/nullification could cause data loss or failures.

Columns to exclude from oversized records processing. Comma separated list in format 'table1.column1,table2.column2'.

Maximum allowed byte size per field. Fields exceeding this size will be truncated or nullified. Required when using Oversized Records transform.

Action for oversized fields: TRUNCATE (trim to max size) or NULLIFY (set to null).

Suffix to append to truncated values (e.g., '...[TRUNCATED]'). Leave empty for no suffix.

Optional overall record size limit in bytes. Records are NOT dropped - only a warning is logged if record exceeds this after field processing. Set to -1 to disable.

Column data types that should never be truncated. Comma-separated. Defaults exclude JSON and XML columns.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Mask (anonymise) string column(s) in-place. Comma separated list of table columns in format 'table1.column1,table2.column2'. Supports wildcards (e.g., 'mytable.*').

Columns to exclude from masking. Comma separated list in format 'table1.column1,table2.column2'.

Masking algorithm. Hash functions are deterministic (same input -> same token) and length-preserving.

Secret salt prepended before hashing (used by the SHA256/MD5 functions). Strongly recommended: without it, low-cardinality values (phone, SSN, email) are reversible via a precomputed rainbow table.

Fill character used by the REDACT function (length is preserved).

Constant replacement used by the FIXED function.

Whether null fields should use schema default values. Set to false to preserve user-set NULLs from source.

Warning: Should only be used in conjunction with numeric conversion or other conversions. If field remained string after previous conversion, rename to field to <previous-field-name>_str. Comma separated list of table columns in format 'table1.column1,table2.column2'

Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name

JSON mapping of source to target column names. Keys should be schema.table.column, table.column or just column. Values must be valid column names (no dots or spaces).

Example: { "public.orders.amount": "Amount", "orders.quantity": "Qty", "customer_id": "CustomerId" }

Drop columns. Comma separated list of table columns to drop in format 'table1.column1,table2.column2'

When enabled, converts ALL optional columns to required (NOT NULL). The exclude list is still respected. Enabling this automatically disables 'Mark Column(s) as Optional'.

Controls how NULL values are handled when columns become required. NONE: throws an error on NULL values. TYPE_MIN: replaces NULLs with type-specific sentinel values (e.g. MIN_VALUE for integers, 'null' for strings, epoch for dates).

Mark columns as optional/nullable. Comma separated list of table columns in format 'table1.column1,table2.column2'

Comma-separated list of field mappings e.g. <code>field1:newField1,field2:newField2</code>.

Headers to columns. Comma separated list of headers using format <code><header name>:<header type>[:field name]</code> e.g. 'headerKey1:STRING,headerKey2:INT32:customKey2Name'

Upsert or append modes are available

Token

JDBC URL

Connection Timeout

The name of the Databricks catalog to use.

Partition tables or not