> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamkap.com/llms.txt > Use this file to discover all available pages before exploring further. # Search pipelines (POST) > POST alternative for GET /pipelines. Use when URL length limits are exceeded (e.g., many filter values). Supports both query parameters and request body. Body parameters take precedence over query parameters. ## OpenAPI ````yaml /openapi/openapi.json post /pipelines/search openapi: 3.1.0 info: title: Streamkap REST API description: >- The Streamkap REST API allows you to programmatically manage your CDC pipelines, sources, destinations, transforms, and more. Authenticate using a bearer token obtained from the [Access Token](/api-reference/auth/access-token) endpoint with your API client credentials. contact: name: Streamkap Support url: https://streamkap.com/ email: support@streamkap.com license: name: Proprietary version: 2.0.0 servers: - url: https://api.streamkap.com description: Production security: [] tags: - name: Alerts description: Manage alert subscribers, preferences, and notification credentials. - name: Authentication description: Obtain and refresh access tokens, manage client credentials and roles. - name: Billing description: Retrieve usage metrics, summaries, and export billing data. - name: Consumer Groups description: List, inspect, and reset Kafka consumer group offsets. - name: Dashboard description: Retrieve organisation-level statistics and overview data. - name: Destinations description: Create, configure, and manage data destinations and their lifecycle. - name: Kafka Access description: Manage Kafka users and their access permissions. - name: Logs description: Query and summarise system logs. - name: Pipelines description: Create, configure, and manage CDC pipelines and their lifecycle. - name: Project Keys description: Manage project keys that bundle API credentials and optional Kafka access. - name: Schema Registry description: Browse schema subjects, versions, and retrieve schema definitions. - name: Services description: View and switch service metadata. - name: Sources description: >- Create, configure, and manage data sources, snapshots, and their lifecycle. - name: Tags description: Create, update, and manage resource tags for organisation. - name: Topics description: Browse topic details, statistics, configurations, metrics, and messages. - name: Transforms description: >- Create, deploy, and manage data transforms, unit tests, and implementation details. paths: /pipelines/search: post: tags: - Pipelines summary: Search pipelines (POST) description: >- POST alternative for GET /pipelines. Use when URL length limits are exceeded (e.g., many filter values). Supports both query parameters and request body. Body parameters take precedence over query parameters. operationId: searchPipelines parameters: - name: id in: query required: false schema: anyOf: - type: string - type: 'null' title: Id - name: partial_name in: query required: false schema: anyOf: - type: string - type: 'null' title: Partial Name - name: tag_ids in: query required: false schema: anyOf: - type: string - type: 'null' title: Tag Ids - name: tag_filter_operation in: query required: false schema: anyOf: - type: string enum: - and - or - type: 'null' description: >- Tag filter logic: 'or' (default) matches any tag, 'and' matches all tags. default: or title: Tag Filter Operation description: >- Tag filter logic: 'or' (default) matches any tag, 'and' matches all tags. - name: page in: query required: false schema: type: integer default: 1 title: Page - name: page_size in: query required: false schema: type: integer default: 10 title: Page Size - name: sort in: query required: false schema: anyOf: - type: string - type: 'null' description: >- Sort field. Can be a DB field (name, created_at) or a metric field (latency, status, recordsLag) title: Sort description: >- Sort field. Can be a DB field (name, created_at) or a metric field (latency, status, recordsLag) - name: sort_dir in: query required: false schema: type: string description: 'Sort direction: ''asc'' (ascending) or ''desc'' (descending)' default: asc title: Sort Dir description: 'Sort direction: ''asc'' (ascending) or ''desc'' (descending)' - name: source_id in: query required: false schema: anyOf: - type: string - type: 'null' title: Source Id - name: destination_id in: query required: false schema: anyOf: - type: string - type: 'null' title: Destination Id - name: status in: query required: false schema: anyOf: - type: string - type: 'null' description: >- Filter by pipeline status. Comma-separated for multiple statuses (e.g., 'Active', 'Active,Paused'). Valid statuses: Active, Broken, Paused, Stopped, Starting, Unassigned, Unknown title: Status description: >- Filter by pipeline status. Comma-separated for multiple statuses (e.g., 'Active', 'Active,Paused'). Valid statuses: Active, Broken, Paused, Stopped, Starting, Unassigned, Unknown - name: latency in: query required: false schema: anyOf: - type: string - type: 'null' description: >- Filter by pipeline latency in milliseconds. Format: 'operator:value' (e.g., 'gt:1000', 'lt:500'). Valid operators: gt (>), lt (<), gte (>=), lte (<=), eq (=), ne (!=) title: Latency description: >- Filter by pipeline latency in milliseconds. Format: 'operator:value' (e.g., 'gt:1000', 'lt:500'). Valid operators: gt (>), lt (<), gte (>=), lte (<=), eq (=), ne (!=) - name: lag in: query required: false schema: anyOf: - type: string - type: 'null' description: >- Filter by pipeline lag (records behind). Format: 'operator:value' (e.g., 'gt:100', 'lt:50'). Valid operators: gt (>), lt (<), gte (>=), lte (<=), eq (=), ne (!=) title: Lag description: >- Filter by pipeline lag (records behind). Format: 'operator:value' (e.g., 'gt:100', 'lt:50'). Valid operators: gt (>), lt (<), gte (>=), lte (<=), eq (=), ne (!=) - name: secret_returned in: query required: false schema: enum: - 'true' - 'false' type: string default: 'false' title: Secret Returned requestBody: content: application/json: schema: anyOf: - $ref: '#/components/schemas/PipelinesPaginatedReqBody' - type: 'null' title: Body responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/PipelinesPaginatedRes' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - FronteggHTTPAuthentication: [] components: schemas: PipelinesPaginatedReqBody: properties: tag_ids: anyOf: - items: type: string type: array - type: string - type: 'null' title: Tag Ids description: >- List of tag IDs (as array) or comma-separated string. Body takes precedence over query params. tag_filter_operation: anyOf: - type: string enum: - and - or - type: 'null' title: Tag Filter Operation description: >- Tag filter logic: 'or' (default) matches any tag, 'and' matches all tags. Body takes precedence over query params. source_id: anyOf: - items: type: string type: array - type: string - type: 'null' title: Source Id description: >- List of source IDs (as array) or comma-separated string. Body takes precedence over query params. destination_id: anyOf: - items: type: string type: array - type: string - type: 'null' title: Destination Id description: >- List of destination IDs (as array) or comma-separated string. Body takes precedence over query params. status: anyOf: - items: type: string type: array - type: string - type: 'null' title: Status description: >- List of statuses (as array) or comma-separated string. Body takes precedence over query params. additionalProperties: true type: object title: PipelinesPaginatedReqBody description: >- Body model for POST /pipelines/search requests. Supports large lists of filter values that would exceed URL length limits when using GET. Body parameters take precedence over query parameters when both are provided. PipelinesPaginatedRes: properties: page: type: integer title: Page description: Current page number default: 1 page_size: type: integer title: Page Size description: Results per page default: 10 total: anyOf: - type: integer - type: 'null' title: Total description: Total number of results has_next: anyOf: - type: boolean - type: 'null' title: Has Next description: Whether more pages exist result: items: $ref: '#/components/schemas/PipelineBasic' type: array title: Result additionalProperties: true type: object required: - result title: PipelinesPaginatedRes description: Get paginated Pipelines response. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError PipelineBasic: properties: name: anyOf: - type: string - type: 'null' title: Name sub_id: anyOf: - type: string - type: 'null' title: Sub Id tenant_id: anyOf: - type: string - type: 'null' title: Tenant Id id: anyOf: - type: string - type: 'null' title: Id snapshot_new_tables: anyOf: - type: boolean - type: 'null' title: Snapshot New Tables topic_ids: anyOf: - items: anyOf: - type: string - type: 'null' type: array - type: 'null' title: Topic Ids topics: anyOf: - items: anyOf: - type: string - type: 'null' type: array - type: 'null' title: Topics source: anyOf: - $ref: '#/components/schemas/SourceConnector' - type: 'null' destination: anyOf: - $ref: '#/components/schemas/DestinationConnector' - type: 'null' transforms: anyOf: - items: {} type: array - type: 'null' title: Transforms tags: anyOf: - items: type: string type: array - type: 'null' title: Tags default: [] periodic_audit: anyOf: - $ref: '#/components/schemas/UpdatePipelineAuditReq' - type: 'null' topic_auto_discovery_transforms: items: $ref: '#/components/schemas/TopicAutoDiscoveryTransform' type: array title: Topic Auto Discovery Transforms default: [] additionalProperties: true type: object title: PipelineBasic ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError SourceConnector: properties: name: anyOf: - type: string - type: 'null' title: Name description: User-defined connector name connector: anyOf: - type: string - type: 'null' title: Connector description: Connector type (e.g., 'postgresql', 'mysql', 'mongodb') id: anyOf: - type: string - type: 'null' title: Id description: Unique identifier connector_display_name: anyOf: - type: string - type: 'null' title: Connector Display Name description: Human-readable connector type name created_timestamp: anyOf: - type: string - type: string format: date-time - type: 'null' title: Created Timestamp description: Creation timestamp (ISO 8601) sub_id: anyOf: - type: string - type: 'null' title: Sub Id description: Subscription identifier tenant_id: anyOf: - type: string - type: 'null' title: Tenant Id description: Tenant identifier for multi-tenancy service_id: anyOf: - type: string - type: 'null' title: Service Id description: Associated service identifier config: anyOf: - oneOf: - $ref: '#/components/schemas/AlloydbSourceConfig' - $ref: '#/components/schemas/Db2SourceConfig' - $ref: '#/components/schemas/DocumentdbSourceConfig' - $ref: '#/components/schemas/DynamodbSourceConfig' - $ref: '#/components/schemas/ElasticsearchSourceConfig' - $ref: '#/components/schemas/KafkadirectSourceConfig' - $ref: '#/components/schemas/MariadbSourceConfig' - $ref: '#/components/schemas/MongodbSourceConfig' - $ref: '#/components/schemas/MongodbhostedSourceConfig' - $ref: '#/components/schemas/MysqlSourceConfig' - $ref: '#/components/schemas/OracleSourceConfig' - $ref: '#/components/schemas/OracleawsSourceConfig' - $ref: '#/components/schemas/PlanetscaleSourceConfig' - $ref: '#/components/schemas/PostgresqlSourceConfig' - $ref: '#/components/schemas/RedisSourceConfig' - $ref: '#/components/schemas/S3SourceConfig' - $ref: '#/components/schemas/SalesforceWebhookSourceConfig' - $ref: '#/components/schemas/SqlserverawsSourceConfig' - $ref: '#/components/schemas/SupabaseSourceConfig' - $ref: '#/components/schemas/VitessSourceConfig' - $ref: '#/components/schemas/WebhookSourceConfig' - $ref: '#/components/schemas/ZendeskWebhookSourceConfig' - type: 'null' title: Config description: Connector configuration parameters topic_ids: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topic Ids description: List of associated topic identifiers topic_map: anyOf: - additionalProperties: items: anyOf: - type: string - type: integer type: array type: object - type: 'null' title: Topic Map description: Mapping of topics to their partitions or related entities topics: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topics description: List of topic names tasks: anyOf: - items: type: integer type: array - type: 'null' title: Tasks description: List of task identifiers connector_status: anyOf: - type: string - type: 'null' title: Connector Status description: >- Current status: Active, Paused, Stopped, Broken, Starting, Unassigned, Unknown, Pending desired_state: anyOf: - type: string - type: 'null' title: Desired State description: 'Desired state: Pending, Active, Paused, Stopped' kc_cluster_id: anyOf: - type: string - type: 'null' title: Kc Cluster Id description: >- KC cluster this connector is deployed to. None means default cluster. task_statuses: anyOf: - additionalProperties: additionalProperties: anyOf: - type: string - type: integer type: object type: object - type: 'null' title: Task Statuses description: Status information for each connector task tags: anyOf: - items: type: string type: array - type: 'null' title: Tags description: List of tag IDs assigned to this source additionalProperties: true type: object title: SourceConnector description: Source connector configuration and status. DestinationConnector: properties: name: anyOf: - type: string - type: 'null' title: Name description: User-defined connector name connector: anyOf: - type: string - type: 'null' title: Connector description: Connector type (e.g., 'snowflake', 'bigquery', 'clickhouse') id: anyOf: - type: string - type: 'null' title: Id description: Unique identifier connector_display_name: anyOf: - type: string - type: 'null' title: Connector Display Name description: Human-readable connector type name created_timestamp: anyOf: - type: string - type: string format: date-time - type: 'null' title: Created Timestamp description: Creation timestamp (ISO 8601) sub_id: anyOf: - type: string - type: 'null' title: Sub Id description: Subscription identifier tenant_id: anyOf: - type: string - type: 'null' title: Tenant Id description: Tenant identifier for multi-tenancy config: anyOf: - oneOf: - $ref: '#/components/schemas/AzblobDestinationConfig' - $ref: '#/components/schemas/BigqueryDestinationConfig' - $ref: '#/components/schemas/ClickhouseDestinationConfig' - $ref: '#/components/schemas/CockroachdbDestinationConfig' - $ref: '#/components/schemas/DatabricksDestinationConfig' - $ref: '#/components/schemas/Db2DestinationConfig' - $ref: '#/components/schemas/GcsDestinationConfig' - $ref: '#/components/schemas/HttpsinkDestinationConfig' - $ref: '#/components/schemas/IcebergDestinationConfig' - $ref: '#/components/schemas/KafkaDestinationConfig' - $ref: '#/components/schemas/KafkadirectDestinationConfig' - $ref: '#/components/schemas/MotherduckDestinationConfig' - $ref: '#/components/schemas/MysqlDestinationConfig' - $ref: '#/components/schemas/OracleDestinationConfig' - $ref: '#/components/schemas/PineconeDestinationConfig' - $ref: '#/components/schemas/PostgresqlDestinationConfig' - $ref: '#/components/schemas/R2DestinationConfig' - $ref: '#/components/schemas/RedisDestinationConfig' - $ref: '#/components/schemas/RedshiftDestinationConfig' - $ref: '#/components/schemas/S3DestinationConfig' - $ref: '#/components/schemas/SnowflakeDestinationConfig' - $ref: '#/components/schemas/SqlserverDestinationConfig' - $ref: '#/components/schemas/StarburstDestinationConfig' - $ref: '#/components/schemas/WeaviateDestinationConfig' - type: 'null' title: Config description: Connector configuration parameters topic_ids: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topic Ids description: List of associated topic identifiers topic_map: anyOf: - additionalProperties: items: anyOf: - type: string - type: integer type: array type: object - type: 'null' title: Topic Map description: Mapping of topics to their partitions or related entities topics: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topics description: List of topic names tasks: anyOf: - items: type: integer type: array - type: 'null' title: Tasks description: List of task identifiers connector_status: anyOf: - type: string - type: 'null' title: Connector Status description: >- Current status: Active, Paused, Stopped, Broken, Starting, Unassigned, Unknown, Pending desired_state: anyOf: - type: string - type: 'null' title: Desired State description: 'Desired state: Pending, Active, Paused, Stopped' kc_cluster_id: anyOf: - type: string - type: 'null' title: Kc Cluster Id description: >- KC cluster this connector is deployed to. None means default cluster. task_statuses: anyOf: - additionalProperties: additionalProperties: anyOf: - type: string - type: integer type: object type: object - type: 'null' title: Task Statuses description: Status information for each connector task tags: anyOf: - items: type: string type: array - type: 'null' title: Tags description: List of tag IDs assigned to this destination additionalProperties: true type: object title: DestinationConnector description: Destination connector configuration and status. UpdatePipelineAuditReq: properties: topics: items: anyOf: - type: string - type: integer type: array title: Topics timestamp_column: type: string title: Timestamp Column interval_minutes: type: integer title: Interval Minutes fix_deletes_only: type: boolean title: Fix Deletes Only default: false additionalProperties: true type: object required: - topics - timestamp_column - interval_minutes title: UpdatePipelineAuditReq description: Update pipeline audit request. TopicAutoDiscoveryTransform: properties: transform_id: type: string title: Transform Id regex: type: string title: Regex additionalProperties: true type: object required: - transform_id - regex title: TopicAutoDiscoveryTransform AlloydbSourceConfig: type: object title: AlloyDB description: Configuration properties for the AlloyDB source connector. properties: database.hostname.user.defined: type: string description: PostgreSQL Hostname. database.port.user.defined: type: string description: PostgreSQL Port. For example, 5432 default: '5432' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.dbname: type: string description: Database from which to stream data snapshot.read.only.user.defined: type: string enum: - 'Yes' - 'No' description: >- When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots default: 'Yes' signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table. column.include.list.toggled: type: boolean description: >- Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On). default: true column.include.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2) column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. slot.name: type: string description: The name of the replication slot for the connector to use. default: streamkap_pgoutput_slot publication.name: type: string description: The name of the publication for the connector to use. default: streamkap_pub schema.include.list: type: string description: Schemas to include. table.include.list.user.defined: type: string description: Source tables to sync. database.sslmode: type: string enum: - require - disable description: Whether to use an encrypted connection to the PostgreSQL server default: require include.source.db.name.in.table.name.user.defined: type: boolean description: Changes the format of topics to 'DatabaseName_TopicName' default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - slot.name - publication.name - schema.include.list - table.include.list.user.defined - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed Db2SourceConfig: type: object title: Db2 description: Configuration properties for the Db2 source connector. properties: database.hostname.user.defined: type: string description: IP address or hostname of the Db2 database server database.port.user.defined: type: string description: Port number of the Db2 database server default: '50000' database.user: type: string description: >- Name of the Db2 database user for connecting to the DB2 database server database.password: type: string format: password description: Password to use when connecting to the Db2 database server database.dbname: type: string description: The name of the Db2 database from which to stream the changes schema.include.list.user.defined: type: string description: Source schemas to sync. table.include.list.user.defined: type: string description: Source tables to sync. signal.data.collection.schema.or.database: type: string description: >- Path to the signal table as schema.table (e.g., 'MYSCHEMA.STREAMKAP_SIGNAL'). The database name will be added automatically. This table is used for incremental snapshotting. schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - schema.include.list.user.defined - table.include.list.user.defined - signal.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed DocumentdbSourceConfig: type: object title: DocumentDB description: Configuration properties for the DocumentDB source connector. properties: mongodb.connection.string.user.defined: type: string format: password description: >- DocumentDB Connection String. See DocumentDB documentation for further details. transforms.unwrap.array.encoding: type: string enum: - value: array label: Native array - value: array_string label: Stringified JSON array description: >- How to encode arrays. 'Array' encodes them as Array objects but requires all values in the array to be of the same type. 'Array_String' encodes them as JSON Strings and should be used if arrays have mixed types default: array_string transforms.unwrap.document.encoding: type: string enum: - value: document label: Nested document - value: string label: Stringified JSON description: >- How to encode nested documents. 'Document' encodes them as JSON Objects, 'String' encodes them as JSON Strings default: document database.include.list: type: string description: Source databases to sync. collection.include.list.user.defined: type: string description: Source collections to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal collection including database and collection name (e.g., 'mydb.streamkap_signal'). This collection is used for incremental snapshotting. Follow the documentation for creating this collection. cursor.oversize.handling.mode: type: string enum: - skip - split description: >- The strategy used to handle change events for documents exceeding specified BSON size. default: skip cursor.oversize.skip.threshold: type: integer minimum: 1000000 maximum: 16000000 description: >- The maximum allowed size in bytes of the stored document for which change events are processed. This includes both, the size before and after database operation, more specifically this limits the size of fullDocument and fullDocumentBeforeChange filed of MongoDB change events. default: 16000000 ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - mongodb.connection.string.user.defined - database.include.list - collection.include.list.user.defined - signal.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed DynamodbSourceConfig: type: object title: DynamoDB description: Configuration properties for the DynamoDB source connector. properties: aws.region: type: string description: AWS Region aws.access.key.id: type: string format: password description: AWS Access Key ID aws.secret.key: type: string format: password description: AWS Secret Key s3.export.bucket.name: type: string description: used for backfill (snapshot) table.include.list.user.defined: type: string description: Source tables to sync. batch.size: type: integer dynamodb.service.endpoint: type: string description: Dynamodb Service Endpoint (optional) tasks.max: type: integer minimum: 1 maximum: 40 description: The maximum number of active tasks default: 10 snapshot.parallel.time.offset: type: integer minimum: 0 maximum: 604800000 description: >- If > 0, snapshot will run in parallel with streaming and snapshot records _streamkap_ts_ms will be set back in time by this amount to prioritize CDC events downstream default: 0 poll.timeout.ms: type: integer description: Poll Timeout (ms) default: '180000' incremental.snapshot.chunk.size: type: integer description: Incremental snapshot chunk size default: '32768' incremental.snapshot.max.threads: type: integer description: Incremental snapshot max threads default: '8' full.export.expiration.time.ms: type: integer description: Full Export Expiration Time (ms) default: '86400000' signal.kafka.poll.timeout.ms: type: integer description: Signal Kafka Poll Timeout (ms) default: '1000' array.encoding.json: type: string description: Force nested lists as JSON string default: true struct.encoding.json: type: string description: Force nested maps as JSON string default: true required: - aws.region - aws.access.key.id - aws.secret.key - s3.export.bucket.name - table.include.list.user.defined ElasticsearchSourceConfig: type: object title: ElasticSearch description: Configuration properties for the ElasticSearch source connector. properties: es.host: type: string description: >- ElasticSearch host. Optionally it is possible to specify many hosts using ; as separator (host1;host2;host3) es.scheme: type: string enum: - http - https description: ElasticSearch protocol (http/https) default: https es.port: type: string description: Port for ElasticSearch HTTP/HTTPS REST API. For example, 443 or 9200 default: 443 http.auth.user.defined: type: string enum: - None - Basic description: Type of Authentication default: Basic http.auth.user: type: string description: Elasticsearch username http.auth.password: type: string format: password description: Elasticsearch password endpoint.include.list.user.defined: type: string description: The name of ElasticSearch indices datetime.field.name: type: string description: The name of datetime field to use to detect new records. datetime.field.value: type: string description: >- The day and time to start consuming records from. Leave it empty if you want to consume records as of now. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - es.host - es.port - http.auth.user - http.auth.password - endpoint.include.list.user.defined - datetime.field.name KafkadirectSourceConfig: type: object title: Kafka Direct description: Configuration properties for the Kafka Direct source connector. properties: topic.prefix: type: string description: Prefix for the topic topic.include.list.user.defined: type: string description: Comma-separated topic names to stream from this Kafka cluster. format: type: string enum: - json - string - avro description: The serialised format of the data written to the Kafka topic default: string schemas.enable: type: boolean description: >- If untoggled (default), Streamkap attempts to infer schema from your data - depending on the Destination. Otherwise, Streamkap assumes the Kafka message key and value contain `schema` and `payload` structures default: false required: - topic.prefix - topic.include.list.user.defined MariadbSourceConfig: type: object title: MariaDB description: Configuration properties for the MariaDB source connector. properties: database.hostname.user.defined: type: string description: >- The IP address or hostname of the MariaDB database server. For example, mariadb.something.rds.amazonaws.com database.port.user.defined: type: string description: Port number of the MariaDB database server. For example, 3306 default: '3306' database.user: type: string description: >- The name of the MariaDB user that the connector uses to connect to the MariaDB database server. database.password: type: string format: password description: >- The password of the MariaDB user that the connector uses to connect to the MariaDB database server. database.include.list.user.defined: type: string description: >- Databases to include. The connector does not capture changes in any database whose name is not included. table.include.list.user.defined: type: string description: >- Source tables to sync. The connector does not capture changes in any table that is not included. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including database and table name (e.g., 'mydb.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The database containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. database.connectionTimeZone: type: string enum: - SERVER - UTC - Africa/Cairo - ' Asia/Riyadh' - Africa/Casablanca - Asia/Seoul - Africa/Harare - Asia/Shanghai - Africa/Monrovia - Asia/Singapore - Africa/Nairobi - Asia/Taipei - Africa/Tripoli - Asia/Tehran - Africa/Windhoek - Asia/Tokyo - America/Araguaina - Asia/Ulaanbaatar - America/Asuncion - Asia/Vladivostok - America/Bogota - Asia/Yakutsk - America/Buenos_Aires - Asia/Yerevan - America/Caracas - Atlantic/Azores - America/Chihuahua - Australia/Adelaide - America/Cuiaba - Australia/Brisbane - America/Denver - Australia/Darwin - America/Fortaleza - Australia/Hobart - America/Guatemala - Australia/Perth - America/Halifax - Australia/Sydney - America/Manaus - Brazil/East - America/Matamoros - Canada/Newfoundland - America/Monterrey - Canada/Saskatchewan - America/Montevideo - Canada/Yukon - America/Phoenix - Europe/Amsterdam - America/Santiago - Europe/Athens - America/Tijuana - Europe/Dublin - Asia/Amman - Europe/Helsinki - Asia/Ashgabat - Europe/Istanbul - Asia/Baghdad - Europe/Kaliningrad - Asia/Baku - Europe/Moscow - Asia/Bangkok - Europe/Paris - Asia/Beirut - Europe/Prague - Asia/Calcutta - Europe/Sarajevo - Asia/Damascus - Pacific/Auckland - Asia/Dhaka - Pacific/Fiji - Asia/Irkutsk - Pacific/Guam - Asia/Jerusalem - Pacific/Honolulu - Asia/Kabul - Pacific/Samoa - Asia/Karachi - US/Alaska - Asia/Kathmandu - US/Central - Asia/Krasnoyarsk - US/Eastern - Asia/Magadan - US/East-Indiana - Asia/Muscat - US/Pacific - Asia/Novosibirsk description: >- Set the connection timezone. If set to SERVER, the source will detect the connection time zone from the values configured on the MariaDB server session variables 'time_zone' or 'system_time_zone' default: SERVER snapshot.gtid: type: string enum: - 'Yes' - 'No' description: >- Whether or not to use a read-only connection. MariaDB has GTID enabled by default, so no additional configuration is necessary. default: 'Yes' SourceRegexSupport.enabled: type: boolean description: >- Enable regex support. Useful for merging multiple tables into the same output topic. NOTE: most times when regex support is enabled there will be 100s of 1000s of tables and "Capture Only Captured Tables DDL?" must also be enabled. default: false transforms.SourceRegexSupport.regex.replacement: type: string description: Replacement string for matching regex snippets. default: REGEX transforms.SourceRegexSupport.key.field.template: type: string description: >- Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId default: '{{database}}.{{table}}' schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false transforms.SourceRegexSupport.metadata.field.name: type: string description: >- Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled. default: _streamkap_source_metadata schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes database.ssl.mode: type: string enum: - disable - trust description: >- Enables SSL/TLS in a specific mode. 'disable' = unencrypted connection. 'trust' = encrypted, no certificate verification. default: trust ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.include.list.user.defined - table.include.list.user.defined - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed MongodbSourceConfig: type: object title: MongoDB Atlas description: Configuration properties for the MongoDB Atlas source connector. properties: mongodb.connection.string.user.defined: type: string format: password description: >- Mongodb Connection String. See Mongodb documentation for further details. mongodb.connection.hostname: type: string description: The hostname(s) extracted from the MongoDB connection string. default: '' transforms.unwrap.array.encoding: type: string enum: - value: array label: Native array - value: array_string label: Stringified JSON array - value: string label: Raw JSON string (Deprecated) description: >- How to encode arrays. 'Array' encodes them as Array objects but requires all values in the array to be of the same type. 'Array_String' encodes them as JSON Strings and should be used if arrays have mixed types. 'String' is deprecated, use 'Array_String' instead default: array_string transforms.unwrap.document.encoding: type: string enum: - value: document label: Nested document - value: string label: Stringified JSON description: >- How to encode nested documents. 'Document' encodes them as JSON Objects, 'String' encodes them as JSON Strings default: document database.include.list: type: string description: Source databases to sync. collection.include.list.user.defined: type: string description: Source collections to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal collection including database and collection name (e.g., 'mydb.streamkap_signal'). This collection is used for incremental snapshotting. Follow the documentation for creating this collection. cursor.pipeline: type: string description: >- A JSON array of additional pipeline stages to apply when reading change events from MongoDB. This can be used to filter or transform change stream events before they are processed by the connector cursor.oversize.handling.mode: type: string enum: - skip - split description: >- The strategy used to handle change events for documents exceeding specified BSON size. 'Skip' ignores oversized documents, 'Split' breaks them into smaller chunks default: skip cursor.oversize.skip.threshold: type: integer minimum: 1000000 maximum: 16000000 description: >- The maximum allowed size in bytes of the stored document for which change events are processed. This includes both the size before and after the database operation, specifically this limits the size of fullDocument and fullDocumentBeforeChange fields of MongoDB change events default: 16000000 transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - mongodb.connection.string.user.defined - database.include.list - collection.include.list.user.defined - signal.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed MongodbhostedSourceConfig: type: object title: MongoDB description: Configuration properties for the MongoDB source connector. properties: mongodb.connection.string.user.defined: type: string format: password description: >- Mongodb Connection String. See Mongodb documentation for further details. mongodb.connection.hostname: type: string description: The hostname(s) extracted from the MongoDB connection string. default: '' transforms.unwrap.array.encoding: type: string enum: - value: array label: Native array - value: array_string label: Stringified JSON array - value: string label: Raw JSON string (Deprecated) description: >- How to encode arrays. 'Array' encodes them as Array objects but requires all values in the array to be of the same type. 'Array_String' encodes them as JSON Strings and should be used if arrays have mixed types. 'String' is deprecated, use 'Array_String' instead default: array_string transforms.unwrap.document.encoding: type: string enum: - value: document label: Nested document - value: string label: Stringified JSON description: >- How to encode nested documents. 'Document' encodes them as JSON Objects, 'String' encodes them as JSON Strings default: document database.include.list: type: string description: Source databases to sync. collection.include.list.user.defined: type: string description: Source collections to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal collection including database and collection name (e.g., 'mydb.streamkap_signal'). This collection is used for incremental snapshotting. Follow the documentation for creating this collection. cursor.pipeline: type: string description: >- A JSON array of additional pipeline stages to apply when reading change events from MongoDB. This can be used to filter or transform change stream events before they are processed by the connector cursor.oversize.handling.mode: type: string enum: - skip - split description: >- The strategy used to handle change events for documents exceeding specified BSON size. 'Skip' ignores oversized documents, 'Split' breaks them into smaller chunks default: skip cursor.oversize.skip.threshold: type: integer minimum: 1000000 maximum: 16000000 description: >- The maximum allowed size in bytes of the stored document for which change events are processed. This includes both the size before and after the database operation, specifically this limits the size of fullDocument and fullDocumentBeforeChange fields of MongoDB change events default: 16000000 transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - mongodb.connection.string.user.defined - database.include.list - collection.include.list.user.defined - signal.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed MysqlSourceConfig: type: object title: MySQL description: Configuration properties for the MySQL source connector. properties: database.hostname.user.defined: type: string description: MySQL Hostname. For example, mysqldb.something.rds.amazonaws.com database.port.user.defined: type: string description: MySQL Port. For example, 3306 default: '3306' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.include.list.user.defined: type: string description: Databases to include. table.include.list.user.defined: type: string description: Source tables to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including database and table name (e.g., 'mydb.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table. column.include.list.toggled: type: boolean description: >- Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On). default: true column.include.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2) column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The database containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. database.connectionTimeZone: type: string enum: - SERVER - UTC - Africa/Cairo - ' Asia/Riyadh' - Africa/Casablanca - Asia/Seoul - Africa/Harare - Asia/Shanghai - Africa/Monrovia - Asia/Singapore - Africa/Nairobi - Asia/Taipei - Africa/Tripoli - Asia/Tehran - Africa/Windhoek - Asia/Tokyo - America/Araguaina - Asia/Ulaanbaatar - America/Asuncion - Asia/Vladivostok - America/Bogota - Asia/Yakutsk - America/Buenos_Aires - Asia/Yerevan - America/Caracas - Atlantic/Azores - America/Chihuahua - Australia/Adelaide - America/Cuiaba - Australia/Brisbane - America/Denver - Australia/Darwin - America/Fortaleza - Australia/Hobart - America/Guatemala - Australia/Perth - America/Halifax - Australia/Sydney - America/Manaus - Brazil/East - America/Matamoros - Canada/Newfoundland - America/Monterrey - Canada/Saskatchewan - America/Montevideo - Canada/Yukon - America/Phoenix - Europe/Amsterdam - America/Santiago - Europe/Athens - America/Tijuana - Europe/Dublin - Asia/Amman - Europe/Helsinki - Asia/Ashgabat - Europe/Istanbul - Asia/Baghdad - Europe/Kaliningrad - Asia/Baku - Europe/Moscow - Asia/Bangkok - Europe/Paris - Asia/Beirut - Europe/Prague - Asia/Calcutta - Europe/Sarajevo - Asia/Damascus - Pacific/Auckland - Asia/Dhaka - Pacific/Fiji - Asia/Irkutsk - Pacific/Guam - Asia/Jerusalem - Pacific/Honolulu - Asia/Kabul - Pacific/Samoa - Asia/Karachi - US/Alaska - Asia/Kathmandu - US/Central - Asia/Krasnoyarsk - US/Eastern - Asia/Magadan - US/East-Indiana - Asia/Muscat - US/Pacific - Asia/Novosibirsk description: >- Set the connection timezone. If set to SERVER, the source will detect the connection time zone from the values configured on the MySQL server session variables 'time_zone' or 'system_time_zone' default: SERVER snapshot.gtid: type: string enum: - 'Yes' - 'No' description: >- Whether or not to use a read-only connection. Requires GTID mode to be enabled on the source database. default: 'Yes' SourceRegexSupport.enabled: type: boolean description: >- Enable regex support. Useful for merging multiple tables into the same output topic. NOTE: most times when regex support is enabled there will be 100s of 1000s of tables and "Capture Only Captured Tables DDL?" must also be enabled. default: false transforms.SourceRegexSupport.regex.replacement: type: string description: Replacement string for matching regex snippets. default: _REGEX_ transforms.SourceRegexSupport.key.field.template: type: string description: >- Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId default: '{{database}}.{{table}}' transforms.SourceRegexSupport.metadata.field.name: type: string description: >- Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled. default: _streamkap_source_metadata schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.include.list.user.defined - table.include.list.user.defined - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed OracleSourceConfig: type: object title: Oracle description: Configuration properties for the Oracle source connector. properties: database.hostname.user.defined: type: string description: Oracle Hostname Or IP address database.port.user.defined: type: string description: Oracle Port. For example, 1521 default: '1521' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.dbname: type: string description: >- Database from which to stream data. If you're working with container databases (CDB), this should be the CDB name. database.pdb.name: type: string description: >- Pluggable database from which to stream data. Use this for container database (CDB) installations only. schema.include.list: type: string description: Schemas to include. table.include.list.user.defined: type: string description: Source tables to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including schema and table name (e.g., 'MYSCHEMA.STREAMKAP_SIGNAL'). This table is used for incremental snapshotting. Follow the documentation for creating this table. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - schema.include.list - table.include.list.user.defined - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed OracleawsSourceConfig: type: object title: Oracle RDS description: Configuration properties for the Oracle RDS source connector. properties: database.hostname.user.defined: type: string description: Oracle Hostname Or IP address database.port.user.defined: type: string description: Oracle Port. For example, 1521 default: '1521' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.dbname: type: string description: Database from which to stream data. schema.include.list: type: string description: Schemas to include. table.include.list.user.defined: type: string description: Source tables to sync. signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including schema and table name (e.g., 'MYSCHEMA.STREAMKAP_SIGNAL'). This table is used for incremental snapshotting. Follow the documentation for creating this table. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - schema.include.list - table.include.list.user.defined - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed PlanetscaleSourceConfig: type: object title: Planetscale Vitess description: Configuration properties for the Planetscale Vitess source connector. properties: database.hostname.user.defined: type: string description: IP address or hostname of the PlanetScale database server (VTGate). database.port.user.defined: type: string description: Integer port number of the PlanetScale database server (VTGate). default: '443' database.user: type: string description: Username of the PlanetScale database server (VTGate). database.password: type: string format: password description: Password of the PlanetScale database server (VTGate). vitess.keyspace: type: string description: The name of the keyspace from which to stream the changes. vitess.tablet.type: type: string enum: - MASTER - REPLICA - RDONLY description: The type of Tablet (hence MySQL) from which to stream the changes default: MASTER converter.tinyintBool.user.defined: type: string description: Convert tinyInt(1) columns to boolean default: false table.include.list.user.defined: type: string description: Source tables to sync. schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - vitess.keyspace - table.include.list.user.defined - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed PostgresqlSourceConfig: type: object title: PostgreSQL description: Configuration properties for the PostgreSQL source connector. properties: database.hostname.user.defined: type: string description: >- PostgreSQL Hostname. For example, postgres.something.rds.amazonaws.com database.port.user.defined: type: string description: PostgreSQL Port. For example, 5432 default: '5432' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.dbname: type: string description: Database from which to stream data snapshot.read.only.user.defined: type: string enum: - 'Yes' - 'No' description: >- When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots default: 'Yes' signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table. column.include.list.toggled: type: boolean description: >- Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On). default: true SourceRegexSupport.enabled: type: boolean description: >- Enable regex support. Useful for merging multiple tables into the same output topic default: false transforms.SourceRegexSupport.regex.replacement: type: string description: Replacement string for matching regex snippets. default: _REGEX_ transforms.SourceRegexSupport.key.field.template: type: string description: >- Regex support key field template. An extra key field is needed to ensure unique data across all tables. Use this template with available variables: database, schema, table, sourceId default: '{{database}}.{{table}}' transforms.SourceRegexSupport.metadata.field.name: type: string description: >- Name of the extra metadata field to store source information and ensure uniqueness across all tables when regex support is enabled. default: _streamkap_source_metadata column.include.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2) column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. slot.name: type: string description: The name of the replication slot for the connector to use. default: streamkap_pgoutput_slot publication.name: type: string description: The name of the publication for the connector to use. default: streamkap_pub schema.include.list: type: string description: >- Comma-separated schemas to capture from this database. Tables inside the listed schemas can be picked individually on the next step. table.include.list.user.defined: type: string description: Source tables to sync. database.sslmode: type: string enum: - require - disable description: Whether to use an encrypted connection to the PostgreSQL server default: require include.source.db.name.in.table.name.user.defined: type: boolean description: Changes the format of topics to 'DatabaseName_TopicName' default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ streamkap.snapshot.large.table.threshold: type: integer minimum: 1 maximum: 64000 description: >- The threshold in MB for a Large Table to require multiple chunks to be read in parallel. default: 20000 streamkap.snapshot.custom.table.config.user.defined: type: string description: >- Explicitly set nb of parallel chunks for tables. Format: {"db.Some_Tbl": {"chunks": 5}}. This allows manual settings for parallelization when stats are outdated and estimated table size cannot be computed reliably. streamkap.snapshot.parallelism: type: integer minimum: 1 maximum: 10 description: How many parallel chunk requests to send to the source DB. default: 1 ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - slot.name - publication.name - schema.include.list - table.include.list.user.defined - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed RedisSourceConfig: type: object title: Redis description: Configuration properties for the Redis source connector. properties: connector.class.type: type: string enum: - Stream - Keys description: Type of Redis source connector default: Stream redis.host.user.defined: type: string description: Redis hostname (e.g., redis.example.com or 127.0.0.1) redis.port.user.defined: type: string description: Redis port number default: '6379' redis.username: type: string description: Redis username (optional, for Redis 6+ ACL) redis.password: type: string format: password description: Redis password (optional) ssl.enabled: type: boolean description: Enable TLS/SSL for secure connections default: true redis.stream.name: type: string description: Name of the Redis stream to read from redis.stream.offset.user.defined: type: string enum: - Latest - Earliest description: Starting point for reading from the stream default: Latest redis.stream.delivery.user.defined: type: string enum: - At Least Once - At Most Once description: Delivery guarantee mode default: At Least Once redis.stream.block.seconds.user.defined: type: integer minimum: 1 maximum: 60 description: Block duration when waiting for new messages default: 1 redis.stream.consumer.group: type: string description: Consumer group name for coordinating multiple consumers default: kafka-consumer-group redis.stream.consumer.name.user.defined: type: string description: Base name for consumer instances default: consumer redis.keys.pattern.user.defined: type: string description: Key pattern to monitor (e.g., user:*, order:*, or * for all keys) default: '*' redis.keys.timeout.seconds.user.defined: type: integer minimum: 60 maximum: 3600 description: Idle timeout before connector stops if no activity default: 300 mode: type: string enum: - LIVE - LIVEONLY description: >- LIVE performs initial snapshot then streams updates. LIVEONLY only streams updates. default: LIVE topic.use.stream.name: type: boolean description: >- Use Redis stream name as Kafka topic name (only for Stream connectors) default: false topic.user.defined: type: string description: Kafka topic name to publish messages to tasks.max: type: integer minimum: 1 maximum: 10 description: 'Number of parallel tasks (Stream: 1-10, Keys: always 1)' default: 1 required: - redis.host.user.defined - redis.port.user.defined - redis.username - redis.password - redis.stream.name - redis.stream.consumer.group - redis.stream.consumer.name.user.defined - redis.keys.pattern.user.defined - topic.user.defined S3SourceConfig: type: object title: S3 description: Configuration properties for the S3 source connector. properties: format: type: string enum: - json - csv - avro description: The input file format default: json csv.has.headers: type: boolean description: >- When enabled, treat the first row of each CSV file as column names. When disabled, columns are auto-generated as column1, column2, ... default: true topic.routing.enabled: type: boolean description: >- When enabled, derive the Kafka topic name per file from the S3 key. When disabled, all files go to the single default topic. default: false topic.routing.folder.skip: type: integer minimum: 0 description: >- Number of leading path segments to drop from the S3 key before using folders for the topic name. Example: with key 'archive/public/users/file.csv' set 2 to drop 'archive/public'. default: 0 topic.routing.folder.levels: type: integer minimum: 0 description: >- Number of folder segments (after the skip) to include in the topic name, joined with dots. Set 0 to skip folder-based routing entirely. default: 0 topic.routing.advanced.expression: type: string description: >- ScEL expression for building the topic suffix. When set, overrides Folder Skip and Folder Levels. The connector ID is always prepended. See the S3 Source documentation for available functions and examples. default: '' topic.postfix: type: string description: >- The default topic name suffix. When Dynamic Topic Routing is disabled, all files are streamed to this single topic. When enabled, this is used as a fallback for files that do not match the routing rules. default: default topic.include.list.user.defined: type: string description: >- Topics produced by this S3 source. Populated automatically as topics are discovered. default: '' aws.access.key.id: type: string description: The AWS Access Key ID used to connect to S3 default: '' aws.secret.access.key: type: string format: password description: The AWS Secret Access Key used to connect to S3 default: '' aws.s3.region: type: string enum: - value: eu-west-2 label: Europe (London) — eu-west-2 - value: eu-west-1 label: Europe (Ireland) — eu-west-1 - value: eu-central-1 label: Europe (Frankfurt) — eu-central-1 - value: ap-south-1 label: Asia Pacific (Mumbai) — ap-south-1 - value: ap-northeast-2 label: Asia Pacific (Seoul) — ap-northeast-2 - value: ap-northeast-1 label: Asia Pacific (Tokyo) — ap-northeast-1 - value: ap-southeast-1 label: Asia Pacific (Singapore) — ap-southeast-1 - value: ap-southeast-2 label: Asia Pacific (Sydney) — ap-southeast-2 - value: us-east-1 label: US East (N. Virginia) — us-east-1 - value: us-east-2 label: US East (Ohio) — us-east-2 - value: us-west-1 label: US West (N. California) — us-west-1 - value: us-west-2 label: US West (Oregon) — us-west-2 description: The AWS region to be used default: us-west-2 aws.s3.bucket.name: type: string description: The S3 Bucket to use default: '' aws.s3.bucket.prefix: type: string description: >- Prefix for S3 object keys to scan (e.g. "data/2024/"). Can be used to specify a directory. default: file-pulse/ fs.scan.interval.ms: type: integer minimum: 100 maximum: 100000 description: The interval in milliseconds at which to scan for new files. default: 10000 fs.cleanup.policy.class.user.defined: type: string enum: - Log - Delete description: >- The policy to use for cleaning up files after processing. Log marks files as processed without deleting. Delete removes files from S3 after processing. default: Log tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - aws.access.key.id - aws.secret.access.key SalesforceWebhookSourceConfig: type: object title: Salesforce CDC description: Configuration properties for the Salesforce CDC source connector. properties: webhook.url: type: string description: Webhook URL for Apex triggers. Generated after source is created. default: '' api.key: type: string format: password description: >- API Key for webhook authentication. Generated after source is created. default: camel.source.snapshot.salesforce.instance.url: type: string description: >- Salesforce instance URL (e.g., https://myorg.my.salesforce.com or https://myorg.lightning.force.com) default: '' camel.source.snapshot.salesforce.auth.client.id: type: string description: >- OAuth2 Consumer Key from your Salesforce Connected App / External Client App default: '' camel.source.snapshot.salesforce.auth.client.secret: type: string format: password description: >- OAuth2 Consumer Secret from your Salesforce Connected App / External Client App default: '' camel.source.snapshot.salesforce.auth.username: type: string description: >- Salesforce username (optional, needed only if client_credentials flow is not enabled) default: '' camel.source.snapshot.salesforce.auth.password: type: string format: password description: >- Salesforce password + security token (optional, needed only if client_credentials flow is not enabled) default: '' topic.include.list.user.defined: type: string description: >- Salesforce objects to capture. Select the objects you want to receive change events for. default: Account,Contact,Lead,Opportunity camel.source.cdc.enabled: type: boolean description: >- Enable native Salesforce CDC subscription. When enabled, the connector subscribes directly to Salesforce change events without needing Apex triggers. default: true camel.source.payload.router.unknown.type.behavior: type: string enum: - DEFAULT_TOPIC - SKIP - FAIL description: How to handle events for unselected objects default: DEFAULT_TOPIC camel.source.payload.router.unknown.type.default.topic: type: string description: Topic for events from unselected objects default: unknown camel.source.payload.router.flatten.detail: type: boolean description: >- Flatten nested record fields to top-level. Recommended for most destinations. default: true camel.source.payload.router.flatten.detail.prefix: type: string description: >- Prefix for flattened fields (empty = no prefix, Salesforce field names like 'Name', 'BillingCity' are used directly) default: '' camel.source.payload.router.include.event: type: boolean description: >- Include change metadata (ChangeEventHeader) in output. Disable for upsert/state-table use cases. default: false camel.source.dlq.enabled: type: boolean description: Enable dead letter queue for failed records default: true required: - camel.source.snapshot.salesforce.instance.url - camel.source.snapshot.salesforce.auth.client.id - camel.source.snapshot.salesforce.auth.client.secret SqlserverawsSourceConfig: type: object title: SQL Server description: Configuration properties for the SQL Server source connector. properties: database.hostname.user.defined: type: string description: >- The Endpoint of the SQL Server database server. For example, sqlserverdb.abcdefgh.us-west-2.rds.amazonaws.com database.port.user.defined: type: string description: SQL Server Port. For example, 1433 default: '1433' database.encrypt: type: boolean description: Use TLS encryption with the SQL Server? default: true database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.names: type: string description: Database to stream from. schema.include.list: type: string description: Schemas to include. table.include.list.user.defined: type: string description: Source tables to sync. signal.data.collection.schema.or.database: type: string description: >- Path to the signal table as schema.table (e.g., 'dbo.streamkap_signal'). The database name will be added automatically. This table is used for incremental snapshotting. default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. default: streamkap schema.history.internal.store.only.captured.databases.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. default: false schema.history.internal.store.only.captured.tables.ddl: type: string description: >- Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes streamkap.snapshot.parallelism: type: integer minimum: 1 maximum: 50 description: How many parallel chunk requests to send to the source DB. default: 1 streamkap.snapshot.large.table.threshold: type: integer minimum: 1 maximum: 64000 description: >- The threshold in MB for a Large Table to require multiple chunks to be read in parallel. default: 20000 streamkap.snapshot.custom.table.config.user.defined: type: string description: >- Explicitly set nb of parallel chunks for tables. Format: {"db.Some_Tbl": {"chunks": 5}}. This allows manual settings for parallelization when stats are outdated and estimated table size cannot be computed reliably. ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.names - schema.include.list - table.include.list.user.defined - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed SupabaseSourceConfig: type: object title: Supabase description: Configuration properties for the Supabase source connector. properties: database.hostname.user.defined: type: string description: >- PostgreSQL Hostname. For example, postgres.something.rds.amazonaws.com database.port.user.defined: type: string description: PostgreSQL Port. For example, 5432 default: '5432' database.user: type: string description: Username to access the database database.password: type: string format: password description: Password to access the database database.dbname: type: string description: Database from which to stream data snapshot.read.only.user.defined: type: string enum: - 'Yes' - 'No' description: >- When connecting to a read replica PostgreSQL database, this must be set to 'Yes' to support Streamkap snapshots default: 'Yes' signal.data.collection.schema.or.database: type: string description: >- Full path to the signal table including schema and table name (e.g., 'public.streamkap_signal'). This table is used for incremental snapshotting. Follow the documentation for creating this table. column.include.list.toggled: type: boolean description: >- Toggle between Inclusion (include only selected columns) and Exclusion (exclude selected columns). Defaults to Inclusion (On). default: true column.include.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName[.]tableName[.](columnName1|columnName2) column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. heartbeat.enabled: type: boolean description: >- When enabled, the connector sends periodic heartbeat messages to a Kafka topic to track connector liveness. In read-write mode, the connector also periodically writes to the 'streamkap_heartbeat' table in the source database to keep the transaction log active — this table must be created before enabling. See the Streamkap documentation for table setup instructions. default: true heartbeat.data.collection.schema.or.database: type: string description: >- The schema containing the 'streamkap_heartbeat' table. This table must be created before enabling heartbeat — see the Streamkap documentation for setup instructions. slot.name: type: string description: The name of the replication slot for the connector to use. default: streamkap_pgoutput_slot publication.name: type: string description: The name of the publication for the connector to use. default: streamkap_pub schema.include.list: type: string description: Schemas to include. table.include.list.user.defined: type: string description: Source tables to sync. database.sslmode: type: string enum: - require - disable description: Whether to use an encrypted connection to the PostgreSQL server default: require include.source.db.name.in.table.name.user.defined: type: boolean description: Changes the format of topics to 'DatabaseName_TopicName' default: false binary.handling.mode: type: string enum: - value: bytes label: Bytes - value: base64 label: Base64 - value: base64-url-safe label: Base64 (URL-safe) - value: hex label: Hex description: >- Specifies how the data for binary columns e.g. blob, binary, varbinary should be represented. This setting depends on what the destination is. See the documentation for more details. default: bytes transforms.InsertStaticKey1.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey1.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue1.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue1.static.value: type: string description: The value of the static field to be added to the message value. transforms.InsertStaticKey2.static.field: type: string description: The name of the static field to be added to the message key. transforms.InsertStaticKey2.static.value: type: string description: The value of the static field to be added to the message key. transforms.InsertStaticValue2.static.field: type: string description: The name of the static field to be added to the message value. transforms.InsertStaticValue2.static.value: type: string description: The value of the static field to be added to the message value. predicates.IsTopicToEnrich.pattern: type: string description: Regex pattern to match topics for enrichment. default: $^ ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - database.dbname - signal.data.collection.schema.or.database - heartbeat.data.collection.schema.or.database - slot.name - publication.name - schema.include.list - table.include.list.user.defined - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed VitessSourceConfig: type: object title: Vitess description: Configuration properties for the Vitess source connector. properties: database.hostname.user.defined: type: string description: IP address or hostname of the Vitess database server (VTGate). database.port.user.defined: type: string description: Integer port number of the Vitess database server (VTGate). default: '15991' database.user: type: string description: >- An optional username of the Vitess database server (VTGate). If not configured, unauthenticated VTGate gRPC is used. database.password: type: string format: password description: >- An optional password of the Vitess database server (VTGate). If not configured, unauthenticated VTGate gRPC is used. vitess.keyspace.user.defined: type: string description: The name of the keyspace from which to stream the changes. vitess.tablet.type: type: string enum: - MASTER - REPLICA - RDONLY description: The type of Tablet (hence MySQL) from which to stream the changes default: MASTER vitess.vtctld.host.user.defined: type: string description: IP address or hostname of the VTCtld server. vitess.vtctld.port.user.defined: type: string description: Integer port number of the VTCtld server. default: '15999' vitess.vtctld.user: type: string description: The username of the VTCtld server. vitess.vtctld.password: type: string format: password description: The password of the VTCtld database server. table.include.list.user.defined: type: string description: Source tables to sync. ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap column.exclude.list.user.defined: type: string description: >- An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.user - database.password - vitess.keyspace.user.defined - vitess.vtctld.host.user.defined - vitess.vtctld.port.user.defined - vitess.vtctld.user - vitess.vtctld.password - table.include.list.user.defined - ssh.host - ssh.port - ssh.user - ssh.public.key.user.displayed WebhookSourceConfig: type: object title: Webhook description: Configuration properties for the Webhook source connector. properties: webhook.url: type: string description: Webhook URL. This URL will be generated after the source is created default: '' api.key: type: string description: API. This key will be generated after the source is created default: camel.source.camelMessageHeaderKey: type: string description: >- Camel Message Header contains key field details for Kafka Message, default is 'key' default: key topic.include.list.user.defined: type: string description: Topic names to use as output default: default_topic format: type: string enum: - json - string description: The incoming format of the data receive by webhook default: json transforms.inferSchema.add.delete.field: type: boolean description: >- Whether to add a 'delete' field to the schema when DELETE HTTP method is used default: false ZendeskWebhookSourceConfig: type: object title: Zendesk Webhook description: Configuration properties for the Zendesk Webhook source connector. properties: webhook.url: type: string description: Webhook URL. This URL will be generated after the source is created default: '' api.key: type: string format: password description: API Key. This key will be generated after the source is created default: camel.source.payload.router.unknown.type.behavior: type: string enum: - DEFAULT_TOPIC - SKIP - FAIL description: Behavior when an unknown Zendesk event type is encountered default: DEFAULT_TOPIC camel.source.payload.router.unknown.type.default.topic: type: string description: >- Topic name for unknown event types (used when Unknown Event Behavior is DEFAULT_TOPIC) default: unknown camel.source.payload.router.flatten.detail: type: boolean description: >- Flatten nested detail fields to top-level with a prefix. Recommended for destinations that do not support nested objects. default: false camel.source.payload.router.flatten.detail.prefix: type: string description: >- Prefix for flattened detail fields to avoid name collisions (e.g. detail.id becomes detail_id) default: detail_ camel.source.payload.router.include.event: type: boolean description: >- Include the event field in output records. Enable for audit-log use cases (append mode). Disable for state-table use cases (upsert mode) where only the entity state matters. default: true camel.source.payload.router.fanout.fields: type: string description: >- Comma-separated list of domain-scoped fields to fan out into separate topics (e.g. ticket.tags, ticket.custom_fields, organization.tags). When empty, arrays stay inline. default: '' camel.source.dlq.enabled: type: boolean description: >- Enable dead letter queue. Failed records are written to a DLQ topic instead of crashing the connector. default: false AzblobDestinationConfig: type: object title: Azure Blob Storage description: >- Configuration properties for the Azure Blob Storage destination connector. properties: azblob.connection.string.user.defined: type: string format: password description: >- The connection string (for Account access keys) or Blob SAS URL (for Shared Access Signatures) azblob.container.name: type: string description: The name of an existing blob container to use format.user.defined: type: string enum: - json - csv - avro - parquet description: The format to use when writing data to file storage default: json format.csv.write.headers: type: string description: Include or exclude column name header row per file default: false topics.dir: type: string description: Top level directory for storing the data e.g. myfolder/subfolder file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' flush.size: type: integer minimum: 0 maximum: 2147483647 description: Number of records to write per file default: 1000 file.size: type: integer minimum: 0 maximum: 2147483647 description: >- Minimum size (in bytes) per file. Records are held in memory until this file size is met or the **Rotate interval** is exceeded default: 65536 rotate.interval.ms: type: integer minimum: -1 maximum: 31556952000 description: >- Maximum time (in milliseconds) to wait before writing records held in memory to file. This ignores the flush and file size settings default: -1 compression: type: string description: The compression type to use when writing data to the storage. required: - azblob.connection.string.user.defined BigqueryDestinationConfig: type: object title: BigQuery description: Configuration properties for the BigQuery destination connector. properties: bigquery.json: type: object format: password description: Upload the Bigquery JSON file. table.name.prefix: type: string description: Name of the destination Dataset for the of the associated table name bigquery.region: type: string enum: - us-east5 - us-central1 - us-west4 - us-west2 - northamerica-northeast1 - us-east4 - us-west1 - us-west3 - southamerica-east1 - southamerica-west1 - us-east1 - northamerica-northeast2 - asia-south2 - asia-east2 - asia-southeast2 - australia-southeast2 - asia-south1 - asia-northeast2 - asia-northeast3 - asia-southeast1 - australia-southeast1 - asia-east1 - asia-northeast1 - europe-west1 - europe-north1 - europe-west3 - europe-west2 - europe-southwest1 - europe-west8 - europe-west4 - europe-west9 - europe-central2 - europe-west6 - EU - US description: ' Region of the bigquery Dataset' default: us-central1 custom.bigquery.cluster.field: type: string description: User can set a Bigquery cluster field. custom.bigquery.partition.field: type: string description: User can set custom name of the partition field bigquery.time.based.partition: type: string description: Is the partition time based? default: false tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - bigquery.json - table.name.prefix ClickhouseDestinationConfig: type: object title: ClickHouse description: Configuration properties for the ClickHouse destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active task default: 5 quote.identifiers: type: string description: Whether to quote identifiers in SQL statements default: false hostname: type: string description: ClickHouse Hostname Or IP address connection.username: type: string description: Username to access ClickHouse connection.password: type: string format: password description: Password to access the ClickHouse port: type: string description: ClickHouse Port. For example, 8443 default: '8443' database: type: string description: ClickHouse database ssl: type: string description: Enable TLS for network connections default: true topics.config.map: type: string description: Per topic configuration in JSON format clickhouse.json.support: type: string description: >- Allow JSON data type in ClickHouse, make sure the ClickHouse server supports it. If not set, the connector will use String type for JSON data. default: false schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic required: - hostname - connection.username - connection.password - port - database CockroachdbDestinationConfig: type: object title: CockroachDB description: Configuration properties for the CockroachDB destination connector. properties: database.hostname.user.defined: type: string description: CockroachDB Hostname database.port.user.defined: type: string description: CockroachDB Port. For example, 26257 default: '26257' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database table.name.prefix: type: string description: Schema for the associated table name default: public schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - table.name.prefix DatabricksDestinationConfig: type: object title: Databricks description: Configuration properties for the Databricks destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert databricks.token: type: string format: password description: Token connection.url.user.defined: type: string description: JDBC URL connection.timeout.user.defined: type: integer minimum: 1 description: Connection Timeout default: '180' databricks.catalog.user.defined: type: string description: The name of the Databricks catalog to use. default: hive_metastore partition.mode: type: string enum: - by_topic - by_partition - by_topic_and_partition description: Partition tables or not default: by_topic schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name default: streamkap hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false tasks.max: type: integer minimum: 1 maximum: 100 description: >- The maximum number of active tasks. NOTE: Increasing this value may increase parallelism and throughput but can also lead to higher costs on databricks side. default: 5 consumer.wait.time.for.larger.batch.ms: type: string enum: - value: '500' label: 0.5s — Realtime - value: '5000' label: 5s — Low latency - value: '10000' label: 10s — Default - value: '20000' label: 20s — Moderate batch - value: '30000' label: 30s — Batch - value: '60000' label: 1min — Large batch - value: '120000' label: 2min — Bulk - value: '180000' label: 3min — Heavy bulk - value: '240000' label: 4min — Max throughput - value: '300000' label: 5min — Maximum batch description: >- Controls how long the connector waits to accumulate a larger batch before writing to Databricks. Higher values = larger batches = better throughput and fewer MERGE INTOs, but higher latency. default: '10000' topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - databricks.token - connection.url.user.defined Db2DestinationConfig: type: object title: Db2 description: Configuration properties for the Db2 destination connector. properties: database.hostname.user.defined: type: string description: Db2 Hostname database.port.user.defined: type: string description: Db2 Port. For example, 50000 default: '50000' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password GcsDestinationConfig: type: object title: GCS description: Configuration properties for the GCS destination connector. properties: gcs.credentials.json: type: string format: password description: >- GCP credentials as a JSON string with escaped quotes (" into \"). \n in the private_key field should also be escaped as \\n. Example format: {\"type\": \"service_account\",\"project_id\": \"XXXXXX\", ...} gcs.bucket.name: type: string description: The GCS Bucket to use. format.user.defined: type: string enum: - CSV - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: CSV file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to GCS. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - gcs.bucket.name HttpsinkDestinationConfig: type: object title: HTTP Sink description: Configuration properties for the HTTP Sink destination connector. properties: http.url: type: string description: The URL to send data to. http.authorization.type: type: string enum: - none - static - oauth2 description: The HTTP authorization type. default: none http.headers.authorization: type: string description: >- Static content of Authorization header (required for STATIC authorization). oauth2.access.token.url: type: string description: URL for fetching OAuth2 access token. oauth2.client.id: type: string description: OAuth2 client id. oauth2.client.secret: type: string format: password description: OAuth2 client secret. oauth2.scope: type: string description: OAuth2 client scope. http.headers.content.type: type: string description: Value of Content-Type header for each request. default: application/json http.headers.additional: type: string description: Additional headers in header:value format, comma-separated. http.proxy.host: type: string description: Proxy hostname. http.proxy.port: type: string description: Proxy port. batching.enabled: type: string description: Enable batching multiple records in a single HTTP request. default: false batch.max.size: type: integer minimum: 1 maximum: 1000000 description: Maximum size of a record batch (1 to 1,000,000). default: 500 batch.buffering.enabled: type: string description: >- Enable buffering and only send records when batch.max.size reaches or batch.max.time.ms hits. default: false batch.max.time.ms: type: integer minimum: 1 maximum: 1000000000 description: >- Maximum time to wait before flushing a batch (1 to 1,000,000,000 ms). default: 10000 batch.prefix: type: string description: Prefix for record batches. default: '[' batch.suffix: type: string description: Suffix for record batches. default: ']' batch.separator: type: string description: Separator for records in a batch. default: ',' max.retries: type: integer minimum: 0 maximum: 2147483647 description: Maximum number of retries on errors. default: 1 retry.backoff.ms: type: integer minimum: 0 maximum: 2147483647 description: Time in ms to wait before retrying. default: 3000 http.timeout: type: integer minimum: 0 maximum: 2147483647 description: HTTP response timeout in seconds. default: 30 decimal.format: type: string enum: - BASE64 - NUMERIC description: 'Format for serializing decimals: BASE64 (default) or NUMERIC.' default: NUMERIC errors.tolerance.user.defined: type: string enum: - none - all description: >- Error tolerance setting. Note: Cannot be used when Batching is enabled. default: none required: - http.url - oauth2.access.token.url - oauth2.client.id - oauth2.client.secret - oauth2.scope IcebergDestinationConfig: type: object title: Iceberg description: Configuration properties for the Iceberg destination connector. properties: iceberg.catalog.type: type: string enum: - value: rest label: REST - value: hive label: Hive - value: glue label: AWS Glue description: Type of Iceberg catalog default: rest iceberg.catalog.auth.mode: type: string enum: - value: none label: None (no auth) - value: token label: Token - value: oauth2 label: OAuth 2.0 description: >- How to authenticate with the Iceberg catalog. 'No Auth' for catalogs that don't require authentication. 'Token' for bearer token authentication (e.g., Cloudflare R2). 'OAuth2 Credential' for OAuth2 client credential authentication (e.g., Polaris, DataHub). default: none iceberg.catalog.name: type: string description: Iceberg catalog name iceberg.catalog.client.assume-role.arn: type: string description: AWS IAM role (e.g., arn:aws:iam:::role/) iceberg.catalog.uri: type: string description: >- Catalog endpoint URL. For REST catalogs use https://… ; for Hive Metastore use thrift://… ; AWS Glue uses the regional endpoint. iceberg.catalog.token: type: string format: password description: Bearer token used for catalog authentication. iceberg.catalog.credential: type: string format: password description: OAuth2 client credential in client_id:client_secret format. iceberg.catalog.scope: type: string description: OAuth2 scope for catalog authentication (e.g., PRINCIPAL_ROLE:ALL). default: PRINCIPAL_ROLE:ALL iceberg.catalog.s3.credentials.enabled: type: boolean description: >- Enable to provide your own S3 access key and secret. Disable if the catalog vends storage credentials automatically. default: false iceberg.catalog.s3.access-key-id: type: string description: The access key ID used to connect to S3 or S3-compatible storage. iceberg.catalog.s3.secret-access-key: type: string format: password description: >- The secret access key used to connect to S3 or S3-compatible storage. iceberg.catalog.client.region.user.defined: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 - auto description: The AWS region to be used. For Cloudflare R2, select 'auto'. default: us-west-2 iceberg.catalog.warehouse: type: string description: >- The Iceberg warehouse identifier. For R2 Data Catalog, this should be the warehouse name configured in the catalog. For S3-based catalogs, this is typically an S3 or S3-compatible path (e.g., s3://bucket/path). table.name.prefix: type: string description: >- Name of the database schema that contains the table (e.g., public, sales, analytics). insert.mode.user.defined: type: string enum: - value: insert label: Insert only - value: upsert label: Upsert (merge) description: Specifies the strategy used to insert events into the database default: insert iceberg.tables.default-id-columns: type: string description: >- Optional. A comma-separated list of field names to use as record identifiers when key fields are not present in Kafka messages iceberg.tables.hard-delete-enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true iceberg.tables.compaction.enabled: type: string description: Enable lightweight compaction of small data files after commits default: false iceberg.tables.compaction.target-file-size-bytes: type: string description: >- Target file size in bytes for compaction. Files smaller than this are candidates for merging default: '134217728' iceberg.tables.compaction.min-small-files: type: string description: Minimum number of small files required before triggering compaction default: '5' iceberg.tables.compaction.max-files-per-run: type: string description: Maximum number of small files to compact in a single run default: '100' iceberg.tables.compaction.commit-threshold: type: string description: Number of Iceberg commits between compaction checks default: '5' iceberg.tables.compaction.expire-snapshots: type: string description: >- Expire old snapshots and delete unreferenced data files after compaction default: false iceberg.tables.compaction.retain-last: type: string description: >- Number of most recent snapshots to retain when expiring snapshots after compaction default: '1' required: - iceberg.catalog.token - iceberg.catalog.credential - iceberg.catalog.scope - iceberg.catalog.s3.access-key-id - iceberg.catalog.s3.secret-access-key - iceberg.catalog.warehouse - table.name.prefix KafkaDestinationConfig: type: object title: Kafka description: Configuration properties for the Kafka destination connector. properties: kafka.sink.bootstrap: type: string description: >- A comma-separated list of host and port pairs that are the addresses of the Destination Kafka brokers. This list should be in the form host1:port1,host2:port2,... destination.format: type: string enum: - avro - json description: The format to use when writing data to kafka default: json json.schema.enable: type: string description: Include schema in json message default: false schema.registry.url.user.defined: type: string description: Destination kafka schema registry url topic.prefix: type: string description: Prefix for destination topics topic.suffix: type: string description: Suffix for destination topics transforms.InsertField.offset.field: type: string description: >- If Kafka destination is used to sync data between services or to apply some transformations, it would be better to rename this field to _streamkap_offset_sync or _streamkap_offset_transform to avoid conflicts with the destination on the other service default: _streamkap_offset transforms.changeTopicName.add.original.topic: type: string enum: - 'off' - header - field description: >- Add the original topic name to the output record. 'off' disables it, 'header' adds it as a Kafka header, 'field' adds it as a _streamkap_topic column in the destination table. default: 'off' tasks.max: type: integer minimum: 1 maximum: 100 description: Maximum number of tasks for the connector default: 5 required: - kafka.sink.bootstrap - schema.registry.url.user.defined KafkadirectDestinationConfig: type: object title: Kafka Direct description: Configuration properties for the Kafka Direct destination connector. properties: password: type: string format: password description: Password for setting up the Kafka proxy whitelist.ips.user.defined: type: string description: A comma-separated list of IPs/CIDRs required: - password - whitelist.ips.user.defined MotherduckDestinationConfig: type: object title: Motherduck description: Configuration properties for the Motherduck destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert motherduck.token: type: string format: password description: Motherduck Token motherduck.catalog.user.defined: type: string description: The name of the Motherduck database/catalog to use. schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name default: streamkap hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false tasks.max: type: integer minimum: 1 maximum: 25 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - motherduck.token - motherduck.catalog.user.defined MysqlDestinationConfig: type: object title: MySQL description: Configuration properties for the MySQL destination connector. properties: database.hostname.user.defined: type: string description: MySQL Hostname database.port.user.defined: type: string description: MySQL Port. For example, 3306 default: '3306' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password OracleDestinationConfig: type: object title: Oracle description: Configuration properties for the Oracle destination connector. properties: database.hostname.user.defined: type: string description: Oracle Hostname database.port.user.defined: type: string description: Oracle Port. For example, 1521 default: '1521' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password PineconeDestinationConfig: type: object title: Pinecone description: Configuration properties for the Pinecone destination connector. properties: pinecone.api.key: type: string format: password description: API key for authenticating with Pinecone pinecone.index.name: type: string description: Name of the target Pinecone index pinecone.proxy.host: type: string description: Proxy host for Pinecone connections pinecone.proxy.port: type: string description: Proxy port for Pinecone connections (required if Proxy Host is set) collection.mapping: type: string description: >- Mapping between Kafka topic and Pinecone namespace. Use ${topic} for dynamic topic name substitution default: ${topic} schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created namespaces, set to `NONE` default: basic document.id.strategy.user.defined: type: string enum: - None - Kafka ID - Field ID description: >- Strategy to generate document IDs. 'None' uses random UUIDs (append-only, no upsert/delete support) default: None document.id.field.name: type: string description: Field name containing the ID in Kafka record default: id vector.field.name: type: string description: Field name containing the embedding vector default: vector delete.enabled: type: string description: >- Treat null record values or __deleted=true as deletes. Requires Document ID Strategy to be set to 'Kafka ID' (not 'None') default: false batch.size: type: string description: Number of records per batch default: '100' retry.max: type: string description: Maximum number of retry attempts for failed operations default: '3' retry.backoff.ms: type: string description: Backoff time between retries in milliseconds default: '1000' required: - pinecone.api.key - pinecone.index.name - vector.field.name PostgresqlDestinationConfig: type: object title: PostgreSQL description: Configuration properties for the PostgreSQL destination connector. properties: database.hostname.user.defined: type: string description: PostgreSQL Hostname database.port.user.defined: type: string description: PostgreSQL Port. For example, 5432 default: '5432' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap table.name.prefix: type: string description: Schema for the associated table name schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - ssh.host - ssh.port - ssh.user - table.name.prefix - ssh.public.key.user.displayed R2DestinationConfig: type: object title: R2 description: Configuration properties for the R2 destination connector. properties: r2.account.user.defined: type: string description: >- Cloudflare R2 account ID. This is the account ID you see in the Cloudflare dashboard. aws.access.key.id: type: string description: The Access Key ID used to connect to R2. aws.secret.access.key: type: string format: password description: The Secret Access Key used to connect to R2. aws.s3.bucket.name: type: string description: The R2 Bucket to use. format.user.defined: type: string enum: - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: JSON Array file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to S3. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - r2.account.user.defined - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name RedisDestinationConfig: type: object title: Redis description: Configuration properties for the Redis destination connector. properties: redis.host.user.defined: type: string description: Redis Hostname redis.port.user.defined: type: string description: Redis Port. For example, 6379 default: '6379' redis.username: type: string description: Username to access the Redis database redis.password: type: string format: password description: Password to access the Redis database ssl.enabled: type: string description: Enable TLS for network connections default: true redis.key: type: string description: Redis key to stream the data redis.key.data.type: type: string enum: - Stream - List - Hash description: >- Data Type of Redis Key. At the moment, only Stream and List are supported default: Stream tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - redis.host.user.defined - redis.port.user.defined - redis.username - redis.password - redis.key RedshiftDestinationConfig: type: object title: Redshift description: Configuration properties for the Redshift destination connector. properties: tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 aws.redshift.domain: type: string description: Domain of the leader of the cluster aws.redshift.port: type: string description: Port number for incoming connections to the leader default: '5439' aws.redshift.database: type: string description: Name of the database on the cluster connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database primary.key.fields: type: string description: List of comma-separated primary key field names default: id schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name required: - aws.redshift.domain - aws.redshift.port - aws.redshift.database - connection.username - connection.password - primary.key.fields - table.name.prefix S3DestinationConfig: type: object title: S3 description: Configuration properties for the S3 destination connector. properties: aws.access.key.id: type: string description: The AWS Access Key ID used to connect to S3. aws.secret.access.key: type: string format: password description: The AWS Secret Access Key used to connect to S3. aws.s3.region: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 description: The AWS region to be used default: us-west-2 aws.s3.bucket.name: type: string description: The S3 Bucket to use. format.user.defined: type: string enum: - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: JSON Array file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to S3. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name SnowflakeDestinationConfig: type: object title: Snowflake description: Configuration properties for the Snowflake destination connector. properties: snowflake.url.name: type: string description: >- The URL for accessing your Snowflake account. This URL must include your account identifier. Note that the protocol (https://) and port number are optional. snowflake.user.name: type: string description: |+ User login name for the Snowflake account. snowflake.private.key: type: string format: password description: >- The private key to authenticate the user. Include only the key, not the header or footer. If the key is split across multiple lines, remove the line breaks. snowflake.private.key.passphrase.secured: type: string description: >- If checked (default), provide your SSH key's passphrase, otherwise, uncheck for SSH keys without passphrase. default: true snowflake.private.key.passphrase: type: string format: password description: The passphrase is used to decrypt the private key. sfwarehouse: type: string description: The name of the snowflake warehouse. default: STREAMKAP_WH snowflake.database.name: type: string description: >- The name of the database that contains the table to insert rows into. snowflake.schema.name: type: string description: The name of the schema that contains the table to insert rows into. create.schema.auto: type: string description: >- Automatically generates a Snowflake schema if it does not already exist. default: true snowflake.role.name: type: string description: >- The name of an existing role with necessary privileges (for Streamkap) assigned to the default: STREAMKAP_ROLE ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: >- Upsert or append modes are available. NOTE: when switching append to upsert, existing data must be deduplicated or deleted. Read more about upsert mode default: append hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic use.hybrid.tables: type: string description: >- Specifies whether the connector should create Hybrid Tables (applies to `upsert` only) default: false apply.dynamic.table.script: type: string description: >- Specifies whether the connector should create Dynamic Tables & Cleanup Tasks (applies to `append` only) default: false create.sql.execute: type: string description: >- These template queries run for each table the first time a record is streamed for them. default: >- CREATE OR REPLACE DYNAMIC TABLE {{table}}_DT TARGET_LAG='15 minutes' WAREHOUSE={{warehouse}} AS SELECT * EXCLUDE dedupe_id FROM( SELECT *, ROW_NUMBER() OVER (PARTITION BY {{primaryKeyColumns}} ORDER BY _streamkap_ts_ms DESC, _streamkap_offset DESC) AS dedupe_id FROM {{table}} ) WHERE dedupe_id = 1 AND __deleted = 'false'; CREATE OR REPLACE TASK {{table}}_CT WAREHOUSE={{warehouse}} SCHEDULE='4380 minutes' TASK_AUTO_RETRY_ATTEMPTS=3 ALLOW_OVERLAPPING_EXECUTION=FALSE AS DELETE FROM {{table}} WHERE NOT EXISTS ( SELECT 1 FROM ( SELECT {{primaryKeyColumns}}, MAX(_streamkap_ts_ms) AS max_timestamp FROM {{table}} GROUP BY {{primaryKeyColumns}} ) AS subquery WHERE {{{keyColumnsAndCondition}}} AND {{table}}._streamkap_ts_ms = subquery.max_timestamp); ALTER TASK {{table}}_CT RESUME sql.table.name: type: string description: >- Can be used as {{dynamicTableName}} in dynamic table creation SQL. It can use input JSON data for more complex mappings and logic. default: '{{table}}_DT' create.sql.data: type: string description: >- Use {"TABLE_DATA": {"{table_name}": {"{key}": "{value}"}, ...}, ...} to set table specific data. This data will be available in the custom SQL templates e.g. SELECT {{key}}. auto.qa.dedupe.table.mapping: type: string description: >- Mapping between the tables that store append-only data and the deduplicated tables. The dedupeTable in mapping will be used for QA scripts. If dedupeSchema is not specified, the deduplicated table will be created in the same schema as the raw table. snowflake.topic2table.map: type: string description: >- Define custom topic-to-table name mapping using regex. Format: matching_pattern:replacement_pattern. Use $1, $2, etc. for captured groups. Example: ^([-\w]+\.)([-\w]+\.)?([-\w]+\.)?([-\w]+\.)?([-\w]+):$5 uses only the last segment as table name. default: >- REGEX_MATCHER>^([-\w]+\.)([-\w]+\.)?([-\w]+\.)?([-\w]+\.)?([-\w]+):$5 required: - snowflake.url.name - snowflake.user.name - snowflake.private.key - snowflake.private.key.passphrase - snowflake.database.name - snowflake.schema.name - snowflake.role.name SqlserverDestinationConfig: type: object title: SQL Server description: Configuration properties for the SQL Server destination connector. properties: database.hostname.user.defined: type: string description: SQL Server Hostname database.port.user.defined: type: string description: SQL Server Port. For example, 1433 default: '1433' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database table.name.prefix: type: string description: Schema for the associated table name schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - table.name.prefix StarburstDestinationConfig: type: object title: Starburst description: Configuration properties for the Starburst destination connector. properties: aws.access.key.id: type: string description: The AWS Access Key ID used to connect to Starburst. aws.secret.access.key: type: string format: password description: The AWS Secret Access Key used to connect to Starburst. aws.s3.region: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 description: The AWS region to be used default: us-west-2 aws.s3.bucket.name: type: string description: The Bucket to use. format.user.defined: type: string enum: - CSV - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: CSV file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to Starburst. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name WeaviateDestinationConfig: type: object title: Weaviate description: Configuration properties for the Weaviate destination connector. properties: weaviate.connection.url: type: string description: Weaviate connection URL default: http://localhost:8080 weaviate.grpc.url: type: string description: Weaviate gRPC connection URL default: localhost:50051 weaviate.grpc.secured: type: string description: Enable TLS encryption for gRPC connection default: false weaviate.auth.scheme: type: string enum: - NONE - API_KEY - OIDC_CLIENT_CREDENTIALS description: Authentication mechanism to use to connect to Weaviate default: NONE weaviate.api.key: type: string format: password description: User API Key for API Key authentication weaviate.oidc.client.secret: type: string format: password description: User OIDC client secret for OIDC authentication weaviate.oidc.scopes: type: string description: OIDC client scopes (comma-separated) default: openid weaviate.headers: type: string description: >- Custom headers to provide (format: key=value, one per line). Example: X-OpenAI-Api-Key=your-key collection.mapping: type: string description: >- Mapping between Kafka topic and Weaviate collection. Use ${topic} for dynamic topic name substitution default: ${topic} schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic document.id.strategy.user.defined: type: string enum: - None - Kafka ID - Field ID description: Strategy to generate document IDs default: None document.id.field.name: type: string description: Field name containing the ID in Kafka record default: id vector.strategy.user.defined: type: string enum: - None - Field Vector description: Strategy to generate or extract document embeddings default: None vector.field.name: type: string description: Field name containing the embedding vector default: vector weaviate.vectorizer: type: string enum: - none - text2vec-weaviate - text2vec-cohere - text2vec-jinaai - text2vec-openai - text2vec-voyageai description: >- Sets the default Weaviate vectorizer to use for objects without explicit vector data. Applies to new collections only. default: text2vec-weaviate delete.enabled: type: string description: >- Treat null record values as deletes. Requires Document ID Strategy to be set to either 'Kafka ID' or 'Field ID' (not 'None') default: false batch.size: type: string description: Number of records per batch default: '100' pool.size: type: string description: Number of thread pools to process batches default: '1' await.termination.ms: type: string description: Timeout for batch processing in milliseconds default: '10000' max.connection.retries: type: string description: Maximum number of retries for connection issues default: '3' max.timeout.retries: type: string description: Maximum number of retries for timeout issues default: '3' retry.interval: type: string description: Interval between retries in milliseconds default: '2000' retry.backoff.ms: type: string description: Backoff time between retries in milliseconds default: '1000' consistency.level: type: string enum: - ALL - ONE - QUORUM description: Consistency level for writes (ALL, ONE, QUORUM) default: QUORUM required: - weaviate.connection.url - weaviate.grpc.url - weaviate.api.key - weaviate.oidc.client.secret securitySchemes: FronteggHTTPAuthentication: type: http scheme: bearer ````