> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamkap.com/llms.txt > Use this file to discover all available pages before exploring further. # Update Existing Destination > Update an existing destination connector. The `config` object contains connector-specific properties using **dot notation**. Use the **GET /destinations/connectors** endpoint with a `connector_code` query parameter to retrieve the full configuration schema for a specific connector type. ## OpenAPI ````yaml /openapi/openapi.json put /destinations/{destination_id} openapi: 3.1.0 info: title: Streamkap REST API description: >- The Streamkap REST API allows you to programmatically manage your CDC pipelines, sources, destinations, transforms, and more. Authenticate using a bearer token obtained from the [Access Token](/api-reference/auth/access-token) endpoint with your API client credentials. contact: name: Streamkap Support url: https://streamkap.com/ email: support@streamkap.com license: name: Proprietary version: 2.0.0 servers: - url: https://api.streamkap.com description: Production security: [] tags: - name: Alerts description: Manage alert subscribers, preferences, and notification credentials. - name: Authentication description: Obtain and refresh access tokens, manage client credentials and roles. - name: Billing description: Retrieve usage metrics, summaries, and export billing data. - name: Consumer Groups description: List, inspect, and reset Kafka consumer group offsets. - name: Dashboard description: Retrieve organisation-level statistics and overview data. - name: Destinations description: Create, configure, and manage data destinations and their lifecycle. - name: Kafka Access description: Manage Kafka users and their access permissions. - name: Logs description: Query and summarise system logs. - name: Pipelines description: Create, configure, and manage CDC pipelines and their lifecycle. - name: Project Keys description: Manage project keys that bundle API credentials and optional Kafka access. - name: Schema Registry description: Browse schema subjects, versions, and retrieve schema definitions. - name: Services description: View and switch service metadata. - name: Sources description: >- Create, configure, and manage data sources, snapshots, and their lifecycle. - name: Tags description: Create, update, and manage resource tags for organisation. - name: Topics description: Browse topic details, statistics, configurations, metrics, and messages. - name: Transforms description: >- Create, deploy, and manage data transforms, unit tests, and implementation details. paths: /destinations/{destination_id}: put: tags: - Destinations summary: Update Existing Destination description: >- Update an existing destination connector. The `config` object contains connector-specific properties using **dot notation**. Use the **GET /destinations/connectors** endpoint with a `connector_code` query parameter to retrieve the full configuration schema for a specific connector type. operationId: updateDestination parameters: - name: destination_id in: path required: true schema: anyOf: - type: string - type: 'null' title: Destination Id - name: secret_returned in: query required: false schema: type: boolean description: Whether to include secret values in the response default: false title: Secret Returned description: Whether to include secret values in the response - name: wait in: query required: false schema: type: boolean description: >- When False, save config and return immediately without waiting for KC update default: true title: Wait description: >- When False, save config and return immediately without waiting for KC update requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/UpdateDestinationReq' responses: '200': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/DestinationConnector' '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - FronteggHTTPAuthentication: [] components: schemas: UpdateDestinationReq: properties: name: type: string title: Name description: Display name for the destination connector. connector: type: string title: Connector description: >- Connector type identifier (e.g., 'snowflake', 'bigquery', 'clickhouse', 'postgresql', 's3'). config: oneOf: - $ref: '#/components/schemas/AzblobDestinationConfig' - $ref: '#/components/schemas/BigqueryDestinationConfig' - $ref: '#/components/schemas/ClickhouseDestinationConfig' - $ref: '#/components/schemas/CockroachdbDestinationConfig' - $ref: '#/components/schemas/DatabricksDestinationConfig' - $ref: '#/components/schemas/Db2DestinationConfig' - $ref: '#/components/schemas/GcsDestinationConfig' - $ref: '#/components/schemas/HttpsinkDestinationConfig' - $ref: '#/components/schemas/IcebergDestinationConfig' - $ref: '#/components/schemas/KafkaDestinationConfig' - $ref: '#/components/schemas/KafkadirectDestinationConfig' - $ref: '#/components/schemas/MotherduckDestinationConfig' - $ref: '#/components/schemas/MysqlDestinationConfig' - $ref: '#/components/schemas/OracleDestinationConfig' - $ref: '#/components/schemas/PineconeDestinationConfig' - $ref: '#/components/schemas/PostgresqlDestinationConfig' - $ref: '#/components/schemas/R2DestinationConfig' - $ref: '#/components/schemas/RedisDestinationConfig' - $ref: '#/components/schemas/RedshiftDestinationConfig' - $ref: '#/components/schemas/S3DestinationConfig' - $ref: '#/components/schemas/SnowflakeDestinationConfig' - $ref: '#/components/schemas/SqlserverDestinationConfig' - $ref: '#/components/schemas/StarburstDestinationConfig' - $ref: '#/components/schemas/WeaviateDestinationConfig' title: Config description: >- Connector-specific configuration properties. The schema depends on the connector type specified in the 'connector' field. kc_cluster_id: anyOf: - type: string - type: 'null' title: Kc Cluster Id description: KC cluster to deploy this connector to. Omit for default cluster. tags: anyOf: - items: type: string type: array - type: 'null' title: Tags description: List of tag IDs to assign to this destination additionalProperties: true type: object required: - name - connector - config title: UpdateDestinationReq description: Request body for updating an existing destination connector. DestinationConnector: properties: name: anyOf: - type: string - type: 'null' title: Name description: User-defined connector name connector: anyOf: - type: string - type: 'null' title: Connector description: Connector type (e.g., 'snowflake', 'bigquery', 'clickhouse') id: anyOf: - type: string - type: 'null' title: Id description: Unique identifier connector_display_name: anyOf: - type: string - type: 'null' title: Connector Display Name description: Human-readable connector type name created_timestamp: anyOf: - type: string - type: string format: date-time - type: 'null' title: Created Timestamp description: Creation timestamp (ISO 8601) sub_id: anyOf: - type: string - type: 'null' title: Sub Id description: Subscription identifier tenant_id: anyOf: - type: string - type: 'null' title: Tenant Id description: Tenant identifier for multi-tenancy config: anyOf: - oneOf: - $ref: '#/components/schemas/AzblobDestinationConfig' - $ref: '#/components/schemas/BigqueryDestinationConfig' - $ref: '#/components/schemas/ClickhouseDestinationConfig' - $ref: '#/components/schemas/CockroachdbDestinationConfig' - $ref: '#/components/schemas/DatabricksDestinationConfig' - $ref: '#/components/schemas/Db2DestinationConfig' - $ref: '#/components/schemas/GcsDestinationConfig' - $ref: '#/components/schemas/HttpsinkDestinationConfig' - $ref: '#/components/schemas/IcebergDestinationConfig' - $ref: '#/components/schemas/KafkaDestinationConfig' - $ref: '#/components/schemas/KafkadirectDestinationConfig' - $ref: '#/components/schemas/MotherduckDestinationConfig' - $ref: '#/components/schemas/MysqlDestinationConfig' - $ref: '#/components/schemas/OracleDestinationConfig' - $ref: '#/components/schemas/PineconeDestinationConfig' - $ref: '#/components/schemas/PostgresqlDestinationConfig' - $ref: '#/components/schemas/R2DestinationConfig' - $ref: '#/components/schemas/RedisDestinationConfig' - $ref: '#/components/schemas/RedshiftDestinationConfig' - $ref: '#/components/schemas/S3DestinationConfig' - $ref: '#/components/schemas/SnowflakeDestinationConfig' - $ref: '#/components/schemas/SqlserverDestinationConfig' - $ref: '#/components/schemas/StarburstDestinationConfig' - $ref: '#/components/schemas/WeaviateDestinationConfig' - type: 'null' title: Config description: Connector configuration parameters topic_ids: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topic Ids description: List of associated topic identifiers topic_map: anyOf: - additionalProperties: items: anyOf: - type: string - type: integer type: array type: object - type: 'null' title: Topic Map description: Mapping of topics to their partitions or related entities topics: anyOf: - items: anyOf: - type: string - type: integer type: array - type: 'null' title: Topics description: List of topic names tasks: anyOf: - items: type: integer type: array - type: 'null' title: Tasks description: List of task identifiers connector_status: anyOf: - type: string - type: 'null' title: Connector Status description: >- Current status: Active, Paused, Stopped, Broken, Starting, Unassigned, Unknown, Pending desired_state: anyOf: - type: string - type: 'null' title: Desired State description: 'Desired state: Pending, Active, Paused, Stopped' kc_cluster_id: anyOf: - type: string - type: 'null' title: Kc Cluster Id description: >- KC cluster this connector is deployed to. None means default cluster. task_statuses: anyOf: - additionalProperties: additionalProperties: anyOf: - type: string - type: integer type: object type: object - type: 'null' title: Task Statuses description: Status information for each connector task tags: anyOf: - items: type: string type: array - type: 'null' title: Tags description: List of tag IDs assigned to this destination additionalProperties: true type: object title: DestinationConnector description: Destination connector configuration and status. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError AzblobDestinationConfig: type: object title: Azure Blob Storage description: >- Configuration properties for the Azure Blob Storage destination connector. properties: azblob.connection.string.user.defined: type: string format: password description: >- The connection string (for Account access keys) or Blob SAS URL (for Shared Access Signatures) azblob.container.name: type: string description: The name of an existing blob container to use format.user.defined: type: string enum: - json - csv - avro - parquet description: The format to use when writing data to file storage default: json format.csv.write.headers: type: string description: Include or exclude column name header row per file default: false topics.dir: type: string description: Top level directory for storing the data e.g. myfolder/subfolder file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' flush.size: type: integer minimum: 0 maximum: 2147483647 description: Number of records to write per file default: 1000 file.size: type: integer minimum: 0 maximum: 2147483647 description: >- Minimum size (in bytes) per file. Records are held in memory until this file size is met or the **Rotate interval** is exceeded default: 65536 rotate.interval.ms: type: integer minimum: -1 maximum: 31556952000 description: >- Maximum time (in milliseconds) to wait before writing records held in memory to file. This ignores the flush and file size settings default: -1 compression: type: string description: The compression type to use when writing data to the storage. required: - azblob.connection.string.user.defined BigqueryDestinationConfig: type: object title: BigQuery description: Configuration properties for the BigQuery destination connector. properties: bigquery.json: type: object format: password description: Upload the Bigquery JSON file. table.name.prefix: type: string description: Name of the destination Dataset for the of the associated table name bigquery.region: type: string enum: - us-east5 - us-central1 - us-west4 - us-west2 - northamerica-northeast1 - us-east4 - us-west1 - us-west3 - southamerica-east1 - southamerica-west1 - us-east1 - northamerica-northeast2 - asia-south2 - asia-east2 - asia-southeast2 - australia-southeast2 - asia-south1 - asia-northeast2 - asia-northeast3 - asia-southeast1 - australia-southeast1 - asia-east1 - asia-northeast1 - europe-west1 - europe-north1 - europe-west3 - europe-west2 - europe-southwest1 - europe-west8 - europe-west4 - europe-west9 - europe-central2 - europe-west6 - EU - US description: ' Region of the bigquery Dataset' default: us-central1 custom.bigquery.cluster.field: type: string description: User can set a Bigquery cluster field. custom.bigquery.partition.field: type: string description: User can set custom name of the partition field bigquery.time.based.partition: type: string description: Is the partition time based? default: false tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - bigquery.json - table.name.prefix ClickhouseDestinationConfig: type: object title: ClickHouse description: Configuration properties for the ClickHouse destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active task default: 5 quote.identifiers: type: string description: Whether to quote identifiers in SQL statements default: false hostname: type: string description: ClickHouse Hostname Or IP address connection.username: type: string description: Username to access ClickHouse connection.password: type: string format: password description: Password to access the ClickHouse port: type: string description: ClickHouse Port. For example, 8443 default: '8443' database: type: string description: ClickHouse database ssl: type: string description: Enable TLS for network connections default: true topics.config.map: type: string description: Per topic configuration in JSON format clickhouse.json.support: type: string description: >- Allow JSON data type in ClickHouse, make sure the ClickHouse server supports it. If not set, the connector will use String type for JSON data. default: false schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic required: - hostname - connection.username - connection.password - port - database CockroachdbDestinationConfig: type: object title: CockroachDB description: Configuration properties for the CockroachDB destination connector. properties: database.hostname.user.defined: type: string description: CockroachDB Hostname database.port.user.defined: type: string description: CockroachDB Port. For example, 26257 default: '26257' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database table.name.prefix: type: string description: Schema for the associated table name default: public schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - table.name.prefix DatabricksDestinationConfig: type: object title: Databricks description: Configuration properties for the Databricks destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert databricks.token: type: string format: password description: Token connection.url.user.defined: type: string description: JDBC URL connection.timeout.user.defined: type: integer minimum: 1 description: Connection Timeout default: '180' databricks.catalog.user.defined: type: string description: The name of the Databricks catalog to use. default: hive_metastore partition.mode: type: string enum: - by_topic - by_partition - by_topic_and_partition description: Partition tables or not default: by_topic schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name default: streamkap hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false tasks.max: type: integer minimum: 1 maximum: 100 description: >- The maximum number of active tasks. NOTE: Increasing this value may increase parallelism and throughput but can also lead to higher costs on databricks side. default: 5 consumer.wait.time.for.larger.batch.ms: type: string enum: - value: '500' label: 0.5s — Realtime - value: '5000' label: 5s — Low latency - value: '10000' label: 10s — Default - value: '20000' label: 20s — Moderate batch - value: '30000' label: 30s — Batch - value: '60000' label: 1min — Large batch - value: '120000' label: 2min — Bulk - value: '180000' label: 3min — Heavy bulk - value: '240000' label: 4min — Max throughput - value: '300000' label: 5min — Maximum batch description: >- Controls how long the connector waits to accumulate a larger batch before writing to Databricks. Higher values = larger batches = better throughput and fewer MERGE INTOs, but higher latency. default: '10000' topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - databricks.token - connection.url.user.defined Db2DestinationConfig: type: object title: Db2 description: Configuration properties for the Db2 destination connector. properties: database.hostname.user.defined: type: string description: Db2 Hostname database.port.user.defined: type: string description: Db2 Port. For example, 50000 default: '50000' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password GcsDestinationConfig: type: object title: GCS description: Configuration properties for the GCS destination connector. properties: gcs.credentials.json: type: string format: password description: >- GCP credentials as a JSON string with escaped quotes (" into \"). \n in the private_key field should also be escaped as \\n. Example format: {\"type\": \"service_account\",\"project_id\": \"XXXXXX\", ...} gcs.bucket.name: type: string description: The GCS Bucket to use. format.user.defined: type: string enum: - CSV - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: CSV file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to GCS. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - gcs.bucket.name HttpsinkDestinationConfig: type: object title: HTTP Sink description: Configuration properties for the HTTP Sink destination connector. properties: http.url: type: string description: The URL to send data to. http.authorization.type: type: string enum: - none - static - oauth2 description: The HTTP authorization type. default: none http.headers.authorization: type: string description: >- Static content of Authorization header (required for STATIC authorization). oauth2.access.token.url: type: string description: URL for fetching OAuth2 access token. oauth2.client.id: type: string description: OAuth2 client id. oauth2.client.secret: type: string format: password description: OAuth2 client secret. oauth2.scope: type: string description: OAuth2 client scope. http.headers.content.type: type: string description: Value of Content-Type header for each request. default: application/json http.headers.additional: type: string description: Additional headers in header:value format, comma-separated. http.proxy.host: type: string description: Proxy hostname. http.proxy.port: type: string description: Proxy port. batching.enabled: type: string description: Enable batching multiple records in a single HTTP request. default: false batch.max.size: type: integer minimum: 1 maximum: 1000000 description: Maximum size of a record batch (1 to 1,000,000). default: 500 batch.buffering.enabled: type: string description: >- Enable buffering and only send records when batch.max.size reaches or batch.max.time.ms hits. default: false batch.max.time.ms: type: integer minimum: 1 maximum: 1000000000 description: >- Maximum time to wait before flushing a batch (1 to 1,000,000,000 ms). default: 10000 batch.prefix: type: string description: Prefix for record batches. default: '[' batch.suffix: type: string description: Suffix for record batches. default: ']' batch.separator: type: string description: Separator for records in a batch. default: ',' max.retries: type: integer minimum: 0 maximum: 2147483647 description: Maximum number of retries on errors. default: 1 retry.backoff.ms: type: integer minimum: 0 maximum: 2147483647 description: Time in ms to wait before retrying. default: 3000 http.timeout: type: integer minimum: 0 maximum: 2147483647 description: HTTP response timeout in seconds. default: 30 decimal.format: type: string enum: - BASE64 - NUMERIC description: 'Format for serializing decimals: BASE64 (default) or NUMERIC.' default: NUMERIC errors.tolerance.user.defined: type: string enum: - none - all description: >- Error tolerance setting. Note: Cannot be used when Batching is enabled. default: none required: - http.url - oauth2.access.token.url - oauth2.client.id - oauth2.client.secret - oauth2.scope IcebergDestinationConfig: type: object title: Iceberg description: Configuration properties for the Iceberg destination connector. properties: iceberg.catalog.type: type: string enum: - value: rest label: REST - value: hive label: Hive - value: glue label: AWS Glue description: Type of Iceberg catalog default: rest iceberg.catalog.auth.mode: type: string enum: - value: none label: None (no auth) - value: token label: Token - value: oauth2 label: OAuth 2.0 description: >- How to authenticate with the Iceberg catalog. 'No Auth' for catalogs that don't require authentication. 'Token' for bearer token authentication (e.g., Cloudflare R2). 'OAuth2 Credential' for OAuth2 client credential authentication (e.g., Polaris, DataHub). default: none iceberg.catalog.name: type: string description: Iceberg catalog name iceberg.catalog.client.assume-role.arn: type: string description: AWS IAM role (e.g., arn:aws:iam:::role/) iceberg.catalog.uri: type: string description: >- Catalog endpoint URL. For REST catalogs use https://… ; for Hive Metastore use thrift://… ; AWS Glue uses the regional endpoint. iceberg.catalog.token: type: string format: password description: Bearer token used for catalog authentication. iceberg.catalog.credential: type: string format: password description: OAuth2 client credential in client_id:client_secret format. iceberg.catalog.scope: type: string description: OAuth2 scope for catalog authentication (e.g., PRINCIPAL_ROLE:ALL). default: PRINCIPAL_ROLE:ALL iceberg.catalog.s3.credentials.enabled: type: boolean description: >- Enable to provide your own S3 access key and secret. Disable if the catalog vends storage credentials automatically. default: false iceberg.catalog.s3.access-key-id: type: string description: The access key ID used to connect to S3 or S3-compatible storage. iceberg.catalog.s3.secret-access-key: type: string format: password description: >- The secret access key used to connect to S3 or S3-compatible storage. iceberg.catalog.client.region.user.defined: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 - auto description: The AWS region to be used. For Cloudflare R2, select 'auto'. default: us-west-2 iceberg.catalog.warehouse: type: string description: >- The Iceberg warehouse identifier. For R2 Data Catalog, this should be the warehouse name configured in the catalog. For S3-based catalogs, this is typically an S3 or S3-compatible path (e.g., s3://bucket/path). table.name.prefix: type: string description: >- Name of the database schema that contains the table (e.g., public, sales, analytics). insert.mode.user.defined: type: string enum: - value: insert label: Insert only - value: upsert label: Upsert (merge) description: Specifies the strategy used to insert events into the database default: insert iceberg.tables.default-id-columns: type: string description: >- Optional. A comma-separated list of field names to use as record identifiers when key fields are not present in Kafka messages iceberg.tables.hard-delete-enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true iceberg.tables.compaction.enabled: type: string description: Enable lightweight compaction of small data files after commits default: false iceberg.tables.compaction.target-file-size-bytes: type: string description: >- Target file size in bytes for compaction. Files smaller than this are candidates for merging default: '134217728' iceberg.tables.compaction.min-small-files: type: string description: Minimum number of small files required before triggering compaction default: '5' iceberg.tables.compaction.max-files-per-run: type: string description: Maximum number of small files to compact in a single run default: '100' iceberg.tables.compaction.commit-threshold: type: string description: Number of Iceberg commits between compaction checks default: '5' iceberg.tables.compaction.expire-snapshots: type: string description: >- Expire old snapshots and delete unreferenced data files after compaction default: false iceberg.tables.compaction.retain-last: type: string description: >- Number of most recent snapshots to retain when expiring snapshots after compaction default: '1' required: - iceberg.catalog.token - iceberg.catalog.credential - iceberg.catalog.scope - iceberg.catalog.s3.access-key-id - iceberg.catalog.s3.secret-access-key - iceberg.catalog.warehouse - table.name.prefix KafkaDestinationConfig: type: object title: Kafka description: Configuration properties for the Kafka destination connector. properties: kafka.sink.bootstrap: type: string description: >- A comma-separated list of host and port pairs that are the addresses of the Destination Kafka brokers. This list should be in the form host1:port1,host2:port2,... destination.format: type: string enum: - avro - json description: The format to use when writing data to kafka default: json json.schema.enable: type: string description: Include schema in json message default: false schema.registry.url.user.defined: type: string description: Destination kafka schema registry url topic.prefix: type: string description: Prefix for destination topics topic.suffix: type: string description: Suffix for destination topics transforms.InsertField.offset.field: type: string description: >- If Kafka destination is used to sync data between services or to apply some transformations, it would be better to rename this field to _streamkap_offset_sync or _streamkap_offset_transform to avoid conflicts with the destination on the other service default: _streamkap_offset transforms.changeTopicName.add.original.topic: type: string enum: - 'off' - header - field description: >- Add the original topic name to the output record. 'off' disables it, 'header' adds it as a Kafka header, 'field' adds it as a _streamkap_topic column in the destination table. default: 'off' tasks.max: type: integer minimum: 1 maximum: 100 description: Maximum number of tasks for the connector default: 5 required: - kafka.sink.bootstrap - schema.registry.url.user.defined KafkadirectDestinationConfig: type: object title: Kafka Direct description: Configuration properties for the Kafka Direct destination connector. properties: password: type: string format: password description: Password for setting up the Kafka proxy whitelist.ips.user.defined: type: string description: A comma-separated list of IPs/CIDRs required: - password - whitelist.ips.user.defined MotherduckDestinationConfig: type: object title: Motherduck description: Configuration properties for the Motherduck destination connector. properties: ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: Upsert or append modes are available default: upsert motherduck.token: type: string format: password description: Motherduck Token motherduck.catalog.user.defined: type: string description: The name of the Motherduck database/catalog to use. schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name default: streamkap hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false tasks.max: type: integer minimum: 1 maximum: 25 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - motherduck.token - motherduck.catalog.user.defined MysqlDestinationConfig: type: object title: MySQL description: Configuration properties for the MySQL destination connector. properties: database.hostname.user.defined: type: string description: MySQL Hostname database.port.user.defined: type: string description: MySQL Port. For example, 3306 default: '3306' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password OracleDestinationConfig: type: object title: Oracle description: Configuration properties for the Oracle destination connector. properties: database.hostname.user.defined: type: string description: Oracle Hostname database.port.user.defined: type: string description: Oracle Port. For example, 1521 default: '1521' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password PineconeDestinationConfig: type: object title: Pinecone description: Configuration properties for the Pinecone destination connector. properties: pinecone.api.key: type: string format: password description: API key for authenticating with Pinecone pinecone.index.name: type: string description: Name of the target Pinecone index pinecone.proxy.host: type: string description: Proxy host for Pinecone connections pinecone.proxy.port: type: string description: Proxy port for Pinecone connections (required if Proxy Host is set) collection.mapping: type: string description: >- Mapping between Kafka topic and Pinecone namespace. Use ${topic} for dynamic topic name substitution default: ${topic} schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created namespaces, set to `NONE` default: basic document.id.strategy.user.defined: type: string enum: - None - Kafka ID - Field ID description: >- Strategy to generate document IDs. 'None' uses random UUIDs (append-only, no upsert/delete support) default: None document.id.field.name: type: string description: Field name containing the ID in Kafka record default: id vector.field.name: type: string description: Field name containing the embedding vector default: vector delete.enabled: type: string description: >- Treat null record values or __deleted=true as deletes. Requires Document ID Strategy to be set to 'Kafka ID' (not 'None') default: false batch.size: type: string description: Number of records per batch default: '100' retry.max: type: string description: Maximum number of retry attempts for failed operations default: '3' retry.backoff.ms: type: string description: Backoff time between retries in milliseconds default: '1000' required: - pinecone.api.key - pinecone.index.name - vector.field.name PostgresqlDestinationConfig: type: object title: PostgreSQL description: Configuration properties for the PostgreSQL destination connector. properties: database.hostname.user.defined: type: string description: PostgreSQL Hostname database.port.user.defined: type: string description: PostgreSQL Port. For example, 5432 default: '5432' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database ssh.enabled: type: boolean description: >- Streamkap will connect to SSH server in your network which has access to your database. This is necessary if Streamkap cannot connect directly to your database. default: false ssh.host: type: string description: Hostname of your SSH server ssh.port: type: string description: Port of your SSH server default: '22' ssh.user: type: string description: User that allows Streamkap to connect to SSH server default: streamkap table.name.prefix: type: string description: Schema for the associated table name schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line ssh.public.key.user.displayed: type: string description: Public key to add to SSH server default: required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - ssh.host - ssh.port - ssh.user - table.name.prefix - ssh.public.key.user.displayed R2DestinationConfig: type: object title: R2 description: Configuration properties for the R2 destination connector. properties: r2.account.user.defined: type: string description: >- Cloudflare R2 account ID. This is the account ID you see in the Cloudflare dashboard. aws.access.key.id: type: string description: The Access Key ID used to connect to R2. aws.secret.access.key: type: string format: password description: The Secret Access Key used to connect to R2. aws.s3.bucket.name: type: string description: The R2 Bucket to use. format.user.defined: type: string enum: - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: JSON Array file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to S3. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - r2.account.user.defined - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name RedisDestinationConfig: type: object title: Redis description: Configuration properties for the Redis destination connector. properties: redis.host.user.defined: type: string description: Redis Hostname redis.port.user.defined: type: string description: Redis Port. For example, 6379 default: '6379' redis.username: type: string description: Username to access the Redis database redis.password: type: string format: password description: Password to access the Redis database ssl.enabled: type: string description: Enable TLS for network connections default: true redis.key: type: string description: Redis key to stream the data redis.key.data.type: type: string enum: - Stream - List - Hash description: >- Data Type of Redis Key. At the moment, only Stream and List are supported default: Stream tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 required: - redis.host.user.defined - redis.port.user.defined - redis.username - redis.password - redis.key RedshiftDestinationConfig: type: object title: Redshift description: Configuration properties for the Redshift destination connector. properties: tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 aws.redshift.domain: type: string description: Domain of the leader of the cluster aws.redshift.port: type: string description: Port number for incoming connections to the leader default: '5439' aws.redshift.database: type: string description: Name of the database on the cluster connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database primary.key.fields: type: string description: List of comma-separated primary key field names default: id schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic table.name.prefix: type: string description: Schema for the associated table name required: - aws.redshift.domain - aws.redshift.port - aws.redshift.database - connection.username - connection.password - primary.key.fields - table.name.prefix S3DestinationConfig: type: object title: S3 description: Configuration properties for the S3 destination connector. properties: aws.access.key.id: type: string description: The AWS Access Key ID used to connect to S3. aws.secret.access.key: type: string format: password description: The AWS Secret Access Key used to connect to S3. aws.s3.region: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 description: The AWS region to be used default: us-west-2 aws.s3.bucket.name: type: string description: The S3 Bucket to use. format.user.defined: type: string enum: - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: JSON Array file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to S3. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name SnowflakeDestinationConfig: type: object title: Snowflake description: Configuration properties for the Snowflake destination connector. properties: snowflake.url.name: type: string description: >- The URL for accessing your Snowflake account. This URL must include your account identifier. Note that the protocol (https://) and port number are optional. snowflake.user.name: type: string description: |+ User login name for the Snowflake account. snowflake.private.key: type: string format: password description: >- The private key to authenticate the user. Include only the key, not the header or footer. If the key is split across multiple lines, remove the line breaks. snowflake.private.key.passphrase.secured: type: string description: >- If checked (default), provide your SSH key's passphrase, otherwise, uncheck for SSH keys without passphrase. default: true snowflake.private.key.passphrase: type: string format: password description: The passphrase is used to decrypt the private key. sfwarehouse: type: string description: The name of the snowflake warehouse. default: STREAMKAP_WH snowflake.database.name: type: string description: >- The name of the database that contains the table to insert rows into. snowflake.schema.name: type: string description: The name of the schema that contains the table to insert rows into. create.schema.auto: type: string description: >- Automatically generates a Snowflake schema if it does not already exist. default: true snowflake.role.name: type: string description: >- The name of an existing role with necessary privileges (for Streamkap) assigned to the default: STREAMKAP_ROLE ingestion.mode: type: string enum: - value: upsert label: Upsert (merge) - value: append label: Append only description: >- Upsert or append modes are available. NOTE: when switching append to upsert, existing data must be deduplicated or deleted. Read more about upsert mode default: append hard.delete: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database (applies to `upsert` only) default: true schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic use.hybrid.tables: type: string description: >- Specifies whether the connector should create Hybrid Tables (applies to `upsert` only) default: false apply.dynamic.table.script: type: string description: >- Specifies whether the connector should create Dynamic Tables & Cleanup Tasks (applies to `append` only) default: false create.sql.execute: type: string description: >- These template queries run for each table the first time a record is streamed for them. default: >- CREATE OR REPLACE DYNAMIC TABLE {{table}}_DT TARGET_LAG='15 minutes' WAREHOUSE={{warehouse}} AS SELECT * EXCLUDE dedupe_id FROM( SELECT *, ROW_NUMBER() OVER (PARTITION BY {{primaryKeyColumns}} ORDER BY _streamkap_ts_ms DESC, _streamkap_offset DESC) AS dedupe_id FROM {{table}} ) WHERE dedupe_id = 1 AND __deleted = 'false'; CREATE OR REPLACE TASK {{table}}_CT WAREHOUSE={{warehouse}} SCHEDULE='4380 minutes' TASK_AUTO_RETRY_ATTEMPTS=3 ALLOW_OVERLAPPING_EXECUTION=FALSE AS DELETE FROM {{table}} WHERE NOT EXISTS ( SELECT 1 FROM ( SELECT {{primaryKeyColumns}}, MAX(_streamkap_ts_ms) AS max_timestamp FROM {{table}} GROUP BY {{primaryKeyColumns}} ) AS subquery WHERE {{{keyColumnsAndCondition}}} AND {{table}}._streamkap_ts_ms = subquery.max_timestamp); ALTER TASK {{table}}_CT RESUME sql.table.name: type: string description: >- Can be used as {{dynamicTableName}} in dynamic table creation SQL. It can use input JSON data for more complex mappings and logic. default: '{{table}}_DT' create.sql.data: type: string description: >- Use {"TABLE_DATA": {"{table_name}": {"{key}": "{value}"}, ...}, ...} to set table specific data. This data will be available in the custom SQL templates e.g. SELECT {{key}}. auto.qa.dedupe.table.mapping: type: string description: >- Mapping between the tables that store append-only data and the deduplicated tables. The dedupeTable in mapping will be used for QA scripts. If dedupeSchema is not specified, the deduplicated table will be created in the same schema as the raw table. snowflake.topic2table.map: type: string description: >- Define custom topic-to-table name mapping using regex. Format: matching_pattern:replacement_pattern. Use $1, $2, etc. for captured groups. Example: ^([-\w]+\.)([-\w]+\.)?([-\w]+\.)?([-\w]+\.)?([-\w]+):$5 uses only the last segment as table name. default: >- REGEX_MATCHER>^([-\w]+\.)([-\w]+\.)?([-\w]+\.)?([-\w]+\.)?([-\w]+):$5 required: - snowflake.url.name - snowflake.user.name - snowflake.private.key - snowflake.private.key.passphrase - snowflake.database.name - snowflake.schema.name - snowflake.role.name SqlserverDestinationConfig: type: object title: SQL Server description: Configuration properties for the SQL Server destination connector. properties: database.hostname.user.defined: type: string description: SQL Server Hostname database.port.user.defined: type: string description: SQL Server Port. For example, 1433 default: '1433' database.database.user.defined: type: string description: Database name connection.username: type: string description: Username to access the database connection.password: type: string format: password description: Password to access with the database table.name.prefix: type: string description: Schema for the associated table name schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic insert.mode: type: string enum: - insert - upsert description: Specifies the strategy used to insert events into the database default: insert delete.enabled: type: string description: >- Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database default: false primary.key.mode.user.defined: type: string enum: - none - record_key - record_value description: >- Specifies how the connector resolves the primary key columns from the event default: record_key primary.key.fields: type: string description: >- Optional. Either the name of the primary key column or a comma-separated list of fields to derive the primary key from. tasks.max: type: integer minimum: 1 maximum: 10 description: The maximum number of active tasks default: 5 topic2table.map.user.defined: type: string description: Falls back to Streamkap's default for tables where no match is found default: false transforms.changeTopicName.match.regex.user.defined: type: string description: >- Regular expression for matching topic name parts to use as the destination table (database) or file (file storage) name transforms.changeTopicName.mapping: type: string description: >- Map source tables to specific destination tables. Input should be the format of `source_table_name:destination_table_name` separated by a new line required: - database.hostname.user.defined - database.port.user.defined - database.database.user.defined - connection.username - connection.password - table.name.prefix StarburstDestinationConfig: type: object title: Starburst description: Configuration properties for the Starburst destination connector. properties: aws.access.key.id: type: string description: The AWS Access Key ID used to connect to Starburst. aws.secret.access.key: type: string format: password description: The AWS Secret Access Key used to connect to Starburst. aws.s3.region: type: string enum: - ap-south-1 - eu-west-2 - eu-west-1 - ap-northeast-2 - ap-northeast-1 - ca-central-1 - sa-east-1 - cn-north-1 - us-gov-west-1 - ap-southeast-1 - ap-southeast-2 - eu-central-1 - us-east-1 - us-east-2 - us-west-1 - us-west-2 description: The AWS region to be used default: us-west-2 aws.s3.bucket.name: type: string description: The Bucket to use. format.user.defined: type: string enum: - CSV - JSON Lines - JSON Array - Parquet description: The format to use when writing data to the store. default: CSV file.name.template: type: string description: >- The format of the filename. See documentation for more information about formatting options. default: '{{topic}}-{{partition}}-{{start_offset}}' file.name.prefix: type: string description: >- Prefix for the filename. Prefixes can be used to specify a directory for the file (e.g. dir1/dir2/). file.compression.type: type: string enum: - none - gzip - snappy - zstd description: Compression type for files written to Starburst. default: gzip format.output.fields.user.defined: type: array items: type: string description: >- A comma separated list of fields to include in output? Options to include key, offset, timestamp, value, headers. default: - value required: - aws.access.key.id - aws.secret.access.key - aws.s3.bucket.name WeaviateDestinationConfig: type: object title: Weaviate description: Configuration properties for the Weaviate destination connector. properties: weaviate.connection.url: type: string description: Weaviate connection URL default: http://localhost:8080 weaviate.grpc.url: type: string description: Weaviate gRPC connection URL default: localhost:50051 weaviate.grpc.secured: type: string description: Enable TLS encryption for gRPC connection default: false weaviate.auth.scheme: type: string enum: - NONE - API_KEY - OIDC_CLIENT_CREDENTIALS description: Authentication mechanism to use to connect to Weaviate default: NONE weaviate.api.key: type: string format: password description: User API Key for API Key authentication weaviate.oidc.client.secret: type: string format: password description: User OIDC client secret for OIDC authentication weaviate.oidc.scopes: type: string description: OIDC client scopes (comma-separated) default: openid weaviate.headers: type: string description: >- Custom headers to provide (format: key=value, one per line). Example: X-OpenAI-Api-Key=your-key collection.mapping: type: string description: >- Mapping between Kafka topic and Weaviate collection. Use ${topic} for dynamic topic name substitution default: ${topic} schema.evolution: type: string enum: - value: basic label: Basic - value: none label: None description: >- Controls how schema evolution is handled by the sink connector. For pipelines with pre-created destination tables, set to `NONE` default: basic document.id.strategy.user.defined: type: string enum: - None - Kafka ID - Field ID description: Strategy to generate document IDs default: None document.id.field.name: type: string description: Field name containing the ID in Kafka record default: id vector.strategy.user.defined: type: string enum: - None - Field Vector description: Strategy to generate or extract document embeddings default: None vector.field.name: type: string description: Field name containing the embedding vector default: vector weaviate.vectorizer: type: string enum: - none - text2vec-weaviate - text2vec-cohere - text2vec-jinaai - text2vec-openai - text2vec-voyageai description: >- Sets the default Weaviate vectorizer to use for objects without explicit vector data. Applies to new collections only. default: text2vec-weaviate delete.enabled: type: string description: >- Treat null record values as deletes. Requires Document ID Strategy to be set to either 'Kafka ID' or 'Field ID' (not 'None') default: false batch.size: type: string description: Number of records per batch default: '100' pool.size: type: string description: Number of thread pools to process batches default: '1' await.termination.ms: type: string description: Timeout for batch processing in milliseconds default: '10000' max.connection.retries: type: string description: Maximum number of retries for connection issues default: '3' max.timeout.retries: type: string description: Maximum number of retries for timeout issues default: '3' retry.interval: type: string description: Interval between retries in milliseconds default: '2000' retry.backoff.ms: type: string description: Backoff time between retries in milliseconds default: '1000' consistency.level: type: string enum: - ALL - ONE - QUORUM description: Consistency level for writes (ALL, ONE, QUORUM) default: QUORUM required: - weaviate.connection.url - weaviate.grpc.url - weaviate.api.key - weaviate.oidc.client.secret ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError securitySchemes: FronteggHTTPAuthentication: type: http scheme: bearer ````