Create New Pipeline

POST/pipelines

Authorization

AuthorizationBearer token · headerrequired

Query parameters

secret_returnedboolean

Whether to include secret values in the response

default: false

waitboolean

When False, return without waiting for the destination connector's configuration to be updated in Kafka Connect; the change is applied asynchronously and the destination's desired_state shows a Pending state until it is. When True (default), wait for the update.

default: true

Request body

requiredapplication/json

namestringrequired

snapshot_new_tablesboolean

default: false

destinationPipelineDestinationConnectorrequired

Show properties

connectorstringrequired

namestringrequired

idstringrequired

sourcePipelineSourceConnectorrequired

Show properties

connectorstringrequired

namestringrequired

idstringrequired

topicsstring[]required

transformsPipelineTransformConnector[] | null

Show properties

Any of:

PipelineTransformConnector[]

Array of PipelineTransformConnector

topic_idstringrequired

null

null

tagsstring[]required

topic_auto_discovery_transformsTopicAutoDiscoveryTransform[]

default: []

Show properties

Array of TopicAutoDiscoveryTransform

transform_idstringrequired

regexstringrequired

created_fromEntityOriginEnum | null

default: "api"

Show properties

Any of:

EntityOriginEnum

string

null

null

Responses

200

Successful Response

namestring | null

Show properties

Any of:

string

string

null

null

sub_idstring | null

Show properties

Any of:

string

string

null

null

tenant_idstring | null

Show properties

Any of:

string

string

null

null

idstring | null

Show properties

Any of:

string

string

null

null

snapshot_new_tablesboolean | null

Show properties

Any of:

boolean

boolean

null

null

managedboolean

default: false

topic_idsstring[] | null

Show properties

Any of:

string[]

Array of string

Any of:

string

string

null

null

null

null

topicsstring[] | null

Show properties

Any of:

string[]

Array of string

Any of:

string

string

null

null

null

null

sourceSourceConnector | null

Show properties

Any of:

SourceConnector

namestring | null

User-defined connector name

Show properties

Any of:

string

string

null

null

connectorstring | null

Connector type (e.g., 'postgresql', 'mysql', 'mongodb')

Show properties

Any of:

string

string

null

null

idstring | null

Unique identifier

Show properties

Any of:

string

string

null

null

connector_display_namestring | null

Human-readable connector type name

Show properties

Any of:

string

string

null

null

created_timestampstring | string<date-time> | null

Creation timestamp (ISO 8601)

Show properties

Any of:

string

string

string<date-time>

string<date-time>

null

null

sub_idstring | null

Subscription identifier

Show properties

Any of:

string

string

null

null

tenant_idstring | null

Tenant identifier for multi-tenancy

Show properties

Any of:

string

string

null

null

service_idstring | null

Associated service identifier

Show properties

Any of:

string

string

null

null

configobject | null

The connector's configuration. Its properties per connector type are documented on the create and update requests' config.

Show properties

Any of:

object

object

null

null

topic_idsstring | integer[] | null

List of associated topic identifiers

Show properties

Any of:

string | integer[]

Array of string | integer

Any of:

string

string

integer

integer

null

null

topic_mapobject | null

Mapping of topics to their partitions or related entities

Show properties

Any of:

object

object

null

null

topicsstring | integer[] | null

List of topic names

Show properties

Any of:

string | integer[]

Array of string | integer

Any of:

string

string

integer

integer

null

null

tasksinteger[] | null

List of task identifiers

Show properties

Any of:

integer[]

Array of integer

integer

null

null

connector_statusstring | null

Current status: Active, Paused, Stopped, Broken, Starting, Unassigned, Unknown, Throttled (an api source waiting out its vendor's rate budget; task_statuses carries suspend_reason and resume_at), or a Pending* state echoed from desired_state while a requested change is being applied

Show properties

Any of:

string

string

null

null

desired_statestring | null

Desired state: Active, Paused, Stopped, or a pending change Streamkap is still applying — Pending, Pending Update, Pending Delete, Pending Stop, Pending Pause, Pending Resume, Pending Restart

Show properties

Any of:

string

string

null

null

reconcile_errorstring | null

Why the connector is Broken, as recorded when Streamkap last tried to apply its state. Secrets are masked; absent while the connector is healthy.

Show properties

Any of:

string

string

null

null

connection_warningstring | null

API sources, on the create and update responses only: what the credential and resource checks found that did not stop the save but needs action, e.g. an ad account that is not active, an access token that expires within 14 days, or a resource that returned no rows (each resource's warning prefixed with its name, <resource>: <warning>). The same texts Test connection returns as its message and as each passing resource's warning. Absent when there is nothing to report.

Show properties

Any of:

string

string

null

null

kc_cluster_idstring | null

Kafka Connect cluster this connector is deployed to. None means the default cluster.

Show properties

Any of:

string

string

null

null

task_statusesobject | null

Status information for each connector task

Show properties

Any of:

object

object

null

null

tagsstring[] | null

List of tag IDs assigned to this source

Show properties

Any of:

string[]

Array of string

string

null

null

null

null

destinationDestinationConnector | null

Show properties

Any of:

DestinationConnector

namestring | null

User-defined connector name

Show properties

Any of:

string

string

null

null

connectorstring | null

Connector type (e.g., 'snowflake', 'bigquery', 'clickhouse')

Show properties

Any of:

string

string

null

null

idstring | null

Unique identifier

Show properties

Any of:

string

string

null

null

connector_display_namestring | null

Human-readable connector type name

Show properties

Any of:

string

string

null

null

created_timestampstring | string<date-time> | null

Creation timestamp (ISO 8601)

Show properties

Any of:

string

string

string<date-time>

string<date-time>

null

null

sub_idstring | null

Subscription identifier

Show properties

Any of:

string

string

null

null

tenant_idstring | null

Tenant identifier for multi-tenancy

Show properties

Any of:

string

string

null

null

configobject | null

The connector's configuration. Its properties per connector type are documented on the create and update requests' config.

Show properties

Any of:

object

object

null

null

topic_idsstring | integer[] | null

List of associated topic identifiers

Show properties

Any of:

string | integer[]

Array of string | integer

Any of:

string

string

integer

integer

null

null

topic_mapobject | null

Mapping of topics to their partitions or related entities

Show properties

Any of:

object

object

null

null

topicsstring | integer[] | null

List of topic names

Show properties

Any of:

string | integer[]

Array of string | integer

Any of:

string

string

integer

integer

null

null

tasksinteger[] | null

List of task identifiers

Show properties

Any of:

integer[]

Array of integer

integer

null

null

connector_statusstring | null

Current status: Active, Paused, Stopped, Broken, Starting, Unassigned, Unknown, or a Pending* state echoed from desired_state while a requested change is being applied

Show properties

Any of:

string

string

null

null

desired_statestring | null

Desired state: Active, Paused, Stopped, or a pending change Streamkap is still applying — Pending, Pending Update, Pending Delete, Pending Stop, Pending Pause, Pending Resume, Pending Restart

Show properties

Any of:

string

string

null

null

reconcile_errorstring | null

Why the connector is Broken, as recorded when Streamkap last tried to apply its state. Secrets are masked; absent while the connector is healthy.

Show properties

Any of:

string

string

null

null

schema_evolution_warningstring | null

Advisory: this destination silently drops fields its API sources start sending, because schema evolution is set to 'none'. Absent whenever nothing is at risk. Nothing is blocked — 'none' is a valid choice for a pre-created destination schema.

Show properties

Any of:

string

string

null

null

kc_cluster_idstring | null

Kafka Connect cluster this connector is deployed to. None means the default cluster.

Show properties

Any of:

string

string

null

null

task_statusesobject | null

Status information for each connector task

Show properties

Any of:

object

object

null

null

tagsstring[] | null

List of tag IDs assigned to this destination

Show properties

Any of:

string[]

Array of string

string

null

null

null

null

transformsany[] | null

Show properties

Any of:

any[]

Array of any

any

null

null

tagsstring[] | null

default: []

Show properties

Any of:

string[]

Array of string

string

null

null

periodic_auditUpdatePipelineAuditReq | null

Show properties

Any of:

UpdatePipelineAuditReq

topicsstring | integer[]required

Show properties

Array of string | integer

Any of:

string

string

integer

integer

timestamp_columnstringrequired

interval_minutesintegerrequired

fix_deletes_onlyboolean

default: false

null

null

topic_auto_discovery_transformsTopicAutoDiscoveryTransform[]

default: []

Show properties

Array of TopicAutoDiscoveryTransform

transform_idstringrequired

regexstringrequired

422

Validation Error

detailValidationError[]

Show properties

Array of ValidationError

locstring | integer[]required

Show properties

Array of string | integer

Any of:

string

string

integer

integer

msgstringrequired

typestringrequired

inputany

ctxobject

Request

curl -X POST 'https://api.streamkap.com/pipelines' \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "string",
  "snapshot_new_tables": false,
  "destination": {
    "connector": "string",
    "name": "string",
    "id": "string"
  },
  "source": {
    "connector": "string",
    "name": "string",
    "id": "string",
    "topics": [
      "string"
    ]
  },
  "transforms": [
    {
      "topic_id": "string"
    }
  ],
  "tags": [
    "string"
  ],
  "topic_auto_discovery_transforms": [],
  "created_from": "api"
}'

const response = await fetch("https://api.streamkap.com/pipelines", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_TOKEN",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"name\": \"string\",\n  \"snapshot_new_tables\": false,\n  \"destination\": {\n    \"connector\": \"string\",\n    \"name\": \"string\",\n    \"id\": \"string\"\n  },\n  \"source\": {\n    \"connector\": \"string\",\n    \"name\": \"string\",\n    \"id\": \"string\",\n    \"topics\": [\n      \"string\"\n    ]\n  },\n  \"transforms\": [\n    {\n      \"topic_id\": \"string\"\n    }\n  ],\n  \"tags\": [\n    \"string\"\n  ],\n  \"topic_auto_discovery_transforms\": [],\n  \"created_from\": \"api\"\n}"
});

import requests

response = requests.post(
    "https://api.streamkap.com/pipelines",
    headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"name\": \"string\",\n  \"snapshot_new_tables\": false,\n  \"destination\": {\n    \"connector\": \"string\",\n    \"name\": \"string\",\n    \"id\": \"string\"\n  },\n  \"source\": {\n    \"connector\": \"string\",\n    \"name\": \"string\",\n    \"id\": \"string\",\n    \"topics\": [\n      \"string\"\n    ]\n  },\n  \"transforms\": [\n    {\n      \"topic_id\": \"string\"\n    }\n  ],\n  \"tags\": [\n    \"string\"\n  ],\n  \"topic_auto_discovery_transforms\": [],\n  \"created_from\": \"api\"\n}",
)

Response

{
  "name": "string",
  "sub_id": "string",
  "tenant_id": "string",
  "id": "string",
  "snapshot_new_tables": true,
  "managed": false,
  "topic_ids": [
    "string"
  ],
  "topics": [
    "string"
  ],
  "source": {
    "name": "string",
    "connector": "string",
    "id": "string",
    "connector_display_name": "string",
    "created_timestamp": "string",
    "sub_id": "string",
    "tenant_id": "string",
    "service_id": "string",
    "config": {
      "property1": "string",
      "property2": "string"
    },
    "topic_ids": [
      "string"
    ],
    "topic_map": {
      "property1": [
        "string"
      ],
      "property2": [
        "string"
      ]
    },
    "topics": [
      "string"
    ],
    "tasks": [
      0
    ],
    "connector_status": "string",
    "desired_state": "string",
    "reconcile_error": "string",
    "connection_warning": "string",
    "kc_cluster_id": "string",
    "task_statuses": {
      "property1": {
        "property1": "string",
        "property2": "string"
      },
      "property2": {
        "property1": "string",
        "property2": "string"
      }
    },
    "tags": [
      "string"
    ]
  },
  "destination": {
    "name": "string",
    "connector": "string",
    "id": "string",
    "connector_display_name": "string",
    "created_timestamp": "string",
    "sub_id": "string",
    "tenant_id": "string",
    "config": {
      "property1": "string",
      "property2": "string"
    },
    "topic_ids": [
      "string"
    ],
    "topic_map": {
      "property1": [
        "string"
      ],
      "property2": [
        "string"
      ]
    },
    "topics": [
      "string"
    ],
    "tasks": [
      0
    ],
    "connector_status": "string",
    "desired_state": "string",
    "reconcile_error": "string",
    "schema_evolution_warning": "string",
    "kc_cluster_id": "string",
    "task_statuses": {
      "property1": {
        "property1": "string",
        "property2": "string"
      },
      "property2": {
        "property1": "string",
        "property2": "string"
      }
    },
    "tags": [
      "string"
    ]
  },
  "transforms": [
    null
  ],
  "tags": [],
  "periodic_audit": {
    "topics": [
      "string"
    ],
    "timestamp_column": "string",
    "interval_minutes": 0,
    "fix_deletes_only": false
  },
  "topic_auto_discovery_transforms": []
}

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string",
      "input": null,
      "ctx": {}
    }
  ]
}