--- title: "Streamkap documentation" description: "Serverless, real-time data integration: stream change data sub-second from operational databases to warehouses, lakes, and search, and transform it in flight." sidebar: # Named rather than hidden. Hiding it read as the tidy fix for two competing intros, but # `sidebar.hidden` is not only a nav flag: the renderer treats a hidden page as not indexable, # dropping it from search, and the sitemap excludes it with no way to opt back in. That made the # page the logo points at the one page nobody could search for and no crawler was told about. # # The original complaint was two entries that both looked like the intro. That is settled by # naming: this is "Home", and the page inside Getting Started is "Overview". label: "Home" --- Streamkap is a serverless, real-time data integration platform. You connect a source database, pick a destination, and it streams changes sub-second — no batch windows, no orchestration to run. ## Start here **[Getting started](/getting-started)** The shortest path to a working pipeline. **[Concepts](/concepts)** Change data capture, and how the parts fit together. **[Pipelines](/pipelines)** Build and operate the flow from source to destination. ## Sources Where change data is captured from. Each page covers the connector and its per-platform setup — managed services and self-hosted alike. ### Relational **[PostgreSQL](/postgresql)** RDS, Aurora, Cloud SQL, Azure, Neon, or self-hosted. **[MySQL](/mysql)** Generic, Amazon RDS, Aurora, Google Cloud, and Azure. **[MariaDB](/mariadb)** Generic deployments and Amazon RDS. **[SQL Server](/sql-server)** Amazon RDS, Google Cloud SQL, Azure SQL, or generic. **[Oracle](/oracle)** LogMiner, XStream, or OpenLogReplicator. **[Db2](/ibm-db2)** The generic connector, with more platforms on request. **[Informix](/informix)** Real-time capture via the generic connector. **[AlloyDB](/alloydb)** Logical replication, the replication role, and heartbeats. **[Supabase](/supabase)** Logical replication, plus the required IPv4 add-on. **[Vitess](/vitess)** The generic setup, with more platforms on request. **[PlanetScale](/planetscale)** Vitess-based MySQL, captured through the VTGate VStream service. ### Document, key-value, and search **[MongoDB](/mongodb)** Setups for Atlas and for self-managed deployments. **[DocumentDB](/documentdb)** Amazon change streams, plus a signal collection. **[DynamoDB](/dynamodb-source)** Streams with new and old images; S3 exports for history. **[Redis](/redis-source)** Generic setup today; contact support for other platforms. **[Elasticsearch](/elasticsearch)** Cluster connection details, authentication, and the indices to capture. **[OpenSearch](/opensearch)** Connection details, index names, and a datetime tracking field. ### Streams, files, and events **[Kafka Direct](/kafka-direct)** Consume from an external cluster: Confluent, MSK, or Redpanda. **[Writing to Kafka](/write-to-kafka)** Push messages into Streamkap topics with Python or kcat. **[S3](/s3-source)** Poll a bucket for new JSON, CSV, or Avro files. **[Webhook](/webhook)** Accept HTTP POST, PUT, and DELETE at a generated endpoint. :::tip Upgrading a source database? Read the [database upgrade guide](/database-upgrade-guide) first — an in-place major version upgrade can invalidate a connector's replication state. ::: ## Destinations Where the stream lands. ### Warehouses and lakehouses **[Snowflake](/snowflake)** Upsert or append, schema evolution, and managed Iceberg tables. **[BigQuery](/google-bigquery)** Writes through the Storage Write API with a service account. **[Databricks](/databricks-delta-lake)** JDBC to all-purpose compute or a SQL warehouse. **[Redshift](/redshift)** At-least-once delivery with upsert writes, over a dedicated role. **[ClickHouse](/clickhouse)** Cloud destination: upsert or append, type mappings, IP access list. **[Apache Iceberg](/iceberg)** The AWS Glue catalog, or a generic catalog. **[Starburst](/starburst)** Lands in an S3 bucket as Avro, JSON, or Parquet. **[Motherduck](/motherduck)** The native destination, or the S3-based option. ### Object storage **[Amazon S3](/s3)** JSON Lines, JSON Array, or Parquet, with file rotation. **[Azure Blob Storage](/azure-blob-storage)** SAS tokens or account access keys, and file formats. **[Cloudflare R2](/r2)** File formats, compression, and filename templates. ### Operational databases **[PostgreSQL](/stream-into-postgresql)** Upsert writes on port 5432, with schema evolution. **[MySQL](/stream-into-mysql)** Upsert writes on port 3306, with schema evolution modes. **[SQL Server](/stream-into-sqlserver)** Port 1433, with default schema setup and evolution. **[Oracle](/stream-into-oracle)** Port 1521, with CDB and non-CDB user setup. **[Db2](/stream-into-db2)** Upsert writes on port 50000, with schema evolution. **[CockroachDB](/cockroachdb)** IP allowlisting, user creation, and schema evolution settings. **[Neon](/neon-destination)** PostgreSQL with upsert writes, delete mode, and schema evolution. **[Redis](/redis-destination)** Redis Cloud, or a generic instance. ### Streams, services, and vector stores **[Reading from Kafka](/reading-from-kafka)** Pull from Streamkap topics with Python or kcat. **[Kafka to Kafka](/kafka-to-kafka)** Replicate topics to another cluster as Avro or JSON. **[HTTP sink](/http-sink)** POST records to any endpoint, with optional OAuth2. **[Weaviate](/weaviate)** Vector collections, with built-in or bring-your-own embeddings. **[Pinecone](/pinecone)** Vector indexes; embeddings must be pre-computed. Managing connectors in the app itself — creating one, editing its authentication, and the actions available on it — is covered by [sources](/sources) and [destinations](/destinations). ## Run it **[Pipelines](/pipelines)** Build and operate the flow from source to destination. **[Transforms](/transforms)** Filter, enrich, join, and reshape records in flight. **[Streaming agents](/streaming-agents)** Put a model on the stream, with its own knowledge bases. **[Snapshots](/snapshots)** Backfill history, then switch to streaming. **[Observability](/observability)** Latency, throughput, and pipeline health. **[Alerts](/alerts)** Get told when a pipeline needs you. ## When something breaks **[Error reference](/error-reference)** Every error the platform reports, and what to do about it. **[Logs](/logs)** Find what a connector was doing when it stopped. **[Recovery](/pipeline-recovery)** Resume a pipeline, and repair a dead-letter queue. ## Build against it **[API reference](/api-reference)** The REST API, generated from the specification. **[Developer tools](/api)** API tokens, the CLI, the MCP server, and Terraform. ## How it is deployed and secured **[Deployment options](/deployment-options)** Streamkap Cloud, or bring your own. **[Connection options](/connection-options)** Reach a private database over PrivateLink, SSH, or VPN. **[Security posture](/security-posture)** Encryption, SSO, audit logs, and sub-accounts.