Streamkap documentation
Serverless, real-time data integration: stream change data sub-second from operational databases to warehouses, lakes, and search, and transform it in flight.
Streamkap is a serverless, real-time data integration platform. You connect a source database, pick a destination, and it streams changes sub-second — no batch windows, no orchestration to run.
Start here
Getting started
The shortest path to a working pipeline.
Concepts
Change data capture, and how the parts fit together.
Pipelines
Build and operate the flow from source to destination.
Sources
Where change data is captured from. Each page covers the connector and its per-platform setup — managed services and self-hosted alike.
Relational
PostgreSQL
RDS, Aurora, Cloud SQL, Azure, Neon, or self-hosted.
MySQL
Generic, Amazon RDS, Aurora, Google Cloud, and Azure.
MariaDB
Generic deployments and Amazon RDS.
SQL Server
Amazon RDS, Google Cloud SQL, Azure SQL, or generic.
Oracle
LogMiner, XStream, or OpenLogReplicator.
Db2
The generic connector, with more platforms on request.
Informix
Real-time capture via the generic connector.
AlloyDB
Logical replication, the replication role, and heartbeats.
Supabase
Logical replication, plus the required IPv4 add-on.
Vitess
The generic setup, with more platforms on request.
PlanetScale
Vitess-based MySQL, captured through the VTGate VStream service.
Document, key-value, and search
MongoDB
Setups for Atlas and for self-managed deployments.
DocumentDB
Amazon change streams, plus a signal collection.
DynamoDB
Streams with new and old images; S3 exports for history.
Redis
Generic setup today; contact support for other platforms.
Elasticsearch
Cluster connection details, authentication, and the indices to capture.
OpenSearch
Connection details, index names, and a datetime tracking field.
Streams, files, and events
Kafka Direct
Consume from an external cluster: Confluent, MSK, or Redpanda.
Writing to Kafka
Push messages into Streamkap topics with Python or kcat.
S3
Poll a bucket for new JSON, CSV, or Avro files.
Webhook
Accept HTTP POST, PUT, and DELETE at a generated endpoint.
Destinations
Where the stream lands.
Warehouses and lakehouses
Snowflake
Upsert or append, schema evolution, and managed Iceberg tables.
BigQuery
Writes through the Storage Write API with a service account.
Databricks
JDBC to all-purpose compute or a SQL warehouse.
Redshift
At-least-once delivery with upsert writes, over a dedicated role.
ClickHouse
Cloud destination: upsert or append, type mappings, IP access list.
Apache Iceberg
The AWS Glue catalog, or a generic catalog.
Starburst
Lands in an S3 bucket as Avro, JSON, or Parquet.
Motherduck
The native destination, or the S3-based option.
Object storage
Amazon S3
JSON Lines, JSON Array, or Parquet, with file rotation.
Azure Blob Storage
SAS tokens or account access keys, and file formats.
Cloudflare R2
File formats, compression, and filename templates.
Operational databases
PostgreSQL
Upsert writes on port 5432, with schema evolution.
MySQL
Upsert writes on port 3306, with schema evolution modes.
SQL Server
Port 1433, with default schema setup and evolution.
Oracle
Port 1521, with CDB and non-CDB user setup.
Db2
Upsert writes on port 50000, with schema evolution.
CockroachDB
IP allowlisting, user creation, and schema evolution settings.
Neon
PostgreSQL with upsert writes, delete mode, and schema evolution.
Redis
Redis Cloud, or a generic instance.
Streams, services, and vector stores
Reading from Kafka
Pull from Streamkap topics with Python or kcat.
Kafka to Kafka
Replicate topics to another cluster as Avro or JSON.
HTTP sink
POST records to any endpoint, with optional OAuth2.
Weaviate
Vector collections, with built-in or bring-your-own embeddings.
Pinecone
Vector indexes; embeddings must be pre-computed.
Managing connectors in the app itself — creating one, editing its authentication, and the actions available on it — is covered by sources and destinations.
Run it
Pipelines
Build and operate the flow from source to destination.
Transforms
Filter, enrich, join, and reshape records in flight.
Streaming agents
Put a model on the stream, with its own knowledge bases.
Snapshots
Backfill history, then switch to streaming.
Observability
Latency, throughput, and pipeline health.
Alerts
Get told when a pipeline needs you.
When something breaks
Error reference
Every error the platform reports, and what to do about it.
Logs
Find what a connector was doing when it stopped.
Recovery
Resume a pipeline, and repair a dead-letter queue.
Build against it
API reference
The REST API, generated from the specification.
Developer tools
API tokens, the CLI, the MCP server, and Terraform.