Prerequisites

• PostgreSQL logical replication enabled
• Connection details
• Streamkap user and role

PostgreSQL Setup

Granting Database Access

• Visit Connection Options to ensure Streamkap can reach your database

Granting Privileges

It's recommended to create a separate user and role for Streamkap to access your PostgreSQL database. Below is an example script that does that.

-- Replace { ... } placeholders as required
CREATE USER streamkap_user PASSWORD '{password}';

-- Create a role for Streamkap
CREATE ROLE streamkap_role nologin PASSWORD '{password}';
GRANT streamkap_role TO streamkap_user;
GRANT rds_replication TO streamkap_role;

-- Grant Streamkap permissions on the database, schema and all tables to capture
-- Create on database is used to create the publication. Not needed if you create yourself
GRANT CONNECT ON DATABASE "{database}" TO streamkap_role; 
GRANT CREATE, USAGE ON SCHEMA "{schema}" TO streamkap_role;
GRANT SELECT ON ALL TABLES IN SCHEMA "{schema}" TO streamkap_role;
ALTER DEFAULT PRIVILEGES IN SCHEMA "{schema}" GRANT SELECT ON TABLES TO streamkap_role;

-- Grant replication role to the user
ALTER USER streamkap_user WITH REPLICATION;

Enable Snapshots

You will need to create the table in the source database and give permissions to the streamkap_role. Streamkap will use this collection for managing snapshots.

❗️

Please create the signal table with the name streamkap_signal. It will not be recognised if given another name.

-- Create the schema
CREATE SCHEMA streamkap;

-- Switch to the newly created schema
SET search_path TO streamkap;

-- Create the table
CREATE TABLE streamkap_signal (
  id VARCHAR(255) PRIMARY KEY, 
  type VARCHAR(32) NOT NULL, 
  data VARCHAR(2000) NULL
);

-- Grant necessary privileges on the table to the role
GRANT SELECT, UPDATE, INSERT ON TABLE streamkap_signal TO streamkap_role;

Streamkap Setup

• Go to Sources and click Create New
• Input
  • Name for your Connector
  • Hostname

  ⚠️

  PgBouncer and pooled connections

  Neon uses PgBouncer to support connection pooling via pooler hostnames like this ep-cool-darkness-123456-pooler.us-east-2.aws.neon.tech (notice the -pooler option). However, PgBouncer has very limited support for PostgreSQL startup options, and Streamkap depends on one PostgreSQL option replication it does not support. The Connector will fail with a PgBouncer "unsupported startup parameter " error in this scenario.

  Because of that, Neon's connection pooling cannot be used, so please remove (if present) the -pooler option from the hostname. This means connections from Streamkap will be unpooled, and Neon has limits on unpooled connections you should be aware of.

  • Port (Default 5432)
  • Username (Username you chose earlier, our scripts use streamkap_user)
  • Password
  • Heartbeat - Required for low volume connectors. See PostgreSQL Heartbeats
  • Signal Table Schema - Streamkap will use a collection in this schema to manage snapshots e.g. streamkap. See Enable Snapshots for more information
  • Database Name
  • Replication Slot Name (Default streamkap_pgoutput_slot)
  • Publication Name (Default streamkap_pub)
  • Connect via SSH Tunnel. See SSH Tunnel
  • Advanced Parameters
    • Represent Binary Data As (Default bytes)
  • Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.
  • Click Save
    The connector will take approximately 1 minute to start processing data.