PostgreSQL Change Data Capture Setup with Streamkap
wal_level = logical
wal_sender_timeout = 0
. A non zero amount may cause disconnects in low volume databases. You should in which case enable PostgreSQL Heartbeats or set a suitable value here.statement_timeout = 0
OR greater than 5 minutespg_hba.conf
file with an entry to authenticate Streamkap’s connection to the WAL.postgresql.conf
file and ensure that the max_wal_senders
parameter is at least twice the total number of logical replication slots.streamkap_user
. The Connector will use this collection for managing snapshots. Below is an example script that does that.
streamkap_signal
. It will not be recognised if given another name.streamkap_signal
table.FOR ALL TABLES
publications to include/exclude tables.If you set up a FOR ALL TABLES
publication and later decide to change that, you have to drop the publication and create another to include specific tables e.g. CREATE PUBLICATION ... TABLE table1, table2, table3, ...
.However, any change events that occur before the new publication’s created will not be included in it, so a snapshot’s required to ensure they are not missed by your Streamkap pipelines.You should also stop the Source before changing the publication.REPLICATION
privileges as per Create Database User.5432
.
streamkap_user
.
streamkap_user
. See Enable Snapshots for setup instructions.streamkap_pgoutput_slot
.streamkap_pub
.DatabaseName_TopicName
blob
, binary
, varbinary
should be interpreted. Your destination for this data can impact which option you choose. Default is bytes
..csv
file without a header.Managing PostgreSQL upgrades
Replica identity and deleted records (PostgreSQL 13 and newer)
REPLICA IDENTITY
table setting controls what data is logged for row updates and deletes.REPLICA IDENTITY
to FULL
for all capture tables.This ensures complete data retention
publish_via_partition_root
on the publication.ALTER PUBLICATION streamkap_pub SET (publish_via_partition_root = true);