Large Column Values (TOAST)

How Streamkap handles large PostgreSQL column values in CDC, and when to use each approach.

PostgreSQL stores column values larger than roughly 8 KB in a separate internal storage area called TOAST. When a row is updated, PostgreSQL only writes values that actually changed to its replication log. If a large column stays the same across an update, it’s omitted entirely — leaving a gap the connector needs to fill before sending the event downstream.

Three approaches

1. Connector-side re-fetch

When the connector detects a missing large column value, it queries the source database to retrieve the current value before forwarding the event. This happens synchronously — the connector pauses, fetches, then continues to the next event.

This is the default behavior for PostgreSQL sources in Streamkap. You can configure which columns are included or excluded in your source’s advanced settings.

Best for:

• Tables updated fewer than ~200 times per second
• Column values under ~100 KB
• Setups where no changes outside Streamkap are desired

2. Full row replication

Setting REPLICA IDENTITY FULL on a table tells PostgreSQL to always write the complete previous row — including all large columns — to the replication log on every update. The connector then has everything it needs without any follow-up queries.

ALTER TABLE your_table REPLICA IDENTITY FULL;

Best for:

• Tables where historical accuracy is important — the captured value always reflects the row state at the time of the change
• Tables where the large column changes frequently anyway (you’re already paying the cost under the default setting)
• Cases where a simple, one-time database-side change is preferred

3. Downstream resolution (Flink)

Events flow from the connector immediately, with a placeholder where the large value is missing. A Flink pipeline resolves those placeholders asynchronously, without blocking the connector.

Best for:

• Tables with high update rates (hundreds per second or more)
• Column values over 100 KB
• Production pipelines where connector lag from synchronous re-fetching is unacceptable

Choosing an approach

When uncertain, connector-side re-fetch requires no changes and is already active. Move to full row replication or a downstream solution if you hit throughput, lag, or accuracy issues.