---
title: "Large Column Values (TOAST)"
description: "How Streamkap handles large PostgreSQL column values in CDC, and when to use each approach."
---

PostgreSQL stores column values larger than roughly 8 KB in a separate internal storage area called TOAST. When a row is updated, PostgreSQL only writes values that actually changed to its replication log. If a large column stays the same across an update, it's omitted entirely — leaving a gap the connector needs to fill before sending the event downstream.

:::info
This affects tables where a column contains large text, JSON, binary data, or other oversized values, **and** rows are frequently updated without changing that column.
:::

## Three approaches

### 1. Connector-side re-fetch

When the connector detects a missing large column value, it queries the source database to retrieve the current value before forwarding the event. This happens synchronously — the connector pauses, fetches, then continues to the next event.

This is the default behavior for PostgreSQL sources in Streamkap. You can configure which columns are included or excluded in your source's advanced settings.

**Best for:**
- Tables updated fewer than ~200 times per second
- Column values under ~100 KB
- Setups where no changes outside Streamkap are desired

:::warning
**Performance and accuracy trade-offs to be aware of:**

- Each re-fetch adds 2–20 ms of latency per event depending on network distance between the connector and your database.
- Above ~200 updates/second, re-fetches can't keep pace — connector lag grows and your database must retain more replication log data on disk.
- When the connector falls behind, the fetched value reflects the **current** row state, not the state at the time of the update. Under sustained lag, this can cause older events to carry newer column values.
- A slow or unresponsive database blocks the connector for up to 10 minutes by default.
:::

---

### 2. Full row replication

Setting `REPLICA IDENTITY FULL` on a table tells PostgreSQL to always write the complete previous row — including all large columns — to the replication log on every update. The connector then has everything it needs without any follow-up queries.

```sql
ALTER TABLE your_table REPLICA IDENTITY FULL;
```

**Best for:**
- Tables where historical accuracy is important — the captured value always reflects the row state at the time of the change
- Tables where the large column changes frequently anyway (you're already paying the cost under the default setting)
- Cases where a simple, one-time database-side change is preferred

:::warning
Every update — even those that don't touch the large column — writes the full old row to the replication log. For example, a 100 KB column on a table receiving 100 updates/second adds roughly 10 MB/s of additional write I/O and proportional storage to your source database.

**Size your PostgreSQL instance to accommodate this additional write throughput and storage.** The impact scales with both update rate and column size, so high-volume tables with large columns may require a meaningfully larger instance.

Replication log retention also grows faster under connector lag, further increasing disk pressure.
:::

---

### 3. Downstream resolution (Flink)

Events flow from the connector immediately, with a placeholder where the large value is missing. A Flink pipeline resolves those placeholders asynchronously, without blocking the connector.

**Best for:**
- Tables with high update rates (hundreds per second or more)
- Column values over 100 KB
- Production pipelines where connector lag from synchronous re-fetching is unacceptable

:::note
Contact [Streamkap support](mailto:support@streamkap.com) to discuss setting up a Flink pipeline for your workload.
:::

---

## Choosing an approach

| Factor | Connector re-fetch | Full row replication | Flink pipeline |
|---|---|---|---|
| Update rate | Under ~200/s | Any (size DB accordingly) | Any |
| Column size | Under ~100 KB | Any | Any |
| Changes required | None — on by default | One DDL statement | Flink infrastructure |
| Source DB impact | Additional read load per event | Additional write load per update | Minimal |
| Value accuracy under lag | Current state (not historical) | Historical, at event time | Historical, at event time |
| Setup complexity | Low | Low | High |

When uncertain, connector-side re-fetch requires no changes and is already active. Move to full row replication or a downstream solution if you hit throughput, lag, or accuracy issues.

:::note
All three approaches apply only to **update** events. Inserts always carry the full row value; deletes carry the key only.
:::