Metadata

Streamkap adds metadata columns to destination tables for delete handling, deduplication in append mode, and measuring pipeline latency.

Streamkap Connectors (where applicable) insert the following metadata columns into the data and destination tables:

Column	Description
`_STREAMKAP_SOURCE_TS_MS`	Timestamp in milliseconds (UTC) for when the event occurred in the source database
`_STREAMKAP_TS_MS`	Timestamp in milliseconds (UTC) for when Streamkap received the event
`__DELETED`	Indicates whether this event/row has been deleted in the source database (`true`/`false`)
`_STREAMKAP_OFFSET`	Offset value in relation to the events processed, useful for debugging
`_STREAMKAP_OP`	Operation type: `c` (create/insert), `u` (update), `d` (delete), `r` (read/snapshot), `t` (truncate)
`_STREAMKAP_PARTITION`	Kafka partition the record was consumed from

Using Metadata Columns

Metadata columns enable several important patterns:

Delete handling: Use the __DELETED column to identify rows deleted at the source. See Delete Handling for configuration options and query patterns.
Deduplication: When using insert/append mode, use _STREAMKAP_TS_MS and _STREAMKAP_OFFSET to build deduplicated final-state tables. See Creating Final State Tables for examples with Snowflake Dynamic Tables, Tasks, and dbt.
Debugging: Use _STREAMKAP_SOURCE_TS_MS to determine when events occurred at the source, and _STREAMKAP_TS_MS to determine when Streamkap processed them. The difference indicates pipeline latency.