Snapshots & Backfilling

Streamkap can perform snapshots to backfill your data.

Please ensure you've followed your connector setup guide to enable this process.

Default Behavior

• When adding a source connector, the default behaviour is not to snapshot but only sync data from the transaction log.
• When you create a pipeline connecting the source to a destination there is a toggle defaulted to ON to take a snapshot of the tables being added in the pipeline

Post Connector & Pipeline Creation

• When you add topics/tables to the pipeline, the default behaviour is to take a snapshot of newly added tables.
• You can trigger snaphot on an ad-hoc basis by drilling into a source and clicking snapshot at connector level or you can also trigger at topic level.

How to Trigger an Adhoc Snapshot

It's possible to trigger a snapshot via the connector or at topic level.

Connector

This will trigger a snapshot across all tables/topics.

Table/Topic

This will carry out a snapshot only for the selected table/topic.

Upon triggering a snapshot, the status will update

It is possible to cancel the snapshot while it's in pending status

Performance

Each connector has a setting called Snapshot Chunk Size.

This represents the number of rows that are fetched upon request which will default at 1024 rows. Increasing this value will linearly increase the speed of the snapshots.