Snapshots & Backfilling
Streamkap Sources can perform snapshots to backfill your data.
- Please ensure you've followed the relevant connector setup guide to enable this process
Default Behavior
By default, an ad-hoc snapshot will sync historical data from the first record in the source table using a process called 'watermarking'. It will do this in small increments to mitigate impact on the source database.
Outlined below are the scenarios where snapshots are managed by you or automatically by our platform:
| When | Behavior |
|---|---|
| At connector creation | The connector starts in streaming mode, reading any change data seen from this point onwards. No snapshots are triggered automatically. |
| After connector creation | You can trigger ad-hoc snapshots for any or all of the tables the connector is configured to capture. No snapshots are triggered automatically. |
| Pipeline creation and edit | Snapshots are triggered automatically after choosing which topics the pipeline will stream to your destination. This behavior can be toggled using the Snapshot topics toggle at the top of the page. |
How to Trigger an Ad-hoc Snapshot
You can trigger an ad-hoc snapshot at the Source level or per Table from the Connector's page.
Source
This will trigger a snapshot for all tables/topics captured by the Source:
Table/Topic
This will trigger a snapshot for the selected table/topic only:
Upon triggering a snapshot, the status will update:
It is possible to cancel the snapshot while it's in pending status:
Updated 8 days ago