Snapshots & Backfilling

Streamkap can perform snapshots to backfill your data.

• Please ensure you've followed your connector setup guide to enable this process.
• All snapshots/backfills are free of charge.

Default Behavior

Upon Connector Creation

• The default behavior is to carry out an initial snapshot across all tables when you create the connector
• After this completes, the connector will switch to reading the log data and streaming mode
• If you restart or make configuration changes during the above snapshot process, the snapshot will stop and will not resume. You should trigger an adhoc snapshot.
• Additional tables added after the connector is created will not be snapshotted by default. You should trigger an adhoc snapshot.

Post Connector Creation

After the connector is created, you can perform adhoc incremental snapshots, which will:

• Sync historical data from today's date and working its way backwards using a process called watermarking.
• It will do this in 1024 row limit increments unless you increase this value in the connector settings.

How to Trigger an Adhoc Snapshot

It's possible to trigger a snapshot via the connector or at table level.

Connector

This will trigger a snapshot across all tables/topics.

Table/Topic

This will carry out a snapshot only for the selected table/topic.

Upon triggering a snapshot, the status will update

It is possible to cancel the snapshot while it's in pending status

Performance

Each connector has a setting called Snapshot Chunk Size.

This represents the number of rows that are fetched upon request which will default at 1024 rows. Increasing this value will linearly increase the speed of the snapshots.