Skip to main content

​
Why Heartbeats Matter

Streamkap connectors use offsets (bookmarks) to track their position in the database’s change log (WAL, binlog, redo log, or change stream). When no data changes occur for an extended period, these offsets can become stale. If the database rotates or purges its change log files before the connector advances past them, you risk:
  • Data loss — the connector can no longer read the changes it missed.
  • Log file accumulation — some databases retain logs indefinitely until the consumer advances, consuming disk space.
  • Connector restart failures — a stale offset may point to a log position that no longer exists.
Heartbeats solve this by generating periodic change events that keep the connector’s position advancing, even when no real data changes occur on your monitored tables.

​
How Heartbeats Work

Connectors use “offsets”—like bookmarks—to track their position in the database’s log or change stream. When no changes occur for long periods, these offsets may become outdated, and the Connector might lose its place or stop capturing changes. Heartbeats ensure the Connector stays active and continues capturing changes. There are two layers of heartbeat protection:

​
Layer 1: Connector heartbeats (enabled by default)

The Connector periodically emits heartbeat messages to an internal topic, even when no actual data changes are detected. This keeps offsets fresh and prevents staleness. No configuration is necessary for this layer; it is automatically enabled. We recommend keeping this layer enabled for all deployments.
Why we recommend configuring Layer 2While Layer 2 is crucial for low-traffic or intermittent databases, we recommend configuring it for all deployments. It provides additional resilience and helps prevent issues during periods of inactivity.
You can configure regular updates to a dedicated heartbeat table in the source database. This simulates activity, ensuring change events are generated consistently, maintaining log progress and providing additional resilience. How this layer is configured depends on the connection type (if supported by the Source):
  • Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
  • Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.
This layer requires you to set up a heartbeat table—and for read-only connections, a scheduled job (e.g., pg_cron for PostgreSQL, event_scheduler for MySQL)—on your source database. The default heartbeat interval is 1 minute, which works well for most deployments. This interval balances the need for timely offset advancement against the minimal overhead of a single row update.
ScenarioRecommended interval
Standard deployments1 minute
Very low-traffic databases (hours between changes)1 minute
High-traffic databases (changes every few seconds)1—5 minutes (heartbeats are less critical but still recommended)

​
Setup Instructions

Heartbeat configuration varies by database type. Follow the setup guide for your specific source connector: MySQL / MariaDB: PostgreSQL: SQL Server: Oracle: MongoDB:

​
Troubleshooting

Symptoms: The connector logs errors about a missing heartbeat table, or heartbeat events are not being generated.Solutions:
  • Verify the heartbeat table exists in the correct schema. Run a SELECT query against it to confirm.
  • For PostgreSQL, check that the streamkap schema exists: SELECT schema_name FROM information_schema.schemata WHERE schema_name = 'streamkap';
  • For MySQL/MariaDB, check that the streamkap database exists: SHOW DATABASES LIKE 'streamkap';
  • For Oracle, remember the table is in the STREAMKAP_USER schema, not a separate streamkap schema.
  • Ensure the initial row has been inserted (INSERT INTO ... VALUES ('test_heartbeat')).
Symptoms: The connector cannot read from or write to the heartbeat table. Errors reference permission denied or access issues.Solutions:
  • Verify the Streamkap user has SELECT, UPDATE, INSERT, and DELETE permissions on the heartbeat table.
  • For PostgreSQL, also verify USAGE permission on the streamkap schema.
  • For SQL Server, verify the streamkap_role has the required grants.
  • For Oracle, verify permissions are granted to both STREAMKAP_USER and C##STREAMKAP_USER.
  • Re-run the GRANT statements from the setup instructions for your database type.
Symptoms: The heartbeat table exists and has a row, but the last_update timestamp is not being updated.Solutions:
  • PostgreSQL: Verify the pg_cron job is running: SELECT * FROM cron.job; and check SELECT * FROM cron.job_run_details ORDER BY start_time DESC LIMIT 5; for errors.
  • MySQL/MariaDB: Verify the event scheduler is enabled: SHOW VARIABLES WHERE VARIABLE_NAME = 'event_scheduler'; should return ON. Check event status: SHOW EVENTS IN streamkap;
  • MongoDB: Verify your scheduled trigger or cron job is executing. For Atlas, check the trigger logs in the App Services console.
  • Confirm the scheduled job is updating the correct row (typically WHERE id = 1).
Symptoms: The heartbeat table is being updated, but the connector does not detect the changes.Solutions:
  • PostgreSQL: If you created a publication for specific tables, add the heartbeat table: ALTER PUBLICATION streamkap_pub ADD TABLE streamkap.streamkap_heartbeat;
  • SQL Server: Verify CDC is enabled on the heartbeat table using SELECT is_tracked_by_cdc FROM sys.tables WHERE name = 'streamkap_heartbeat'; — the result should be 1.
  • MongoDB: Ensure the heartbeat collection is included in the connector’s namespace configuration. If you configured specific databases/collections, add streamkap.streamkap_heartbeat.
Symptoms: Even with heartbeats configured, the connector reports stale offsets or cannot resume from its last position.Solutions:
  • Verify that both Layer 1 (connector heartbeats) and Layer 2 (database heartbeats) are active.
  • Check your database’s log retention settings:
  • Ensure the heartbeat interval (1 minute) is shorter than your log rotation period.
  • Contact Streamkap support if the issue persists.

​
Quick Reference

DatabaseHeartbeat tableSchema/LocationRead-only supportScheduler for read-only
PostgreSQLstreamkap_heartbeatstreamkap schemaYespg_cron extension
MySQLstreamkap_heartbeatstreamkap databaseYesMySQL Event Scheduler
OracleSTREAMKAP_HEARTBEATSTREAMKAP_USER schemaNoN/A
SQL Serverstreamkap_heartbeatstreamkap schemaNoN/A
MariaDBstreamkap_heartbeatstreamkap databaseYesMariaDB Event Scheduler
MongoDBstreamkap_heartbeatstreamkap database (collection)N/A (always external)Atlas Triggers / cron / K8s CronJob