Oracle (Generic)

Configure Oracle 12c+ as a Streamkap CDC source: enable LogMiner and archive logs, choose a log mining strategy, capture LOB columns, create the connector user, and grant required privileges and quotas.

Prerequisites

Oracle 12c or above, Standard or Enterprise Edition
Hostname and Port
(single-tenant architecture): Database name
(multi-tenant architecture): Container database name (CDB) and pluggable database name (PDB)
The Oracle database master user credentials or equivalent

Oracle Setup

For the Connector to ingest changes from your database it is dependent on Oracle’s redo logs and archive logs. It is important that these redo logs are large enough and, the archive logs are retained for long enough to ensure all changes are captured.

1. Grant Database Access

Configure one of the Connection Options to ensure Streamkap can reach your database.

2. Enable Archive Logs

When redo logs fill up, Oracle archives groups of them into archive logs. Archive logs should be retained for at least 24 hours. However, we recommend retaining them for longer. Too short a retention period and changes may not be captured and processed.

Retention periods and database storage

Archive logs are retained on your database instance using up its storage capacity. It is important to make sure it has enough space, otherwise, performance issues and outages can occur.

In general, the more tables (and columns) there are, the more capacity is required. For the Connector, additional capacity is essential because it depends on Oracle’s supplemental logging.

To estimate what storage capacity you might need and assuming your Oracle database has supplemental logging enabled, you can look at the last 1 hour of log storage usage and multiply that by the retention period (at least 24 hours) you choose. Here’s an example script for that:

SELECT
 SUM(BLOCKS * BLOCK_SIZE) bytes, -- usage
 SUM(BLOCKS * BLOCK_SIZE) * 72 estimated_bytes -- assuming 72 hours archivelog retention
  FROM V$ARCHIVED_LOG
 WHERE 
  FIRST_TIME >= SYSDATE-(1/24) -- last hour
  AND 
  DEST_ID=1;

You should set the DB_RECOVERY_FILE_DEST_SIZE parameter to a value that is appropriate for your available disk space.

-- Replace the {...} placeholders as required
alter system set db_recovery_file_dest_size = {recovery_file_size};
alter system set db_recovery_file_dest = '{recovery_file_location}' scope=spfile;

Then, configure Oracle RMAN to retain backups and archive logs for at least 24 hours. We recommend retaining data for longer.

RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 3 DAYS;

To enable ARCHIVELOG mode, run this script:

SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;

To confirm if ARCHIVELOG mode has been enabled, run this query:

archive log list;

If the Database log mode is Archive Mode then it is enabled.

3. Enable LogMiner

For the Connector to query the redo and archive logs, it is dependent on the Oracle LogMiner utility. To enable that, supplemental logging must be enabled.

-- Enable database supplement logging
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

After supplemental logging has been enabled at the database level, you then need to enable table level supplemental logging.

-- To enable all supplemental logging, run the following SQL statement for each table:
ALTER TABLE {schema}.{table} ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS

To confirm if supplemental logging has been enabled, run this query:

SELECT NAME, SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE;

If the SUPPLEMENTAL_LOG_DATA_MIN is YES it is enabled.

Resize Redo Logs

An Oracle instance typically starts with three, online redo log files, 50MB each. That is too small and too few; more log files are necessary, and the logs should be resized to at least 1024MB or more, especially for production databases.

Before making any changes, run this query to check the current log file sizes:

SELECT GROUP#, BYTES/1024/1024 SIZE_MB, STATUS FROM V$LOG ORDER BY 1;

GROUP#	`SIZE_MB`	STATUS
1	50	INACTIVE
2	50	INACTIVE
3	50	CURRENT

Also, we need to determine how many log files per group there are and their location.

SELECT GROUP#, LOCATION FROM V$LOGFILE ORDER BY 1, 2;

GROUP#	LOCATION	STATUS
1	/opt/oracle/oradata/ORCLCDB/redo01.log	ACTIVE
2	/opt/oracle/oradata/ORCLCDB/redo02.log	INACTIVE
3	/opt/oracle/oradata/ORCLCDB/redo03.log	UNUSED

Now replace the old log files with new, larger log files. Only INACTIVE and UNUSED groups can be dropped and recreated.

-- Execute these statements as many times as required for existing INACTIVE and UNUSED log groups
-- Replace {group_number} placeholder and {log_location}, using a comma separated list of filenames if database uses log multiplexing
ALTER DATABASE CLEAR LOGFILE GROUP {group_number};
ALTER DATABASE DROP LOGFILE GROUP {group_number};
ALTER DATABASE ADD LOGFILE GROUP {group_number} ('{log_location}') size 1024M REUSE;

Now switch the ACTIVE log so we can drop it.

ALTER SYSTEM SWITCH LOGFILE;

We need to wait for the database to eventually switch the status of the ACTIVE group to INACTIVE. The switch could take several minutes, so be patient and recheck the size periodically. Once the status reaches INACTIVE, replace the last, old log file and confirm all logs are resized:

-- Execute these statements as many times as required for existing INACTIVE and UNUSED log groups
-- Replace {group_number} placeholder and {log_location}, using a comma separated list of filenames if database uses log multiplexing
ALTER DATABASE CLEAR LOGFILE GROUP {group_number};
ALTER DATABASE DROP LOGFILE GROUP {group_number};
ALTER DATABASE ADD LOGFILE GROUP {group_number} ('{log_location}') size 1024M REUSE;

SELECT GROUP#, BYTES/1024/1024 SIZE_MB, STATUS FROM V$LOG ORDER BY 1;

GROUP#	`SIZE_MB`	STATUS
1	1024	CURRENT
2	1024	UNUSED
3	1024	ACTIVE
…	…	…

Now add additional, new log files as necessary. We suggest starting with 4 additional log files.

-- Execute these statements as many times as required
-- Replace {group_number} placeholder and {log_location}, using a comma separated list of filenames if database uses log multiplexing
ALTER DATABASE ADD LOGFILE GROUP {group_number} ('{log_location}') size 1024M;

SELECT GROUP#, BYTES/1024/1024 SIZE_MB, STATUS FROM V$LOG ORDER BY 1;

GROUP#	`SIZE_MB`	STATUS
1	1024	CURRENT
2	1024	UNUSED
3	1024	ACTIVE
4	1024	INACTIVE
5	1024	INACTIVE
…	…	…

4. Create Database User

Depending on your database architecture, choose the correct script below to create a database user with privileges for the Connector.

-- Replace {...} placeholders as needed
ALTER SESSION SET CONTAINER=CDB$ROOT;
CREATE USER C##STREAMKAP_USER IDENTIFIED BY {password};
ALTER USER C##STREAMKAP_USER QUOTA UNLIMITED ON USERS;

ALTER SESSION SET CONTAINER={PDB};
CREATE TABLESPACE STREAMKAP_LOGMINER_TBS DATAFILE {filename} SIZE 25M AUTOEXTEND ON MAXSIZE UNLIMITED;
ALTER USER C##STREAMKAP_USER DEFAULT TABLESPACE STREAMKAP_LOGMINER_TBS;
ALTER USER C##STREAMKAP_USER QUOTA UNLIMITED ON STREAMKAP_LOGMINER_TBS;

-- Grant permissions
GRANT CREATE SESSION TO C##STREAMKAP_USER CONTAINER=ALL; 
GRANT SET CONTAINER TO C##STREAMKAP_USER CONTAINER=ALL; 

-- Allows the Connector to use LogMiner
GRANT LOGMINING TO C##STREAMKAP_USER CONTAINER=ALL;

-- Flashback queries used for performing initial snapshots of the data
GRANT FLASHBACK ANY TABLE TO C##STREAMKAP_USER CONTAINER=ALL; 
GRANT SELECT ANY TRANSACTION TO C##STREAMKAP_USER CONTAINER=ALL; 

-- Required for schema history when performing initial snapshots
GRANT SELECT_CATALOG_ROLE TO C##STREAMKAP_USER CONTAINER=ALL; 
GRANT EXECUTE_CATALOG_ROLE TO C##STREAMKAP_USER CONTAINER=ALL; 

-- Connector creates a table for explicitly managing the flushing of internal log buffers (LGWR)
GRANT CREATE TABLE TO C##STREAMKAP_USER CONTAINER=ALL;

GRANT CREATE SEQUENCE TO C##STREAMKAP_USER CONTAINER=ALL; 

-- Grant the Streamkap user permission to read each schema and table you wish to sync
ALTER SESSION SET CONTAINER={PDB};
GRANT SELECT ON {schema}.{table} TO C##STREAMKAP_USER CONTAINER=ALL;
-- Alternatively, you can grant access to all
-- GRANT SELECT ANY TABLE TO C##STREAMKAP_USER CONTAINER=ALL;

-- Grant the Streamkap user access to the DBA_EXTENTS, DBA_TABLESPACES, DBA_SEGMENTS, and TRANSACTION system views. 
GRANT SELECT ON DBA_EXTENTS TO C##STREAMKAP_USER;
GRANT SELECT ON DBA_TABLESPACES TO C##STREAMKAP_USER;
GRANT SELECT ON DBA_SEGMENTS TO C##STREAMKAP_USER;
GRANT SELECT ANY TRANSACTION TO C##STREAMKAP_USER;

-- The following grants must be run as SYS or a user with SYSDBA privilege (e.g. sqlplus / as sysdba)
-- SYSTEM alone does not have sufficient privileges to grant access to SYS-owned objects
ALTER SESSION SET CONTAINER=CDB$ROOT;
GRANT SELECT ON SYS.V_$DATABASE TO C##STREAMKAP_USER;
GRANT SELECT ON SYS.V_$PARAMETER TO C##STREAMKAP_USER;
GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO C##STREAMKAP_USER;
GRANT SELECT ON SYS.V_$ARCHIVE_DEST TO C##STREAMKAP_USER;
GRANT SELECT ON SYS.V_$LOGMNR_CONTENTS TO C##STREAMKAP_USER;
GRANT EXECUTE ON DBMS_LOGMNR TO C##STREAMKAP_USER;
GRANT EXECUTE ON DBMS_LOGMNR_D TO C##STREAMKAP_USER;
GRANT SELECT ANY TRANSACTION TO C##STREAMKAP_USER;
GRANT EXECUTE_CATALOG_ROLE TO C##STREAMKAP_USER;

-- Replace {...} placeholders as needed
CREATE USER STREAMKAP_USER IDENTIFIED BY {password};

CREATE TABLESPACE STREAMKAP_LOGMINER_TBS DATAFILE {filename} SIZE 25M AUTOEXTEND ON MAXSIZE UNLIMITED;
ALTER USER STREAMKAP_USER DEFAULT TABLESPACE STREAMKAP_LOGMINER_TBS;
ALTER USER STREAMKAP_USER QUOTA UNLIMITED ON STREAMKAP_LOGMINER_TBS;

-- Grant permissions
GRANT CREATE SESSION TO STREAMKAP_USER; 
GRANT SET CONTAINER TO STREAMKAP_USER; 

-- Allows the Connector to use LogMiner
GRANT LOGMINING TO STREAMKAP_USER; 

-- Flashback queries used for performing initial snapshots of the data
GRANT FLASHBACK ANY TABLE TO STREAMKAP_USER; 
GRANT SELECT ANY TRANSACTION TO STREAMKAP_USER; 

-- Required for schema history when performing initial snapshots
GRANT SELECT_CATALOG_ROLE TO STREAMKAP_USER; 
GRANT EXECUTE_CATALOG_ROLE TO STREAMKAP_USER; 

-- Connector creates a table for explicitly managing the flushing of internal log buffers (LGWR)
GRANT CREATE TABLE TO STREAMKAP_USER; 

GRANT CREATE SEQUENCE TO STREAMKAP_USER;

-- Grant the Streamkap user permission to read each schema and table you wish to sync
GRANT SELECT ON {schema}.{table} TO STREAMKAP_USER;
-- Alternatively, you can grant access to all
-- GRANT SELECT ANY TABLE TO STREAMKAP_USER;

-- The following grants must be run as SYS or a user with SYSDBA privilege (e.g. sqlplus / as sysdba)
-- SYSTEM alone does not have sufficient privileges to grant access to SYS-owned objects
GRANT SELECT ON SYS.V_$DATABASE TO STREAMKAP_USER;
GRANT SELECT ON SYS.V_$PARAMETER TO STREAMKAP_USER;
GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO STREAMKAP_USER;
GRANT SELECT ON SYS.V_$LOGMNR_CONTENTS TO STREAMKAP_USER;
GRANT EXECUTE ON DBMS_LOGMNR TO STREAMKAP_USER;
GRANT EXECUTE ON DBMS_LOGMNR_D TO STREAMKAP_USER;
GRANT SELECT ANY TRANSACTION TO STREAMKAP_USER;
GRANT EXECUTE_CATALOG_ROLE TO STREAMKAP_USER;

5. Enable Snapshots

To backfill your data, the Connector needs to be able to perform Snapshots (See Snapshots & Backfilling for more information). To enable this process, a table must be created for the Connector to use. Depending on your database architecture, choose the correct script below.

-- Replace {...} placeholders as needed

ALTER SESSION SET CONTAINER={PDB};

-- Create a local (PDB) user
CREATE USER STREAMKAP_USER IDENTIFIED BY {password};
GRANT CREATE SESSION TO STREAMKAP_USER; 
GRANT SET CONTAINER TO STREAMKAP_USER;

ALTER USER STREAMKAP_USER DEFAULT TABLESPACE STREAMKAP_LOGMINER_TBS;
ALTER USER STREAMKAP_USER QUOTA UNLIMITED ON STREAMKAP_LOGMINER_TBS;

-- Create the table
CREATE TABLE STREAMKAP_USER.STREAMKAP_SIGNAL (
  id VARCHAR2(255) PRIMARY KEY, 
  type VARCHAR2(32) NOT NULL, 
  data VARCHAR2(2000) NULL
);

-- Grant necessary privileges on the table to the common user
GRANT SELECT, UPDATE, INSERT, DELETE ON STREAMKAP_USER.STREAMKAP_SIGNAL TO C##STREAMKAP_USER;

-- Create the table
CREATE TABLE STREAMKAP_USER.STREAMKAP_SIGNAL (
  id VARCHAR2(255) PRIMARY KEY, 
  type VARCHAR2(32) NOT NULL, 
  data VARCHAR2(2000) NULL
);

-- Grant necessary privileges on the table to the user
GRANT SELECT, UPDATE, INSERT, DELETE ON STREAMKAP_USER.STREAMKAP_SIGNAL TO STREAMKAP_USER;

6. Heartbeats

Connectors use “offsets”—like bookmarks—to track their position in the database’s log or change stream. When no changes occur for long periods, these offsets may become outdated, and the Connector might lose its place or stop capturing changes.

Heartbeats ensure the Connector stays active and continues capturing changes.

There are two layers of heartbeat protection:

Layer 1: Connector heartbeats (enabled by default)

The Connector periodically emits heartbeat messages to an internal topic, even when no actual data changes are detected. This keeps offsets fresh and prevents staleness.

No configuration is necessary for this layer; it is automatically enabled. We recommend keeping this layer enabled for all deployments.

Layer 2: Source database heartbeats (recommended)

You can configure regular updates to a dedicated heartbeat table in the source database. This simulates activity, ensuring change events are generated consistently, maintaining log progress and providing additional resilience.

How this layer is configured depends on the connection type (if supported by the Source):

Read-write connections (when Read only is No during Streamkap Setup): The Connector updates the heartbeat table directly.
Read-only connections (when Read only is Yes during Streamkap Setup): A scheduled job on the primary database updates the heartbeat table, and these changes replicate to the read replica for the Connector to consume.

This layer requires you to set up a heartbeat table—and for read-only connections, a scheduled job (e.g., pg_cron for PostgreSQL, event_scheduler for MySQL)—on your source database.

For read-write connections (when Read only is No during Streamkap Setup), the Connector writes to the heartbeat table directly.

-- Replace {...} placeholders as needed
ALTER SESSION SET CONTAINER={PDB};

-- Create the heartbeat table with id, text, and last_update fields
CREATE TABLE STREAMKAP_USER.STREAMKAP_HEARTBEAT (
    id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY PRIMARY KEY,
    text VARCHAR2(4000),
    last_update TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Grant permission to the Streamkap user
GRANT SELECT, UPDATE, INSERT, DELETE ON STREAMKAP_USER.STREAMKAP_HEARTBEAT TO STREAMKAP_USER;

-- Grant necessary privileges on the table to the common user
GRANT SELECT, UPDATE, INSERT, DELETE ON STREAMKAP_USER.STREAMKAP_HEARTBEAT TO C##STREAMKAP_USER;

-- Insert the first row into the heartbeat table
INSERT INTO STREAMKAP_USER.STREAMKAP_HEARTBEAT (text) VALUES ('test_heartbeat');

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Source

Navigate to Add Connectors.
Choose Oracle.

2. Connection Settings

Name: Enter a name for your connector.
Hostname: Specify the database endpoint.
Port: Default is 1521.
Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your database. This is necessary if the Connector cannot connect directly to your database.
- See SSH Tunnel for setup instructions.
Username (case sensitive): Username to access the database. By default, Streamkap scripts use STREAMKAP_USER.
Password: Password to access the database.
Database: The database name (single-tenant architecture) or container database name (multi-tenant architecture).
Pluggable Database (optional, multi-tenant architecture only): The pluggable database name.
Heartbeats:
- Heartbeat Table Schema: Streamkap will use a table in this schema to manage heartbeats. Usually this is the same as the Signal Table. See Heartbeats for setup instructions.

3. Snapshot Settings

Signal Table: Full path to the signal table including schema and table name (e.g., STREAMKAP_USER.STREAMKAP_SIGNAL). This table is used for incremental snapshotting. See Enable Snapshots for setup instructions.

4. Advanced Parameters

Represent binary data as: Specifies how the data for binary columns should be interpreted. Your destination for this data can impact which option you choose. Default is bytes.
Capture Only Captured Databases DDL: Specifies whether the connector records schema structures from all logical databases in the database instance or only captured databases. Enabling this when you have many databases in your instance can improve performance and avoid timeouts. Default is false. See Schema History Optimization for details.
Capture Only Captured Tables DDL: Specifies whether the connector records schema structures from all logical tables in the captured schemas or databases, or only captured tables. Enabling this when you have many tables can improve performance and avoid timeouts. Default is false. See Schema History Optimization for details.
Log Mining Strategy (Advanced): Controls how LogMiner resolves the column metadata it needs to decode a change. Online Catalog (the default) reads it from the live data dictionary and is the fastest option, but it cannot decode changes to a table whose structure has since changed. Redo Log Catalog mines the dictionary out of the redo logs, so it handles schema changes better and is required for LOB capture, at the cost of more redo log volume and slower mining. Hybrid combines the two but cannot capture LOB columns. Default is Online Catalog.
Enable LOB Column Capture (Advanced): Captures CLOB, NCLOB and BLOB columns. Requires the Redo Log Catalog strategy — the setting is hidden when Hybrid is selected, and has no effect with Online Catalog. Default is false.
Re-select Post Processor (Advanced): Re-fetches columns that the redo log cannot carry — LOB columns in particular — from the source database at event time, merging them into a single event per row change. This is a new capability on Oracle and is off by default, so existing sources are unaffected until you opt in. Default is false.
Re-select Error Handling (Advanced): What happens when a re-select fails. Fail (the default) stops the connector; Warn logs a warning and continues with the value missing. Only shown when the Re-select Post Processor is enabled.

Click Next.

5. Schema and Table Capture

Add Schemas/Tables: Specify the schema(s) and table(s) for capture.
- You can bulk upload here. The format is a simple list of schemas and tables, with each entry on a new row. Save as a .csv file without a header.

Click Save.