MongoDB Atlas
MongoDB Atlas Change Data Capture Setup with Streamkap
Prerequisites
- MongoDB version ≥ 4.4
- MongoDB Replica Set or Sharded Cluster
- Connection details
- Streamkap user and role
Obtain Connection String
MongoDB Atlas
- Login to MongoDB Atlas and navigate to the cluster to which that Streamkap should connect
- Click
Connect

- Click
Connect your Application
- Copy this connection string
MongoDB Shell
- Connect to your replica set or primary node using the MongoDB shell as an Admin user.
- Execute the
db.adminCommand( { replSetGetStatus : 1 } ).members
command. - Copy the host identifier and optionally the alternative host identifiers.
- A valid connection string:
- The connection string must be in either of these formats:
- Standard (for non-MongoDB Atlas deployments)
- DNS Seed List (for MongoDB Atlas deployments)
- The connection string must have the following parameters. They will be added automatically if they are not included:
retryWrites=true
retryReads=true
w=majority
- The connection string must be in either of these formats:
Granting Privileges
MongoDB Atlas
- Login to MongoDB Atlas
- In the left-hand navigation menu, go to Security > Database Access.
- Click New Database User.
- Choose the password authentication method.
- Enter the username and password for the new Streamkap user (Suggest the username
streamkap_user
) - In the Database User Privileges drop-down menu, select Grant Specific User Privileges.
- Under Specific Privileges, add the following roles/privileges:
readAnyDatabase
read
on thelocal
database
- Click Add User.
Enable Snapshots through MongoDB Atlas
You will need to create a streamkap_signal
collection and give permissions to the streamkap user/role. Streamkap will use this collection for managing snapshots.
This collection can exist in a different database (on the same MongoDB cluster) to the database Streamkap captures data from.
Please create the signal collection with the name
streamkap_signal
. It will not be recognised if given another name.
- Create the collection
streamkap_signal
. This can be done via the MongoDB Atlas web app or Mongo Compass:- Navigate to the streaming cluster
- Navigate to
Collections
and create a new collection in the streaming database by clicking the+
button next to the database name. Name the collectionstreamkap_signal
. - Give the Streamkap user permissions to
readWrite
on thestreamkap_signal
collection- Return to Database User Privileges drop-down menu, select Grant Specific User Privileges
- Under Specific Privileges, add the following roles/privileges:
readWrite@<database>.streamkap_signal
MongoDB Shell
- Using MongoDB Shell, connect to your primary node or replica set
- Create a user for Streamkap. Replace password with your choice.
use admin
db.createUser({
user: "streamkap_user",
pwd: "<password>",
roles: [ "readAnyDatabase", {role: "read", db: "local"} ]
})
Enable Snapshots through MongoDB Shell
You will need to create the collection and give permissions to the streamkap user/role.
db.createCollection("streamkap_signal")
db.grantRolesToUser("streamkap_user", [
{ role: "read", db: "{database}" },
{ role: "readWrite", db: "{database}", collection: "streamkap_signal" }
])
--When later setting up the connector, you must include this collection
Consider Access Restrictions
- Visit Connection Options to ensure Streamkap can reach your database
Setup MongoDB Connector in Streamkap
- Create a new MongoDB Atlas Source
- Enter the following information:
- Name for your Connector
- Connection String- Copy the connection string from earlier steps but replace username and password in the string with the one you created earlier.
- Connection Mode (Default
replica_set
): Specifies the strategy that the connector uses when it connects to a MongoDB cluster - Array Encoding: Specify how Streamkap should encode MongoDB array types.
Array
is the optimal method but requires all elements in the array to be of the same type.Document
orString
should be used if the MongoDB arrays have mixed types - Signal Table Database: Streamkap will use a collection in this database to manage snapshots e.g.
public
. See Enable Snapshots for more information - (Optional) Include Schema?: If you plan on streaming data from this Mongo Source to Rockset, set this option to No
- Advanced Parameters
- Snapshot Mode (Default
When Needed
) See MongoDB Snapshot Modes for more information - Represent Binary Data As (Default
bytes
) - Snapshot Chunk Size (Default
1024
) - This is the number of rows read at a time when snapshotting. This is a low safe value. As a guide, if you have 100m + rows of data you may want to move this to5120
. If you have 1bn then a higher number still will allow you to backfill faster. - Max Batch Size (Default
2048
) - A value that specifies the maximum size of each batch of events that the connector processes. Only increase if experiencing lag
- Snapshot Mode (Default
- Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.
- Click Save
The connector will take approximately 1 minute to start processing data.
Updated 20 days ago