MongoDB Atlas

​

Prerequisites

• MongoDB version ≥ 5.0
• A MongoDB user with sufficient privileges to create database users and collections

​

MongoDB Atlas Setup

​

1. Grant Database Access

• Configure one of the Connection Options to ensure Streamkap can reach your database.

​

2. Create Database User

​

MongoDB Atlas

• Log in to MongoDB Atlas and navigate to the MongoDB cluster.
• In the left-hand navigation menu, go to Security > Database Access.
• Click New Database User.
• Choose the password authentication method.
• Enter the username and password for the new Streamkap user (e.g. streamkap_user).
• In the Database User Privileges drop-down menu, select Grant Specific User Privileges.
• Under Specific Privileges, add the following roles/privileges:
  • readAnyDatabase
  • read on the local database
• Click Add User.

​

MongoDB Shell

• Using MongoDB Shell, connect to your primary node or replica set.
• Create a user for Streamkap using the script below. Replace password with your choice.

use admin
db.createUser({
  user: "streamkap_user",
  pwd: "{password}",
  roles: [ "readAnyDatabase", {role: "read", db: "local"} ]
})

​

3. Enable Snapshots

​

MongoDB Atlas

• Navigate to the MongoDB cluster.
• Navigate to Collections and create a new collection in the database by clicking the + button next to the database name.
  • Name the collection streamkap_signal.
  • Give the Streamkap user permissions to readWrite on the streamkap_signal collection.
  • Return to the Database User Privileges drop-down menu, select Grant Specific User Privileges.
    • Under Specific Privileges, add the following roles/privileges: readWrite@{database}.streamkap_signal

​

MongoDB Shell

db.createCollection("streamkap_signal")

db.grantRolesToUser("streamkap_user", [
  { role: "read", db: "{database}" },
  { role: "readWrite", db: "{database}", collection: "streamkap_signal" }
])

​

4. Heartbeats

​

Layer 1: Connector heartbeats (enabled by default)

​

Layer 2: Source database heartbeats (recommended)

use streamkap
db.createCollection("streamkap_heartbeat")

db.grantRolesToUser("streamkap_user", [
  { role: "read", db: "streamkap" }
])

exports = async function() {
  const serviceName = "mongodb-atlas"; // Your cluster service name
  const database = "streamkap";
  const collection = "streamkap_heartbeat";

  const coll = context.services
    .get(serviceName)
    .db(database)
    .collection(collection);

  const result = await coll.updateOne(
    { _id: "heartbeat" },
    {
      $set: {
        last_update: new Date()
      }
    },
    { upsert: true }
  );

  console.log(`Heartbeat updated: ${JSON.stringify(result)}`);
  return result;
};

​

5. Obtain Connection String

​

MongoDB Atlas

• Log in to MongoDB Atlas and navigate to the cluster to which Streamkap should connect.
• Click Connect.

• Click Connect your Application.

• Copy the connection string.

​

MongoDB Shell

• Connect to your replica set or primary node using the MongoDB shell as an Admin user.
• Run db.getMongo() method to return your connection string.
  • We recommend the connection string have the following parameters. They will be added automatically if not included:
    • w=majority
    • readPreference=primaryPreferred

​

Streamkap Setup

​

1. Create the Source

• Navigate to Add Connectors.
• Choose MongoDB.
  • Select MongoDB Atlas.

​

2. Connection Settings

• Name: Enter a name for your connector.
• Connection String: Copy the connection string from earlier steps but replace username and password in the string with the one you created earlier.
• Array Encoding: Specify how Streamkap should encode MongoDB array types. Array encodes them as a JSON array but requires all elements in the arrays to be of the same type e.g. array of integers. Array_String encodes them as a JSON string and must be used if the MongoDB arrays have mixed types.
• Nested Document Encoding: Specify how Streamkap should encode nested documents. Document encodes them as JSON objects but may be problematic for complex (e.g. multiple levels of nested sub documents and arrays, sub arrays of nested documents) documents. String encodes them as a JSON string and we recommend it if the MongoDB nested documents are complex.
• Connect via SSH Tunnel: The Connector will connect to an SSH server in your network which has access to your database. This is necessary if the Connector cannot connect directly to your database.
  • See SSH Tunnel for setup instructions.

​

3. Snapshot Settings

• Signal Table Database: Streamkap will use a collection in this database to manage snapshots. See Enable Snapshots for more information.

​

4. Database and Collection Capture

• Add Database/Collections: Specify the database(s) and collection(s) for capture.
  • You can bulk upload here. The format is a simple list of databases and collections, with each entry on a new row. Save as a .csv file without a header.
  • If you configured Layer 2 heartbeats, include the heartbeat collection (e.g., streamkap.streamkap_heartbeat). See Heartbeats for setup instructions.