MongoDB Atlas

MongoDB Atlas Change Data Capture Setup with Streamkap

Prerequisites

  • MongoDB version ≥ 4.4
  • MongoDB Replica Set or Sharded Cluster
  • Connection details
  • Streamkap user and role

MongoDB Atlas Setup

Obtain Connection String

MongoDB Atlas

  • Login to MongoDB Atlas and navigate to the cluster to which that Streamkap should connect
  • Click Connect
  • Click Connect your Application

  • Copy this connection string

Granting Privileges

MongoDB Atlas

  • Login to MongoDB Atlas
  • In the left-hand navigation menu, go to Security > Database Access.
  • Click New Database User.
  • Choose the password authentication method.
  • Enter the username and password for the new Streamkap user (Suggest the username streamkap_user)
  • In the Database User Privileges drop-down menu, select Grant Specific User Privileges.
  • Under Specific Privileges, add the following roles/privileges:
    • readAnyDatabase
    • read on the local database
  • Click Add User.

Enable Snapshots through MongoDB Atlas

You will need to create a streamkap_signal collection and give permissions to the streamkap user/role. Streamkap will use this collection for managing snapshots.

This collection can exist in a different database (on the same MongoDB cluster) to the database Streamkap captures data from.

❗️

Please create the signal collection with the name streamkap_signal. It will not be recognised if given another name.

  • Create the collection streamkap_signal. This can be done via the MongoDB Atlas web app or Mongo Compass:
    • Navigate to the streaming cluster
    • Navigate to Collections and create a new collection in the streaming database by clicking the + button next to the database name. Name the collection streamkap_signal.
    • Give the Streamkap user permissions to readWrite on the streamkap_signal collection
      • Return to Database User Privileges drop-down menu, select Grant Specific User Privileges
      • Under Specific Privileges, add the following roles/privileges:
        • readWrite@<database>.streamkap_signal

MongoDB Shell

  • Using MongoDB Shell, connect to your primary node or replica set
  • Create a user for Streamkap. Replace password with your choice.
use admin  
   db.createUser({  
     user: "streamkap_user",  
     pwd: "<password>",  
     roles: [ "readAnyDatabase", {role: "read", db: "local"} ]  
   })

Enable Snapshots through MongoDB Shell

You will need to create the collection and give permissions to the streamkap user/role.

db.createCollection("streamkap_signal")

db.grantRolesToUser("streamkap_user", [
  { role: "read", db: "{database}" },
  { role: "readWrite", db: "{database}", collection: "streamkap_signal" }
])
--When later setting up the connector, you must include this collection

Consider Access Restrictions

Streamkap Setup

  • Go to Sources, choose MongoDB then MongoDB Atlas
  • Enter the following information:
    • Name for your Connector
    • Connection String: Copy the connection string from earlier steps but replace username and password in the string with the one you created earlier.
    • Connection Mode (default replica_set): Specifies the strategy that the connector uses when it connects to a MongoDB cluster
    • Array Encoding: Specify how Streamkap should encode MongoDB array types. Array encodes them as a JSON array but requires all elements in the arrays to be of the same type e.g. array of integers. Array_String encodes them as a JSON string and must be used if the MongoDB arrays have mixed types
    • Nested Document Encoding: Specify how Streamkap should encode nested documents. Document encodes them as JSON objects but may be problematic for complex (e.g. multiple levels of nested sub documents and arrays, sub arrays of nested documents) documents. String encodes them as a JSON string and we recommend it if the MongoDB nested documents are complex
    • Signal Table Database: Streamkap will use a collection in this database to manage snapshots e.g. public. See Enable Snapshots for more information
    • Connect via SSH Tunnel. See SSH Tunnel
    • Add Schemas/Tables. Can also bulk upload here. The format is a simple list of each schema or table per row saved in csv format without a header.
    • Click Save

The connector will take approximately 1 minute to start processing data.

Updated 5 days ago