Skip to content
Streamkap
Esc
↑↓navigate↵open⌘Jpreview
On this page

DocumentDB

Setup guide for connecting Amazon DocumentDB as a Streamkap source, covering change streams, the signal collection, and connector configuration.

Prerequisites

  • DocumentDB version ≥ 4.x
  • A database user with sufficient privileges to create database users and collections

DocumentDB Setup

1. Grant Database Access

2. Enable Change Streams

Change streams allow applications to access real-time data changes. The Connector relies on DocumentDB’s implementation of this.

Set Change Stream Log Retention Policy

Change stream logs should be retained for a minimum of 48 hours. We recommend 7 days.

3. Create Database User

It’s recommended to create a separate user for the Connector to access your DocumentDB database.

  • Using MongoDB Shell, connect to your primary node or replica set.
  • Create a user for Streamkap using the script below. Replace password with your choice.
use admin
db.createUser({
  user: "streamkap_user",
  pwd: "{password}",
  roles: [ "readAnyDatabase", {role: "read", db: "local"} ]
})

4. Enable Snapshots

To backfill your data, the Connector needs to be able to perform snapshots. See Snapshots & Backfilling for more information.

You will need to create a signal collection and give permissions to the streamkap_user. The Connector will use this collection for managing snapshots.

This collection can exist in a different database (on the same DocumentDB instance) to the database Streamkap captures data from.

db.createCollection("streamkap_signal")

db.grantRolesToUser("streamkap_user", [
  { role: "read", db: "{database}" },
  { role: "readWrite", db: "{database}", collection: "streamkap_signal" }
])

Streamkap Setup

Follow these steps to configure your new connector:

1. Create the Source

2. Connection Settings

  • Name: Enter a name for your connector.

  • Connection String: The DocumentDB connection string.

  • Connection Mode (optional): Default is replica_set.

  • Array Encoding: Specify how Streamkap should encode DocumentDB array types. Array is the optimal method but requires all elements in the array to be of the same type. Document or String should be used if the DocumentDB arrays have mixed types.

  • Include Schema? (optional): If you plan on streaming data from this DocumentDB Source to Rockset, set this option to No.

3. Snapshot Settings

  • Signal Collection: Full path to the signal collection including database and collection name (e.g., streamkap.streamkap_signal). This collection is used for incremental snapshotting. See Enable Snapshots for setup instructions.

4. Advanced Parameters

  • Represent binary data as: Specifies how the data for binary columns should be interpreted. Your destination for this data can impact which option you choose. Default is bytes.

Click Next.

5. Database and Collection Capture

  • Add Database/Collections: Specify the database(s) and collection(s) for capture.
    • You can bulk upload here. The format is a simple list of databases and collections, with each entry on a new row. Save as a .csv file without a header.

Click Save.