Static MongoDB CDC with Gluesync: Change Streams Setup

Source data from MongoDB

Prerequisites

To have Gluesync working on your MongoDB instance you will need to have:

  • Valid user credentials with permission to read and write to the source database.

  • Valid user credentials with permission to modify collection properties in order to allow Gluesync to enable data capture stream if necessary. This can be done through the role dbAdmin or through a custom role.

Custom Gluesync Role

As mentioned before, Gluesync needs the permission to check and enable the data capture stream feature for collections that have it disabled using the collMod command. This is a privileged command, allowed for users with dbAdmin role. If this role is too permissive, you can create a custom role.

You can create this role from MongoDB shell with the command below:

db.runCommand({ createRole: "GluesyncUserRole",
  privileges: [
    { resource: { db: "", collection: "" }, actions: [ "collMod" ] }
  ],
  roles: [ { role: 'readWriteAnyDatabase', db: 'admin' } ]
})

This role already allow the user to read and write on any database, allowing the collMod command usage. It’s also possible to fine-tune the role according to your preferences. This is the minimum requirement for Gluesync.

Once done, you need to update the role of the user using the command below:

db.updateUser("your_user_name", { roles: [ { role: 'GluesyncUserRole', db: 'admin' } ] })

Setup via Web UI

  • Hostname / IP Address: DNS record or IP Address of your server;

  • Port: Optional, defaults to 27017;

  • Database name: Name of your source database;

  • Username: Username with read & write access to the source;

  • Password: Password belonging to the given username.

  • Tls certificates: (optional) File browser to let you upload your certificates;

Connecting to MongoDB Atlas

If you are using MongoDB Atlas DBaaS, you can connect to your cluster by enabling the Use Dns Seed List param avaible in the Host Credentials section, just like this screenshot here below:

MongoDB Atlas Use Dns Seed List setting

This will let the agent retrieve the cluster definition from the SRV record in the provided DNS address, this is mandatory when you are performing a connection to MongoDB Atlas (for example).

Custom host credentials

This can be also set via Rest API by specifying customHostCredentials param.

  • Use DNS Seed List: (optional, defaults to false) Set it to true to let the MongoDB SDK retrieve the cluster definition from the SRV record in the provided DNS address, this is mandatory when you are performing a connection to MongoDB Atlas (for example);

  • Auth Database: (optional, defaults to null) Use this field when authenticating against a MongoDB server that requires you to specify a different authentication database;

  • Direct Connection (optional, defaults to false) Uses Direct connection over MongoDB;

  • Replica Set (optional), defaults to null: Configure the MongoDB SDK to talk with the specified replica set.

  • Read Timeout (optional), defaults to 60: Configure the MongoDB SDK to use this specific read timeout.

  • Connect Timeout (optional), defaults to 60: Configure the MongoDB SDK to use this specific connection timeout.

Specific configuration

The following example shows how to apply the agent-specific configurations via Rest API.

  • Enable Recursion Protection: (optional, defaults to false) Boolean value that avoids recursion when two-way sync capability is configured on the same table;

Setup via Rest APIs

Here following an example of calling the CoreHub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "host-address",
        "port": 27017,
        "databaseName": "db_name",
        "username": "",
        "password": "",
        "enableTls": true,
        "certificatePath": "/myPath/cert.pem"
      },
      "customHostCredentials": {
        "replicaSet": "rs0",
        "directConnection": true
      }
}'

Setup specific configuration

The following example shows how to apply the agent-specific configurations via Rest API.

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/specific' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "configuration": {
      "enableRecursionProtection": false
  }
}'

Deploying in MongoDB Atlas DBaaS

Gluesync is fully compatible with working within MongoDB Atlas DBaaS. To establish a connection to a MongoDB Atlas Cluster you just need to have:

  • valid .pem certificate file;

  • the hostname of your Atlas instance, for example: cluster0.instancereferencename.mongodb.net

credentials keypair is not needed when connecting using a .pem certificate to your instance.

Troubleshooting

Here is the list of common errors.

  • Unauthorized - Error 13: It is possible that the wrong roles have been specified for the user. Check the roles and ensure that the user has at least read/write permissions and the collMod privilege.

  • AuthenticationFailed - Error 18: You might either have specified a wrong database name as an authentication database or the user that you’re using to connect against the source database does not belong to it, you should issue the correct authentication database where this user is defined. For more detailed info please check out the official MongoDB documentation by visiting the following link: using Authentication Database.