Search docs.molo17.com
    Configuration guide Connectors

    Set up a MongoDB change streams source

    The MongoDB agent reads changes with MongoDB change streams. A change stream needs a replica set, and Gluesync needs a user that can read the collection, open a change stream and run collMod on it. This guide shows how to check the replica set, create that user, connect Gluesync, copy a collection to PostgreSQL with a snapshot and CDC, stop and resume the entity, and recognize the errors you get when the replica set, the rights, the oplog or the document types do not fit.

    In this guide

    You will check that MongoDB runs as a replica set, create a role and a user for Gluesync, create the connection, map a collection to a PostgreSQL table, start a snapshot and CDC, check inserts, updates and deletes on the target, and stop and resume the entity. You will also see how the oplog window limits a resume, and what to do when it is too short. A symptom, cause and fix table closes the guide.

    Quick information

    Goal

    Replicate a MongoDB collection to a PostgreSQL table and confirm inserts, updates and deletes

    Audience

    Data engineers and MongoDB administrators

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Control Plane → Data pipelines → Entities; mongosh

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a MongoDB server that Core Hub can reach on port 27017 by default. This guide was tested with MongoDB 8.0.3 as a one-node replica set with access control on;

    2. You can run mongosh as an administrator of the server;

    3. You have a PostgreSQL database that Core Hub can reach, and a user that can create tables in it. The target is only there to check the result, any target works;

    4. You can sign in to the Control Plane with a role that can create pipelines.

    A standalone MongoDB server has no change streams, so Gluesync cannot capture changes from it. It can still run a snapshot, see A standalone server. The reference page MongoDB CDC with Gluesync: Change Streams Setup lists every connection option, MongoDB Atlas included.

    Guide steps

    Check that MongoDB is a replica set

    CLI

    1. Connect to the server with mongosh and run:

      rs.status().members.map(m => m.name + " " + m.stateStr)
    2. If the answer is an error, the server is standalone. Start mongod with the --replSet option (or set replication.replSetName in the configuration file), then initiate a one-node set:

      rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] })

      With access control on, the members of a replica set also need a key file (the --keyFile option) or X.509 certificates.

    Expected result

    rs.status() lists the member as PRIMARY.

    Create a role and a user for Gluesync

    CLI

    Gluesync reads documents, opens a change stream on each collection that runs CDC, and, when an entity starts, runs collMod on that collection to turn on changeStreamPreAndPostImages. MongoDB then keeps the before and after image of each change, which Gluesync needs to read the document of a delete and to read the old and new values of an update. This costs storage on the source, and it is only turned on for the collections that run CDC. The collMod command is reserved to the dbAdmin role or to a custom role.

    1. Sign in as an administrator and create a custom role. This is the role of the reference page: it reads and writes any database and may run collMod:

      db.getSiblingDB("admin").runCommand({ createRole: "GluesyncUserRole",
        privileges: [
          { resource: { db: "", collection: "" }, actions: [ "collMod" ] }
        ],
        roles: [ { role: "readWriteAnyDatabase", db: "admin" } ]
      })
    2. For a source that only reads, a narrower role is enough. In our test this role let Gluesync browse the collections, run the snapshot, enable the images and follow the change stream. We did not test with fewer actions. Replace shop with your database:

      db.getSiblingDB("admin").runCommand({ createRole: "GluesyncSourceRole",
        privileges: [
          { resource: { db: "shop", collection: "" },
            actions: [ "find", "changeStream", "collMod", "listCollections",
                       "listIndexes", "collStats", "dbStats" ] }
        ],
        roles: []
      })
    3. Create the user in the admin database, with the role you chose:

      db.getSiblingDB("admin").createUser({
        user: "gluesync", pwd: "<a strong password>",
        roles: [ { role: "GluesyncSourceRole", db: "admin" } ]
      })
    Expected result

    Each call answers { ok: 1 } with a $clusterTime.

    The user lives in the admin database. When you create the connection you must tell Gluesync, with the Auth Database property of step 4. Without it Gluesync authenticates against your database and fails.

    Create the example collection and the target table

    CLI

    Skip this step if you already have a collection and a table. The sample collection has a nested document, to show what Gluesync does with it. The PostgreSQL table is created first, so that the field mapping reads its types.

    1. In mongosh, create the collection:

      db = db.getSiblingDB("shop");
      db.orders.insertMany([
        { _id: 1, customer: "Acme Spa", status: "new", amount: 1500.50, quantity: 3,
          paid: true, created_at: new Date("2026-11-01T10:00:00Z"),
          tags: ["a", "b"], address: { city: "Milan", zip: "20100" } },
        { _id: 2, customer: "Globex", status: "paid", amount: 99.99, quantity: 1,
          paid: false, created_at: new Date("2026-11-01T11:15:00Z") }
      ]);
    2. On the PostgreSQL database, create the table:

      CREATE TABLE public.orders (
        _id integer PRIMARY KEY, customer varchar(100), status varchar(20),
        amount double precision, quantity integer, paid boolean,
        created_at timestamp(3), tags text[], address jsonb);
    Expected result

    db.orders.countDocuments() returns 2, and the table public.orders exists and is empty.

    Give every field of a collection one BSON type. Gluesync reads a sample of documents to decide the type of a field. A field that holds an integer in one document and a double in the next stops the entity, see the troubleshooting table. In mongosh, a number without a decimal part is stored as Int32: write Double(5) to store a double.

    Create the pipeline and the MongoDB connection

    UI

    1. Open Data pipelines;

    2. Select the + button (Create pipeline), or Create your first pipeline on a new installation. The Agents setup wizard opens on 1. Setup source;

    3. In Search, type mongo and select the MongoDB CDC agent tile under NoSQL. The form that opens is titled MongoDB;

      Setup source wizard step with the search field filled with mongo and the MongoDB CDC agent tile under NoSQL
    4. Enter Host name / Address, Port (27017), Database name (the database that holds the collections, shop here), Username and Password;

    5. Under Advanced connection properties, select Add property, pick Auth Database in Key and enter admin in Value. The other keys are Connection Timeout Seconds, Read Timeout Seconds, Use Dns Seed List (for an Atlas SRV address), Replica Set and Direct connection;

      MongoDB connection form with the host
    6. Optional: turn on Disable auth for a server without access control. Under Security, Upload 'trust store' file, Upload 'key store' file and Upload 'certificate store' file take the TLS files, each with Upload files or Use a path;

    7. Select Test and continue, then Next;

    8. On 2. Setup target(s), search for postgres, select the PostgreSQL CDC agent tile, enter the connection of your PostgreSQL database, and select Test and continue and Next;

    9. On 3. Ready, enter a Pipeline name and select Finish.

    Expected result

    The source card shows a green check mark and the connection (<user>@<host>:27017), and the pipeline opens with the message No entities found. In our test the connection test passed against a replica set and against a standalone server: it does not check for a replica set. Test and continue does check the user, so a wrong password or a wrong Auth Database fails here, with the message in a Failed to save banner of the form (see the troubleshooting table).

    localhost in the connection means the machine that runs Core Hub. If Core Hub runs in a container, enter a name that the container can resolve, for example the service name of MongoDB in the same Docker network.

    Choose the collection and map the fields

    UI

    1. In the pipeline, select Entities. The Add entities page opens;

    2. In the Object browser, expand the database shop and tick orders. The Field mapping dialog opens;

    3. Set Target Schema to public and Target Table to orders. Gluesync reads the table and links every field that has a column of the same name. The _id field is marked as the key, because it is the primary key of the table;

    4. Untick the row address on the source side. The dialog unticks its column on the target side too. See Data type notes for why;

      Field mapping dialog with the source collection orders on the left and the target table public.orders on the right
    5. Select Confirm, then Confirm and continue. The entity shop.orders is in the Hold group.

    Expected result

    The dialog shows Error (0) and Confirm is enabled. The Object browser lists the collections of the database named in the connection, with the number of collections.

    Start the snapshot and CDC

    UI

    1. In the row of the entity, select the start icon (Start). The dialog Sync changes (CDC) opens;

    2. In Start from, keep Full copy (marked Recommended for the first start) and keep Keep and merge under Data already in the target;

    3. Select Start entity.

    Expected result

    The entity moves to the Active group, and the pipeline shows the Healthy group with a green check mark. On the collection, db.getCollectionInfos({name:"orders"})[0].options now answers { changeStreamPreAndPostImages: { enabled: true } }: Gluesync turned the images on when it started.

    If the entity fails to start with the notification Entity Upsert Failed and the error 13 (Unauthorized), the user cannot run collMod: see the troubleshooting table.

    Verify the snapshot and the changes

    CLI

    1. Read the target table:

      SELECT _id, customer, status, amount, created_at, tags FROM public.orders ORDER BY _id;
    2. Change the source. These commands insert, update and delete a document:

      db = db.getSiblingDB("shop");
      db.orders.insertOne({ _id: 3, customer: "Initech", status: "new", amount: Double(10.10),
        quantity: 2, paid: false, created_at: new Date(), tags: ["x"] });
      db.orders.updateOne({ _id: 2 }, { $set: { status: "shipped", amount: 120.75 } });
      db.orders.deleteOne({ _id: 1 });
    3. Read the table again.

    Expected result

    After the snapshot the table has the two rows, with created_at in UTC and tags as {a,b} for the first. A few seconds after the three commands the row of _id 2 has the status shipped and the amount 120.75, the row 3 exists with tags {x}, and the row 1 is gone.

    Stop the entity and resume it

    UI

    Gluesync saves a position in the change stream, the resume token of the last change it delivered. When you start the entity again it asks MongoDB to replay everything after that token.

    1. In the row of the entity, select the pause icon (Pause), then Pause entity;

    2. In mongosh, change the source while the entity is stopped:

      db = db.getSiblingDB("shop");
      db.orders.insertOne({ _id: 6, customer: "Oscorp", status: "new", amount: Double(7),
        quantity: 1, paid: true, created_at: new Date() });
      db.orders.updateOne({ _id: 3 }, { $set: { status: "shipped" } });
      db.orders.deleteOne({ _id: 2 });
    3. Select the start icon. In Start from, Where it stopped is now available and selected (Continue from the last checkpoint. Nothing is lost.). Select Start entity.

    Expected result

    A few seconds later the target has the new row 6, the row 3 with the status shipped, and no row 2. The three changes made during the stop were replayed.

    The oplog window

    MongoDB can only replay a change while the change is still in the oplog of the replica set. The time between the oldest and the newest entry is the oplog window. If you stop an entity for longer than that, the saved token is gone and MongoDB refuses it with ChangeStreamHistoryLost.

    1. Read the window on the primary:

      rs.printReplicationInfo()

      The line log length start to end is the window.

    2. Make the oplog big enough to cover your longest stop, plus the time you need to notice it. The size is in megabytes:

      db.adminCommand({ replSetResizeOplog: 1, size: <size in MB> })

    We reproduced the failure. We shrank the oplog of the test server to 990 MB, paused the entity, and wrote about 4 GB of other data. After that, starting the entity from Where it stopped failed on every attempt. The entity page showed Source read failed with the message Generic Exception, and the Core Hub log held the line Reset last event id checkpoint due to lost history issue for shop.orders followed by ChangeStreamHistoryLostException. Start the entity with Full copy to recover, see the troubleshooting table.

    Entity page with the Source read failed banner and the message Generic Exception

    Data type notes

    Gluesync reads a sample of the documents to find the fields and their BSON types, and shows them in the field mapping. What we saw with the collection above:

    Source field Gluesync type Target column and notes

    _id (an integer)

    INT32

    integer. The _id field is the key of the entity. OBJECT_ID is among the source types the mapping offers, we did not test it.

    customer, status (text)

    STRING

    varchar. Give the length in the table.

    amount

    DOUBLE

    double precision.

    quantity

    INT32

    integer.

    paid

    BOOLEAN

    boolean.

    created_at (a date)

    DATE_TIME

    timestamp. The value arrives in UTC, for example 2026-11-05 14:45:20.959.

    tags (an array of text)

    ARRAY

    text[], for example {x}. A document without the field gives NULL.

    address (a nested document)

    DOCUMENT

    The GA field mapping marks a DOCUMENT linked to jsonb, json, text or varchar of a PostgreSQL table as Incompatible data types, so leave the field unticked for a PostgreSQL target. Do not set the source type to STRING to work around it: the read then fails.

    The source types in the mapping are STRING, REGULAR_EXPRESSION, OBJECT_ID, BOOLEAN, INT32, INT64, DOUBLE, DECIMAL128, DATE_TIME, DATE, TIME, BINARY, DOCUMENT and ARRAY.

    Settings that change the read

    The source settings are on the pipeline, in the gear menu → Edit source. The Advanced settings tab offers, after Add property: Source change retention in hours, Enable Recursion Protection, Query Studio maxTimeMS, Schema discovery sample size and Technical fields. The reference page MongoDB source advanced settings explains each one. Turn on Enable Recursion Protection only for two-way pipelines that write to the same collection.

    The entity settings are in the three-dot menu of the entity → Edit settings → Fine tuning:

    Setting What it does

    Max number of items per iteration on fetch

    The window of documents read at once (10000 by default).

    Max CDC message size (KB)

    The size of a packet of changes (4096 by default).

    Max snapshot page size (KB)

    The size of a snapshot page. Empty means the default, 8192 KB.

    Polling interval (ms)

    The time between two polls of the source (500 by default).

    Snapshot writing concurrency

    The number of threads that write one transaction during the snapshot, from 1 to 10 (2 by default).

    Snapshot only

    The entity runs the snapshot and never opens a change stream.

    The Speed choices of the start dialog (Standard, Faster, Custom) and Bulk load are write-path choices on the target. See Tune replication throughput.

    A standalone server

    With Snapshot only on, an entity ran its snapshot against a standalone server and stopped in the Hold group, with no error. Without it, the snapshot ran too, and then the start failed on the change stream (see the troubleshooting table). Use Snapshot only for a one-time copy, and a replica set for CDC.

    Pipeline page with a notification MongoDB Command Error that says the changeStream stage is only supported on replica sets

    Troubleshooting

    Problem Possible cause What to do

    Test and continue fails with Failed to save and Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='gluesync', source='shop', password=<hidden>, mechanismProperties=<hidden>}

    The user is defined in the admin database, but Gluesync authenticates against the database of the connection (source='shop').

    Add the property Auth Database with the value admin under Advanced connection properties.

    Test and continue fails with Exception authenticating MongoCredential{mechanism=SCRAM-SHA-256, userName='gluesync', source='admin', ...} after you set Auth Database

    The user name or the password is wrong.

    Check them with mongosh --host <host> -u gluesync --authenticationDatabase admin.

    The notification Entity Upsert Failed with Command execution failed on MongoDB server with error 13 (Unauthorized): 'not authorized on shop to execute command { collMod: "orders", changeStreamPreAndPostImages: { enabled: true } .... The entity stays in Hold

    The user cannot run collMod. A role such as readWrite is not enough.

    Grant collMod with a custom role (step 2) or the dbAdmin role, then start the entity again.

    The notification MongoDB Command Error with error 40573 (Location40573): 'The $changeStream stage is only supported on replica sets'. The snapshot is done, the entity is in the Error group

    The server is standalone.

    Run MongoDB as a replica set (step 1), or turn on Snapshot only in the entity settings for a one-time copy.

    Source read failed with class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap'), and the entity retries without end

    A field typed DOUBLE holds an Int32 in a later document. The insert is in the change stream, so every retry reads it again.

    Store one type per field (Double(5) in mongosh), then start the entity with Full copy under Start from.

    The mapping shows Incompatible data types on a DOCUMENT field, or the notification Class Cast says class org.bson.Document cannot be cast to class java.lang.String after you set the source type to STRING

    A nested document cannot be linked to a PostgreSQL column in the GA field mapping, and it cannot be read as text.

    Untick the field in the mapping, and keep the source type DOCUMENT. See Data type notes.

    The mapping shows Field function is required or define a UDF and Confirm is disabled

    A column of the target table is ticked, and no source field is linked to it.

    Untick the column, or link a field to it.

    Source read failed, Generic Exception, and the log line Reset last event id checkpoint due to lost history issue

    The saved resume token is older than the oplog window. MongoDB answers ChangeStreamHistoryLost.

    Start the entity with Full copy. With Keep and merge, the target keeps rows that were deleted on the source during the gap: choose Empty the table first to remove them. Then make the oplog bigger (The oplog window).

    A document you deleted on the source is still in the target after a Full copy

    Keep and merge only overwrites rows with the same key.

    Start with Empty the table first, or delete the row on the target.