Set up a MongoDB target for snapshots and CDC

The simplest choice is the built-in role readWrite on the target database:

use shop_mirror
db.createUser({
  user: "gluesync",
  pwd: "<a strong password>",
  roles: [ { role: "readWrite", db: "shop_mirror" } ]
})

If you want fewer rights, create a role with only the actions that the writes use, and give it to the user. This role was tested end to end, with snapshot, insert, update and delete:

db.getSiblingDB("admin").createRole({
  role: "GluesyncTargetWriter",
  privileges: [ {
    resource: { db: "shop_mirror", collection: "" },
    actions: [ "find", "insert", "update", "remove", "listCollections" ]
  } ],
  roles: []
})
db.getSiblingDB("shop_mirror").createUser({
  user: "gluesync",
  pwd: "<a strong password>",
  roles: [ { role: "GluesyncTargetWriter", db: "admin" } ]
})

Check the user:

db.getSiblingDB("shop_mirror").getUser("gluesync")

In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

In 1. Setup source, choose the PostgreSQL CDC agent tile and fill Base settings with the Host name / Address, Port, Database name, Username and Password of your source. Select Test and continue, then Next;

In 2. Setup target(s), search for mongo and choose the MongoDB CDC agent tile. The tile is the same for a source and for a target;

Fill Base settings:

Select Test and continue. The target appears as <user>@<host>:27017 with a green check mark. Select Next, enter a Pipeline name, and select Finish.

In the pipeline, select Entities. The Add entities page opens with the Object browser of the source. Expand the schema public and select orders and order_lines. Each table gets a check mark and a gear icon;

Select Confirm and continue. Both entities appear in the pipeline on Hold;

Open public.orders and select Edit, then the mapping icon next to the table. The Field mapping dialog opens.

Select the play button of public.orders. The Sync changes (CDC) dialog opens;

Keep Full copy under Start from and Keep and merge under Data already in the target. Rows with the same key are overwritten and the others stay. This is the choice that never deletes a document;

Select Start entity. Do the same for public.order_lines.

Open the entity and select Edit. Select the gear icon of the table, then the tab with the name of the MongoDB target. You can do this while the entity runs, and the entity then stops;

Under Document key, turn on Customize document key. The switch reads User defined key;

Set a Prefix, a Separator (::, -, _, / or none), the Fields and the Suffix. Key order sets the order of the key fields. The Preview shows the result;

Close the dialog and select Confirm and continue. The entity shows a Warning. Select its play button and keep Where it stopped, which continues from the last checkpoint.

Writers. In the start dialog, under Speed, choose Faster, or choose Custom and set Writers during the copy (1 to 10) and Parallel reads (1 to 10, split by a column such as the key). The entity setting Snapshot writing concurrency (the Fine tuning tab) has the same range and a default of 2;

Batch size. In Fine tuning, Max items per iteration on fetch (10000 by default) is the window of rows that the source reads and sends. Max CDC message size (KB) (4096) and Max snapshot page size (KB) (8192) cap the size of a message;

Indexes. Every secondary index you add on the collection makes each upsert slower. Create only the indexes that your queries need, and see step 8 for the unique ones.

A unique index on the collection. Create a unique index on a field that the documents carry, then insert a source row with a value that already exists:

db.orders.createIndex({ note: 1 }, { unique: true, name: "note_unique" })

INSERT INTO public.orders VALUES (100002, 'dup', 1.00, 'new', now(), current_date, false, '{x}', 'note 5');

Bulk write operation error on MongoDB server mongodb:27017. Write errors: [BulkWriteError{index=0, code=11000, message='E11000 duplicate key error collection: shop_mirror.orders index: note_unique dup key: { note: "note 5" }', details={}}].

A user that cannot write. Give the user the role read instead of readWrite, and insert a row on the source. The entity goes to Error, and the Core Hub log shows Command execution failed on MongoDB server with error 13 (Unauthorized): 'not authorized on shop_mirror to execute command { update: "orders", ordered: true, writeConcern: { w: "majority" }, .... Grant readWrite, or a role with find, insert, update and remove, as in step 1;

A user in another database. A user created in admin fails in the target form with Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='gs_admin_scope', source='shop_mirror', ...}. Add the property Auth Database with the value admin, and Test and continue passes.