Search docs.molo17.com
    Configuration guide Connectors

    Set up a Couchbase DCP source with CDC

    The Couchbase DCP agent reads the changes of a bucket from the Database Change Protocol (DCP) stream of the data service. It does not need the Eventing service and it does not deploy any function. This guide creates a Couchbase user with the roles the agent needs, copies a collection of JSON documents into a PostgreSQL table with a snapshot, follows inserts, updates and deletes with CDC, and shows what a document becomes as a row. It then stops and resumes the entity and lists the errors that you are most likely to meet: a missing role, a missing index, a stream that cannot open and a field that does not fit a column.

    In this guide

    You will create a Couchbase user with four roles on one bucket, and an index for the snapshot. You will create the Couchbase DCP source and a PostgreSQL target in the Control Plane, add the collection as an entity, and map the fields of its documents to columns. You will start the entity with a full copy, change documents in Couchbase, check the three operations on the target, stop the entity, change more documents, and resume it from where it stopped. A table at the end lists the errors that were reproduced for this guide.

    Quick information

    Goal

    Replicate a Couchbase collection to a relational table with a snapshot followed by change data capture (CDC), with the least roles

    Audience

    Data engineers and Couchbase administrators

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Couchbase → RBAC users and indexes; Control Plane → Pipelines → Couchbase DCP agent → Base settings and Advanced settings; entity → Field mapping and start dialog

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a Couchbase Server 7.0 or later (Enterprise or Community) or a Capella cluster, with the Data, Index and Query services. This guide was tested with Couchbase Server Enterprise 7.6. You can run couchbase-cli and N1QL statements as an administrator;

    2. Core Hub can reach the cluster on port 8091 (management), 11210 (data) and 8093 (query). Use 18091, 11207 and 18093 with TLS. The Eventing ports 8096 and 18096 are not needed;

    3. You have a target to write to. This guide uses PostgreSQL 16 and a user that can create tables in the schema public. Any relational target works, and the steps for the source are the same;

    4. You can sign in to the Control Plane with a role that can create pipelines;

    5. You have a collection of documents. This guide uses the bucket sales, the scope shop and the collection orders, with 1,000 documents. The key of each document is order::<number>, and the fields are orderId (number), customer, status and createdAt (text), amount (number with decimals), paid (boolean), tags (array) and shipping (object with city and country).

    The statements and the messages quoted in this guide were run and captured against Couchbase Server Enterprise 7.6 as the source and PostgreSQL 16 as the target, with the Control Plane that ships with Gluesync 2.3. The Couchbase DCP reference lists every option and the REST API setup.

    How a document becomes a row

    The agent maps one collection to one entity, and one document to one row. Gluesync discovers the fields by sampling the documents of the collection with the N1QL INFER statement, so the field list reflects the documents that exist when you add the entity.

    In Couchbase In the target row

    The document key, for example order::1

    The column metaId. It is the key of the entity, and you mark it as the key in the field mapping. A DELETE reaches Gluesync with the document key and nothing else, so the key is how the target finds the row to delete

    A field with a text, number or boolean value

    A column. You choose the column type, and the length, precision and scale where the type needs them

    A nested object or an array, such as shipping or tags

    No column in the PostgreSQL target of this guide. The Control Plane marks the field as an error, so you leave it out of the entity

    A field that a document does not have

    NULL in the column

    A field that appears in new documents after you added the entity, such as a coupon code

    Nothing. The value is not replicated until you add the field to the mapping

    When you turn off Use Scope and Collections, the agent reads the whole bucket and matches documents to entities through the type field. A DELETE then needs a key that the agent can match to one entity, and the agent rejects a setup of several entities with a plain metaId key. See Entity and deletion matching without collections in the reference.

    Guide steps

    Create the user and grant the roles

    CLI

    The agent opens the DCP stream, creates a scope and an index in the bucket, and reads documents with N1QL. It does not need the admin role. Connect to a node as an administrator and create the user with four roles on the source bucket. Replace the passwords:

    couchbase-cli user-manage -c localhost:8091 -u Administrator -p '<admin password>' \
  --set --rbac-username gsuser --rbac-password '<a strong password>' \
  --roles 'data_dcp_reader[sales],scope_admin[sales],query_select[sales],query_manage_index[sales]' \
  --auth-domain local
    Role Why Gluesync needs it

    data_dcp_reader[sales]

    Opens the DCP stream and reads the changes. It cannot write

    scope_admin[sales]

    Creates the scope gluesync-scope in the bucket when you test the connection, with the collections gluesync-collection and metadata-collection

    query_select[sales]

    Discovers the fields of a collection with INFER and reads the snapshot

    query_manage_index[sales]

    Creates the index that the agent builds on each collection when the entity starts
    Expected result

    The command answers SUCCESS: User gsuser set. In the Web Console, Security lists gsuser with the four roles on sales.

    The reference page says that the DCP agent writes nothing to the source bucket. The documents of the bucket stay untouched, but the agent does create the scope gluesync-scope and one index for each collection it follows, and the user needs scope_admin and query_manage_index for it. The index is named gluesync_migration_<scope>_<collection>_type_index and covers the field type. Do not delete the scope or the index while an entity runs.

    On Community Edition, the granular roles are not available. The reference page tells you to give the user bucket_full_access on the source bucket instead, see Couchbase DCP reference. This guide did not test Community Edition.

    Create a primary index for the snapshot

    SQL

    The snapshot reads the collection with a N1QL SELECT and needs an index it can use. Create a primary index on each collection you replicate. Run it as an administrator, for example in the Query workbench:

    CREATE PRIMARY INDEX ON `sales`.`shop`.`orders`;
    Expected result

    The statement succeeds. SELECT name, keyspace_id FROM system:indexes WHERE bucket_id = "sales" lists #primary for the collection.

    If you do not want a primary index, you can instead grant the role query_use_sequential_scans on the collection to gsuser. Without one of the two, the snapshot fails, see the troubleshooting table.
    Create the source and the target in the Control Plane

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

    2. In 1. Setup source, open the Couchbase tile, which offers two connectors, and choose Couchbase DCP agent. Fill Base settings:

      • Host name / Address: a node of your cluster. Gluesync discovers the other nodes;

      • Port: 8091. Use 18091 for TLS and for Capella;

      • Bucket name: sales;

      • Username and Password: the user of step 1;

    3. Select Test and continue. The source appears with gsuser and the address. For TLS, upload your certificates under Security before you test, see Enable HTTPS and TLS to databases;

    Edit source page of the Couchbase DCP agent with host name couchbase
    Figure 1. The Base settings of the Couchbase DCP agent, in the Edit source page of a pipeline. The page is the same as in the wizard.

    1. Select Next. In 2. Setup target(s), search for PostgreSQL, choose the PostgreSQL CDC agent tile and fill the host, the port 5432, the database and the user of your PostgreSQL server. Select Test and continue, then Next;

    2. Enter a Pipeline name and select Finish.

    Expected result

    The pipeline appears in the HEALTHY group of the pipeline list with the Couchbase and PostgreSQL icons, and the entity table is empty. In Couchbase, the bucket has the new scope gluesync-scope.

    The Advanced settings tab of the source offers more properties, for example Use Scope and Collections (on by default), Index Replica Count and Custom 'type' field name. The Eventing properties in the same list do not apply to the DCP agent. See Couchbase source advanced settings.

    If the test fails, the connection is saved anyway and the notification Failed to save shows the cause. The troubleshooting table lists the messages.

    Add the collection and map the fields

    UI

    1. In the pipeline, select Entities. The Add entities page opens with the Object browser. It lists the scopes of the bucket, including _default and _system. Expand shop and tick the collection orders;

    2. In the Field mapping dialog, choose the Target Schema public, type orders in Target Table and choose Create table "orders". The dialog fills the nine fields of the sample documents: the eight fields and metaId, the document key. Scroll the lists if they are longer than the dialog;

    3. Fix what the dialog marks as an error, as listed here:

      • Untick the fields shipping and tags. The dialog marks the source type object as an error and offers no PostgreSQL type for it, and array becomes the invalid type array in the statement. See the troubleshooting table;

      • Give each varchar column a Length, for example 40 for createdAt, 50 for customer, 20 for status and 100 for metaId;

      • Give amount the Precision 12 and the Scale 2;

      • Change the Source type of orderId from number to long, and its target type to bigint, so the column holds an integer;

      • Tick the key box of metaId, on the source side;

    Field mapping dialog with the source collection shop.orders and the target table public.orders
    Figure 2. The field mapping of the collection. metaId is the key, shipping and tags are left out, and no error remains.

    1. Select Confirm. The Create table orders dialog shows the read-only CREATE TABLE statement. It has no primary key yet: tick Force conversion of logical keys into physicals, and the statement ends with PRIMARY KEY ("metaId"). Select Create table orders;

    2. Back on the Add entities page, select Confirm and continue.

    Expected result

    The entity appears in the pipeline on Hold as shop.orders. In PostgreSQL, the table has the columns amount numeric(12,2), createdAt varchar(40), customer varchar(50), orderId bigint, paid boolean, status varchar(20) and metaId varchar(100), with metaId as the primary key.

    The key of the document is the only key Gluesync can use for a DELETE, so keep metaId as the key. A key built from document fields works only if it can be rebuilt from the document key, see Couchbase DCP reference. If you want a nested object in the target, use a target that stores JSON, or flatten the field before it reaches Gluesync.
    Start the entity with a full copy

    UI + SQL

    1. On the row of the entity, select the play button. The Sync changes (CDC) dialog opens;

    2. Under Start from keep Full copy, Recommended for the first start. It copies all the documents, then follows the changes. Under Data already in the target keep Keep and merge, and under Speed keep Standard;

    3. Select Start entity.

    Sync changes CDC dialog for shop.orders with Where it stopped disabled
    Figure 3. The start dialog of a new entity, with Full copy and Keep and merge.

    Check the rows on the target, and the index that the agent created on the source:

    -- on the target
SELECT count(*) FROM public.orders;
SELECT * FROM public.orders ORDER BY "orderId" LIMIT 3;
    -- on the source, in the Query workbench
SELECT name, keyspace_id, index_key FROM system:indexes WHERE bucket_id = "sales";
    Expected result

    The count is 1000, and the first rows show the document key in metaId (order::1), the text new or paid in status, and the amount with two decimals. The index list has gluesync_migration_shop_orders_type_index and #primary on orders. The entity is Active and Where it stopped is available from now on.

    Entity page of shop.orders with the Active status
    Figure 4. The entity is active and follows the changes. Last Transaction ID is the DCP position of the last change.
    Follow inserts, updates and deletes

    SQL

    Change the documents in Couchbase. Insert one document, update one and delete one:

    INSERT INTO `sales`.`shop`.`orders` (KEY, VALUE)
VALUES ("order::1001", {"orderId": 1001, "customer": "customer-new", "status": "new",
  "amount": 19.9, "paid": false, "createdAt": "2026-11-10T08:00:00Z"});
UPDATE `sales`.`shop`.`orders` SET status = "paid", paid = true, amount = 25.00
WHERE META().id = "order::1";
DELETE FROM `sales`.`shop`.`orders` WHERE META().id = "order::2";

    After a few seconds, read the three keys on the target:

    SELECT "metaId", status, paid, amount FROM public.orders
WHERE "metaId" IN ('order::1', 'order::2', 'order::1001') ORDER BY 1;
SELECT count(*) FROM public.orders;
    Expected result

    order::1 shows paid, t and 25.00, order::1001 exists, and order::2 is gone. The count is still 1000: one row was added and one was deleted. The delete worked because the table is keyed on the document key.

    A document that lacks a field gives NULL in its column, and a field that is not in the mapping is not replicated. In the test, a document without status and createdAt landed with both columns empty, and a document with an extra coupon field landed without it and without an error.

    Stop the entity and resume it

    UI + SQL

    The agent keeps the DCP position of the entity in Core Hub, not in Couchbase. Stopping the entity or Core Hub does not lose changes: when it starts again from the checkpoint, it reads the changes made in the meantime.

    1. Select the pause button of the entity, then Pause entity;

    2. In Couchbase, change documents while the entity is stopped: insert one, update one and delete one;

      INSERT INTO `sales`.`shop`.`orders` (KEY, VALUE)
VALUES ("order::1002", {"orderId": 1002, "customer": "customer-paused", "status": "new",
  "amount": 5.5, "paid": false, "createdAt": "2026-11-10T09:00:00Z"});
UPDATE `sales`.`shop`.`orders` SET status = "paid" WHERE META().id = "order::4";
DELETE FROM `sales`.`shop`.`orders` WHERE META().id = "order::5";

    3. Read the target. Nothing has changed yet. Then select the play button, choose Where it stopped and select Start entity.

    Expected result

    The dialog says Pick up from the checkpoint, with the time of the checkpoint, and Changes made while the entity was stopped are copied first. A few seconds after the start, order::1002 exists, order::4 is paid and order::5 is gone. The same happens when you stop and start Core Hub: the entity returns to Active on its own and the changes made while Core Hub was down arrive.

    Now skips the changes made while the entity was stopped. Full copy with Keep and merge copies every document again and overwrites the rows with the same key. The rows of documents that were deleted in the meantime stay in the target.
    Tune the speed

    UI

    1. In the start dialog, under Speed, select Faster or Custom. The diagram shows 1 read for the source: the agent opens one DCP stream for the whole source;

    2. Under Custom, Reading offers Read in parallel, off by default, which splits the table into ranges read at the same time during the copy. Writing offers Standard or Bulk load during the copy and during live changes, and Writers during the copy from 1 to 10. More lanes copy faster and put more load on both databases;

    3. Bulk load depends on the target. For PostgreSQL, see Tune replication throughput and Bulk load;

    If the rows per second stay low, check the target first.

    Troubleshooting

    Symptom or message Cause Fix

    Failed to save with Unexpected HTTP status 403 Forbidden and User needs the following permissions cluster.bucket[<bucket>].collections!write

    The user has no scope_admin role. The agent cannot create gluesync-scope when you test the connection. The values are saved anyway

    Grant scope_admin[<bucket>], open Edit on the source and test again

    Failed to save with WaitUntilReady timed out in stage WAIT_FOR_CONFIG and User authentication failed; check username and password.

    Wrong user or password. The notification appears after about a minute, and the values are saved anyway

    Correct the values in the Edit source page and test again

    Tables Columns Fetch fails with Could not authenticate query and User does not have credentials to run SELECT queries on default:sales.shop.orders. Add role query_select on default:sales.shop.orders to allow the statement to run.

    The user has no query_select role. The Field mapping dialog stays empty, because discovery runs INFER

    Grant query_select[<bucket>], then select the collection again

    Authentication Failed when you start the entity, with User does not have credentials to run index operations. Add role query_manage_index on default:sales.shop.orders to allow the statement to run. The statement in the message is CREATE INDEX gluesync_migration_shop_orders_type_index

    The user has no query_manage_index role. The agent cannot create its index

    Grant query_manage_index[<bucket>] and start the entity again

    Authentication Failed when you start the entity, with User does not have credentials to use sequential scans. Add role query_use_sequential_scans on the collection. The Core Hub log says Entity shop.orders require index

    The collection has no primary index, so the snapshot query has no index to use

    Create it with CREATE PRIMARY INDEX ON <bucket>.<scope>.<collection>;, or grant query_use_sequential_scans on the collection, and start the entity again

    The entity moves to the Error group with Could not open the Couchbase DCP stream on <host> (bucket <bucket>), not using TLS. A bootstrap failure looks the same whether the KV port is unreachable or the cluster certificate is not trusted, so check both.

    The message reports plaintext mode, not a refused TLS connection. In the test, the user had no data_dcp_reader role. The same message appears when Core Hub cannot reach the data port 11210, or when a certificate is not trusted

    Check the roles first, then the data port from the host of Core Hub, then the certificates. Grant data_dcp_reader[<bucket>] and start the entity with Where it stopped. The changes made while the entity was in error arrive

    In Field mapping, a shipping or tags row is red. The notifications say Incompatible data types and Missing data type, and the target type is Choose an option

    The source type object has no column type in the PostgreSQL target. The type array is accepted but becomes the invalid column type array

    Untick the field, or use a target that stores JSON. Do not leave the field in the entity

    The entity fails with Source read failed and Element class kotlinx.serialization.json.JsonObject is not a JsonPrimitive

    A nested object was forced to the source type string to remove the error in the dialog. The value of the document is an object, not a string

    Open Edit mapping and untick the field, then start the entity again

    Missing required field on a column

    A varchar column has no Length, or a decimal column has no Precision and Scale

    Fill the values, see step 4

    A field that you added to the documents later is not in the target

    The mapping lists the fields that existed when you added the entity. A new field is ignored without an error

    Open Edit mapping from the menu of the entity and add the field and a column for it

    The count of the target is lower than the source after a stop

    The entity was started with Now, which skips what changed while it was stopped

    Start it with Full copy and Keep and merge
    Entity page of shop.orders with the Error status and the Pipeline error Could not open the Couchbase DCP stream on couchbase bucket sales not using TLS
    Figure 5. A start that fails because the user has no data_dcp_reader role: the entity page shows the pipeline error with the message of the DCP stream.

    Related guides

    Contact MOLO17 Support if you need assistance with a Couchbase DCP source.