Search docs.molo17.com
    Configuration guide Connectors

    Set up a YugabyteDB source and target

    The YugabyteDB agent reads and writes through the YSQL API with the YugabyteDB JDBC smart driver. As a source it reads changes from a logical replication slot decoded with the wal2json plugin. As a target it writes with batched upserts. This guide creates the users and grants for both roles, replicates a table with an array column from one YugabyteDB database to another with a snapshot followed by live changes, and shows the failures you are most likely to meet.

    In this guide

    You will check the replication slot limit, create a source user with the REPLICATION attribute and a target user with the grants the writer needs, and create the tables. You will create the two connections in the Control Plane, map a table with an array column to a table that already exists, and start the entity with a full copy. You will check an insert, an update and a delete on the target, pause and resume the entity, and read the numbers of a 200,000-row copy.

    Quick information

    Goal

    Replicate YugabyteDB tables, arrays included, with a snapshot followed by change data capture (CDC), and keep the users, slots and grants healthy

    Audience

    Data engineers and database administrators who own a YugabyteDB cluster

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    YugabyteDB → YSQL users and grants; Control Plane → Pipelines → YugabyteDB agent → Base settings; entity → start dialog

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a YugabyteDB cluster and you can run YSQL as an administrator such as yugabyte. This guide was tested with YugabyteDB 2026.1 (PostgreSQL 15.12-YB), started with yugabyted as a single node;

    2. Core Hub can reach the YSQL port, 5433 by default, of the node you use;

    3. You can sign in to the Control Plane with a role that can create pipelines;

    4. You know that one replication slot is created for each entity that follows changes. The server limit max_replication_slots must cover them;

    5. You have a source database and a target database. This guide uses one database, demo, with two schemas: src for the source and tgt for the target. Both roles use the same agent, so everything you read for the source also applies to a YugabyteDB that is only a source, and the grants for the target apply to a YugabyteDB that is only a target.

    The messages quoted in this guide were captured in the Control Plane and in the Core Hub log with YugabyteDB 2026.1 as both source and target. The options of the agent are listed in the YugabyteDB CDC reference and in the YugabyteDB target reference.

    Guide steps

    Check the replication slot limit

    SQL

    Gluesync creates one replication slot, named gs_<entity id>_slot, for each entity that follows changes. A slot is created when you add the entity, not when you start it. Read the limit and the slots that exist:

    SHOW max_replication_slots;
SELECT slot_name, plugin, active FROM pg_replication_slots;
SELECT version();
    Expected result

    max_replication_slots returns a number, 10 in the test. The query on pg_replication_slots returns no row on a new cluster. Plan the limit for the number of entities that follow changes, plus the slots of other tools.

    An entity that is snapshot-only gets no slot, and a slot of an entity that you delete is dropped. On YugabyteDB, the limit of slots is a server setting. The YugabyteDB documentation describes max_replication_slots and cdc_max_virtual_wal_per_tserver: change them with your cluster tooling and restart the servers.
    Create the users and the tables

    SQL

    Gluesync does not need a superuser. The source user needs the REPLICATION attribute and SELECT on the tables it copies. The target user needs to read and write the target tables. Connect to the database as an administrator and replace the passwords:

    CREATE DATABASE demo;
\c demo

CREATE USER gluesync_src WITH REPLICATION LOGIN PASSWORD '<a strong password>';
CREATE USER gluesync_tgt WITH LOGIN PASSWORD '<another strong password>';

CREATE SCHEMA src;
CREATE SCHEMA tgt;

CREATE TABLE src.orders (
  id        integer PRIMARY KEY,
  customer  text NOT NULL,
  amount    numeric(12,2),
  tags      text[],
  placed_at timestamptz DEFAULT now()
);
INSERT INTO src.orders VALUES
  (1, 'Acme',    10.50, '{rush,gift}', now()),
  (2, 'Globex',  99.99, '{}',          now()),
  (3, 'Initech', NULL,  NULL,          now());

-- source user: read the source schema, now and for tables created later
GRANT USAGE ON SCHEMA src TO gluesync_src;
GRANT SELECT ON ALL TABLES IN SCHEMA src TO gluesync_src;
ALTER DEFAULT PRIVILEGES IN SCHEMA src GRANT SELECT ON TABLES TO gluesync_src;

    The default privileges apply to the tables created by the role that runs the statement. Repeat the grants for every other schema you replicate.

    Now create the target table as the target user, so that the user owns it. The table needs the same array column. Gluesync cannot propose an array type when it creates a table (see the troubleshooting table), so you create the table yourself:

    GRANT USAGE ON SCHEMA tgt TO gluesync_tgt;
GRANT CREATE ON SCHEMA tgt TO gluesync_tgt;
GRANT TEMPORARY ON DATABASE demo TO gluesync_tgt;
    -- connect as gluesync_tgt
CREATE TABLE tgt.orders (
  id        integer PRIMARY KEY,
  customer  text NOT NULL,
  amount    numeric(12,2),
  tags      text[],
  placed_at timestamptz
);

    For a target table that another role owns, grant what the writer uses instead of CREATE:

    GRANT SELECT, INSERT, UPDATE, DELETE ON tgt.orders TO gluesync_tgt;
GRANT TRUNCATE ON tgt.orders TO gluesync_tgt; -- only for "Empty the table first"
    Expected result

    \du lists gluesync_src with the Replication attribute. The target table exists and is empty. The TEMPORARY grant is usually already there, because YSQL grants it to PUBLIC. It matters when you or your security policy revoked it: the writer uses a temporary table for large batches, see the troubleshooting table.

    A table with a primary key works with the default replica identity: in the test, an update and a delete reached the target after ALTER TABLE src.orders REPLICA IDENTITY DEFAULT. FULL also works, and it writes the old values of every column to the log. Gluesync does not change the replica identity of your tables.
    Create the source connection

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

    2. In 1. Setup source, choose the YugabyteDB tile and fill Base settings:

      • Host name / Address: the DNS name or address of the node, as Core Hub sees it;

      • Port: 5433, or the YSQL port of your cluster;

      • Database name: the database that holds the tables. A slot reads one database;

      • Username and Password: the source user of step 2;

    3. Keep Load balance and Enable Recursion Protection off. Turn on Load balance only for a multi-node cluster, and add the other nodes under Settings → Additional hosts in the format host1:port1,host2:port2. Keep Enable Recursion Protection off, unless the same database is also a target of a bidirectional setup, see Set up bidirectional replication with recursion protection;

    4. Select Test and continue. The source appears with a green check mark.

    Agents setup wizard with the YugabyteDB agent
    Figure 1. The Base settings of the YugabyteDB agent as a source.

    1. Select Next. In 2. Setup target(s), choose the YugabyteDB tile again and fill the same fields with the target user of step 2. Select Test and continue, then Next;

    2. Enter a Pipeline name and select Finish.

    Expected result

    The pipeline appears in the HEALTHY group of the pipeline list with two YugabyteDB icons, and the entity table is empty. The connection test checks that the pool opens and the user signs in. It does not check the REPLICATION attribute or the free slots.

    In a lab that you start with yugabyted and no security options, the server accepts any password, so a wrong password passes the connection test. A cluster with authentication turned on rejects it. A wrong database name fails the test at once with Failed to initialize pool: FATAL: database "nodb" does not exist.
    Add the entity and map it to the existing table

    UI

    1. In the pipeline, select Entities. The Add entities page opens with the Object browser. Expand the schema src and tick the table orders. The Field mapping dialog opens;

    2. Under Target, choose the Target Schema tgt, then choose the existing table orders in Target Table. The dialog shows the five columns joined by arrows, with no error;

    3. Select Confirm, then Confirm and continue on the Add entities page.

    Field mapping dialog with the source table src.orders mapped to tgt.orders
    Figure 2. The mapping to the existing table. The array column tags keeps its type on both sides and the dialog shows no errors.
    Expected result

    The entity appears in the pipeline on Hold as src.orders. In the entity page, the Field Mappings section lists id with PK and NN, tags as array, and placed_at as timestamp with time zone (6). Entity Settings shows Max Write Batch Size 1000, Snapshot writing concurrency 2 and On duplicate key Upsert. A new slot gs_<entity id>_slot exists on the server.

    Start the entity with a full copy

    UI + SQL

    1. On the row of the entity, select the play button. The Sync changes (CDC) dialog opens;

    2. Under Start from keep Full copy, Recommended for the first start. The other choices are Where it stopped, which is not available before the first run, and Now, which skips what changed before the start;

    3. Under Data already in the target keep Keep and merge, and under Speed keep Standard;

    4. Select Start entity.

    Sync changes CDC dialog for src.orders with Start from Full copy selected
    Figure 3. The start dialog of a new entity, with Full copy, Keep and merge and Standard speed.

    Check the target and the slot:

    SELECT * FROM tgt.orders ORDER BY id;
SELECT slot_name, plugin, active FROM pg_replication_slots;
    Expected result

    The target has the three rows. Row 1 keeps {rush,gift}, row 2 keeps the empty array {}, and row 3 keeps NULL for amount and for tags. pg_replication_slots shows one slot named gs_<entity id>_slot with the plugin wal2json and active set to t. In the entity page, select Count both: the source shows 3, the target shows 3 and the difference is 0 with Match.

    Entity page of src.orders in the Active state with the end-to-end latency
    Figure 4. The entity is active. Count both shows 3 rows on each side.
    Follow live changes and check the three operations

    SQL

    On the source, insert, update and delete one row each, with an array in the insert and in the update:

    INSERT INTO src.orders VALUES (4, 'Umbrella', 5.00, '{a,b}', now());
UPDATE src.orders SET amount = 11.25, tags = '{rush,gift,vip}' WHERE id = 1;
DELETE FROM src.orders WHERE id = 2;

    After a few seconds, read the target:

    SELECT id, customer, amount, tags FROM tgt.orders ORDER BY id;
    Expected result

    Row 1 shows 11.25 and {rush,gift,vip}, row 4 exists with {a,b}, and row 2 is gone. Row 3 is unchanged.

    Pause and resume the entity

    UI + SQL

    Pausing an entity keeps its slot, so the source keeps the log that the entity has not read. Resuming with Where it stopped continues from the last checkpoint.

    1. Select the pause button of the entity, then Pause entity. The entity goes to Hold;

    2. On the source, insert a row;

    3. Select the play button, keep Where it stopped and select Start entity.

    Expected result

    The slot stays on the server during the pause, and the row inserted during the pause reaches the target after the resume. In the test, a second row inserted during another pause also arrived, and the counts of source and target matched. A slot that nothing reads keeps log files on the cluster, so delete the entities and the slots you no longer use.

    In the test the server refused to drop the slot of a paused entity (ERROR: replication slot "gs_<entity id>_slot" is active), so a slot cannot be dropped by mistake during a pause. Do not rely on it for a long pause: check pg_replication_slots before a resume.
    Tune the speed

    UI

    The YugabyteDB target has no bulk load: in the start dialog, Speed → Custom shows Write method Standard with the badge This target has no bulk load. The levers are the number of readers and writers and the size of a batch.

    1. In the start dialog, under Speed select Faster or Custom to raise the parallel reads of the source and the writers of the target. More lanes copy faster and put more load on both databases. Faster sets four reads and four writers;

    2. In the entity page, Entity Settings → Advanced settings holds Max Write Batch Size and Max Delete Batch Size, both 1000 by default. A batch of 2,000 query parameters or more, which means rows times columns, is written through a temporary table that is merged into the target table. Smaller batches are merged with one statement;

    3. For the source, the Query parallelism of the snapshot is 4 by default, see YugabyteDB source advanced settings. See also YugabyteDB target advanced settings;

    4. If a row with the same key is already there stays on Upsert: Gluesync updates the rows that exist and inserts the others.

    In the test, a table of 200,000 rows with three columns (integer, text, numeric(12,2)) copied in 20.1 seconds with Standard, about 9.9 thousand rows per second. The same copy with Faster took 21.8 seconds. The lab was one YugabyteDB node that was both source and target, so the server was the limit, not the number of lanes. Test the setting on your own cluster, and watch the load of the nodes. See Tune replication throughput.

    Troubleshooting

    Symptom or message Cause Fix

    The entity could not be saved. FATAL: must be superuser or replication role to start walsender, with the notification Entity Upsert Failed

    The source user has no REPLICATION attribute. The connection test passed, because it signs in only. The failure comes when you add the entity and Gluesync opens the replication connection.

    Run ALTER ROLE gluesync_src REPLICATION; and add the entity again.

    The entity could not be saved. ERROR: all replication slots are in use, with the hint Free one or increase max_replication_slots.

    The server has no free slot for a new entity.

    List the slots with SELECT slot_name, active FROM pg_replication_slots;. Drop the ones that nothing uses with SELECT pg_drop_replication_slot('<name>');, or raise the limit on the cluster and restart. A slot that a session just created can stay active for a while: wait and retry.

    Failed to initialize pool: FATAL: database "nodb" does not exist, with the notification Host Credentials Update Failed

    The Database name of the connection is wrong.

    Correct the name in Base settings and select Test and continue.

    Target write failed, SQL Error: Code 0, State 42501, Message ERROR: permission denied for table orders2

    The target user cannot write the table, because another role owns it and the user has no grant.

    Run GRANT SELECT, INSERT, UPDATE, DELETE ON tgt.orders2 TO gluesync_tgt;. Gluesync retries the write by itself: in the test, the row arrived without a restart.

    Target write failed, SQL Error: Code 0, State 42501, Message ERROR: permission denied to create temporary tables in database "demo"

    TEMPORARY was revoked from PUBLIC and the target user does not have it. Inserts of a snapshot still work. A batch of 2,000 query parameters or more, for example an update of 1,000 rows of three columns, is written through a temporary table and fails.

    Run GRANT TEMPORARY ON DATABASE demo TO gluesync_tgt;. Gluesync retries, and the rows arrived in the test without a restart.

    The Field mapping dialog shows Error (3) for a new target table: array in red as the source type, Choose an option as the target type of an array column, and a length 0 on a text column

    The target type list of the agent has no array type, and a new text column needs a length.

    Create the table yourself with the array column, as in step 2, and map the entity to it: the dialog then shows no error. Keep the same element type on both sides.

    The cluster keeps log files and its disk grows

    A slot is not advancing: the entity is paused, in error, the target is slow, or a slot is left from a deleted pipeline.

    Check pg_replication_slots. Resume or fix the entity, or drop the slot that nothing reads.
    Add entities page with the error The entity could not be saved and the notification Entity Upsert Failed with the message all replication slots are in use and the hint Free one or increase max_replication_slots
    Figure 5. An entity that cannot be saved because the server has no free replication slot.

    Related guides