Search docs.molo17.com
    Configuration guide Connectors

    Set up a Vertica source with epoch-based CDC

    Vertica has no log that a client can read and no row triggers, so Gluesync captures changes another way. Every commit in Vertica has a number, the epoch, and Gluesync reads each table as it was at two epochs and sends the difference. It creates nothing in your database and only runs SELECT statements. This guide prepares Vertica, creates the connection and a pipeline to PostgreSQL, runs a snapshot and live changes, and shows what a TRUNCATE and a table without a key do.

    In this guide

    You will keep the history of the database long enough for CDC to read it, create a read-only login, and connect the Vertica source. You will add entities, copy two tables, change rows in Vertica and read them back on the target, and see a key change and a TRUNCATE arrive. You will also set the snapshot read parallelism and recognize the errors you are most likely to meet. A symptom, cause and fix table closes the guide.

    Quick information

    Goal

    Replicate Vertica tables to PostgreSQL with a parallel snapshot and epoch-based change data capture (CDC), and confirm inserts, updates, deletes and key changes

    Audience

    Data engineers and database administrators who own a Vertica database

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Vertica → SQL; Control Plane → Pipelines → Vertica source → Base settings and Advanced settings; entity → start options

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a Vertica 24.1 database, and you can run SQL on it as dbadmin or another superuser. Vertica is a source here, so you also need a target. This guide was tested with Vertica 24.1 Community Edition and a PostgreSQL 16 target;

    2. Core Hub can open a connection to Vertica on its client port, 5433 by default;

    3. You can sign in to the Control Plane with a role that can create pipelines;

    4. The tables you replicate have a primary key, or you can choose their keys. This guide uses a schema sales with customers (3 rows), orders (103 rows) and audit_log (1 row, no key). The names are examples.

    The messages quoted here were captured in the Control Plane and in the Core Hub log of a Core Hub 2.3 test lab with a Vertica 24.1 container. The agent reference describes the design in detail: Vertica source.

    Guide steps

    Prepare Vertica: history, sessions and a read-only login

    SQL

    CDC reads the database as of an older epoch. Vertica keeps that history only for HistoryRetentionTime seconds, and the default is 0: the Ancient History Mark (AHM) then follows the last good epoch within minutes, and an older epoch cannot be read any more. Connect as dbadmin and run:

    1. Keep the history for a day. Gluesync needs at least one hour (3600):

      ALTER DATABASE DEFAULT SET HistoryRetentionTime = 86400;
    2. Leave room for the sessions of the agent. The snapshot opens one connection for each parallel bucket of a table (four by default), and every entity needs sessions of its own for CDC:

      ALTER DATABASE DEFAULT SET MaxClientSessions = 100;
    3. Create the login that Gluesync uses. It is not a superuser and has no role. Replace the password:

      CREATE USER gluesync IDENTIFIED BY '<a strong password>';
      GRANT USAGE ON SCHEMA public TO gluesync;
      GRANT USAGE ON SCHEMA sales TO gluesync;
      GRANT SELECT ON SCHEMA sales TO gluesync;
      -- Tables that are created later in the schema inherit the grants of the schema
      ALTER SCHEMA sales DEFAULT INCLUDE PRIVILEGES;

      USAGE on public is for the geospatial functions that the agent uses to read GEOMETRY and GEOGRAPHY columns. The agent reference lists it as a prerequisite.

    4. Check the setting and the epochs:

      SELECT parameter_name, current_value
      FROM configuration_parameters
      WHERE parameter_name IN ('HistoryRetentionTime', 'HistoryRetentionEpochs');
      
      SELECT current_epoch, ahm_epoch, last_good_epoch FROM v_monitor.system;
    Expected result

    The first query returns 86400 for HistoryRetentionTime and -1 for HistoryRetentionEpochs, which means that only the time limit applies. In the second query ahm_epoch stays far below current_epoch instead of following it, so the epochs that CDC needs stay readable.

    The schema sales is an example. Repeat the GRANT USAGE and GRANT SELECT statements for every schema you replicate. A schema the login cannot use does not appear in the Control Plane, see the troubleshooting table.

    Create the pipeline with the Vertica source and the target

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

    2. In 1. Setup source, search for vertica and choose the Vertica agent tile. Fill Base settings:

      • Host name / Address: the Vertica server, as Core Hub reaches it;

      • Port: 5433, or the client port of your server;

      • Database name: the database to read;

      • Username and Password: the login of step 1;

    3. Select Test and continue. The source appears as a card with the Vertica logo and <user>@<host>:5433. Select Next;

    4. In 2. Setup target(s), search for postgres and choose the PostgreSQL CDC agent tile. Fill the same five fields for your PostgreSQL database and select Test and continue, then Next;

    5. In 3. Ready, enter a Pipeline name and select Finish.

    Agents setup wizard step Setup source with the Vertica agent and the Base settings fields Host name
    Figure 1. The Vertica source connection in the wizard: host, port, database and login.

    The Base settings tab has more options below the login:

    • Enable TLS and Trust server certificate under Security;

    • Connection string, Min connections count (at least 3) and Max connections count under Settings. The defaults are 5 and 25;

    • Advanced connection properties, with the keys Use connection pool, Gluesync schema name (necessary for bulk operations) and Disable automatic schema creation. The last two belong to the Vertica target.

    Expected result

    The pipeline appears in the HEALTHY group of the pipeline list, with no entities yet. If Test and continue fails, the message names the cause. A wrong password or user gives Authentication failed for username, see the troubleshooting table.

    The same connection through the REST API is in the agent reference.

    Add entities and let Gluesync create the target tables

    UI

    1. In the pipeline, select Entities. The Add entities page opens with the Object browser. It lists the schemas that the login can use. Expand sales;

    2. Select the table orders. The Field mapping dialog opens;

    3. Under Target, choose the Target Schema public. In Target Table type orders and choose Create table "orders". The dialog shows New table orders will be created on confirm. and the proposed target types. The key column id is marked as a key: the dialog says that the table has keys inherited from the primary key of orders;

    4. Select Confirm. The Create table orders dialog shows the Read-only SQL statement. Select Create table orders to run it on PostgreSQL;

    5. Select the table customers and repeat steps 3 and 4;

    6. Select Confirm and continue.

    Field mapping dialog with source schema sales and table orders
    Figure 2. The Field mapping dialog for orders: Vertica types on the left, PostgreSQL types on the right, and id as the key.

    The statement for orders has this form:

    CREATE TABLE "public"."orders" (
       "id" bigint,
       "customer_id" bigint,
       "amount" numeric(10, 2),
       "status" varchar(20),
       "placed_at" timestamptz,
       "note" varchar(2000),
       PRIMARY KEY ("id")
    )
    Expected result

    The two entities appear in the pipeline on Hold as sales.customers and sales.orders. Adding an entity also runs the agent setup checks, including Check Epoch History Retention. If the history of the database is shorter than one hour, the page shows The entity could not be saved. with the remedy, see the troubleshooting table.

    A table with no primary key has no key column to inherit. In the Field mapping dialog the red banner There are multiple errors. Click View errors to see them all. shows and Confirm stays disabled. Select the key checkbox of the column or columns that identify a row, in the last column of the source field list, and Confirm works. CDC matches changes by key, so a key is the price of CDC on Vertica: Vertica does not have to enforce a primary key, and many tables have none. See Choose keys for tables without a primary key.

    See Target table creation for the dialog.

    Run the snapshot and check the rows

    UI + SQL

    The snapshot reads each table AT EPOCH of one epoch, pinned when it starts, in parallel hash buckets. Rows that applications write while it runs are not lost: CDC delivers them afterwards.

    1. In the pipeline, find the entity sales.orders on Hold. Select the start options button, the one next to the play button. The Snapshot (copy-only) dialog opens;

    2. Keep Keep and merge under Data already in the target. It overwrites rows with the same key and keeps the others. Keep the Standard speed, which uses one read and two writers;

    3. Select Start snapshot. Do the same for sales.customers.

    Then count the rows on both sides. On Vertica:

    SELECT 'orders', count(*) FROM sales.orders
    UNION ALL SELECT 'customers', count(*) FROM sales.customers;

    On PostgreSQL:

    SELECT 'orders', count(*) FROM public.orders
    UNION ALL SELECT 'customers', count(*) FROM public.customers;

    The Core Hub log shows how the table was read, at INFO level:

    docker logs <core hub container> 2>&1 | grep -E "read in key order|Read completed"
    Expected result

    Both queries return 103 and 3. The log shows Snapshot of entity sales.orders read in key order, it can be continued: primary key [id], entity keys [id] and then four lines such as Read completed 100.00% in 0.19s for sales.orders and bucket 4, one for each bucket. The entities stay on Hold: a copy-only snapshot does not start CDC.

    Compare the counts every time. If the target has fewer rows than the source, run the snapshot of that entity again with Keep and merge. Rows with the same key are overwritten, so a second run adds the missing rows and does not duplicate any.

    The placed_at column of orders is a timestamptz. The agent reads it converted to UTC, and PostgreSQL stores it as timestamp with time zone.

    Start CDC and follow inserts, updates and deletes

    UI + SQL

    1. Select the play button of sales.orders. The Sync changes (CDC) dialog opens. Where it stopped is selected, because the snapshot left a checkpoint. Select Start entity. Do the same for sales.customers;

    2. On Vertica, as the owner of the tables, insert, update and delete one row each:

      INSERT INTO sales.orders (id, customer_id, amount, status, placed_at, note)
      VALUES (1001, 2, 99.90, 'NEW', '2026-11-04 10:00:00+00', 'cdc insert');
      COMMIT;
      
      UPDATE sales.orders SET status = 'PAID', amount = 11.00 WHERE id = 1;
      DELETE FROM sales.orders WHERE id = 2;
      COMMIT;
    3. After a few seconds, read the three keys on PostgreSQL:

      SELECT id, status, amount, note FROM public.orders WHERE id IN (1, 2, 1001) ORDER BY id;
      SELECT count(*) FROM public.orders;
    Sync changes (CDC) dialog for sales.orders with Start from Where it stopped selected
    Figure 3. The Sync changes (CDC) dialog. After a snapshot it offers to continue from where the entity stopped.
    Expected result

    The first query returns the row 1 with status PAID and amount 11.00, and the row 1001, and no row 2. The count is 103 again: one row added, one deleted. In the pipeline both entities are in the Active group.

    CDC polls the database at most once for each polling interval of the entity (500 ms by default) and delivers the net change of each window. A row that changes twice inside one window arrives once, with its last values.

    Change a key and truncate a table

    SQL

    Two operations behave differently from a log-based source.

    1. Change a key. On Vertica run:

      UPDATE sales.customers SET id = 30 WHERE id = 3;
      COMMIT;

      On PostgreSQL, SELECT id, name FROM public.customers ORDER BY id; returns the rows 1, 2 and 30: the old key was deleted and the new one inserted;

    2. Truncate a table. TRUNCATE removes the history of the table together with its rows, so CDC cannot compare epochs any more. On Vertica run:

      TRUNCATE TABLE sales.orders;
      INSERT INTO sales.orders (id, customer_id, amount, status, placed_at, note)
      VALUES (2001, 1, 1.00, 'NEW', '2026-11-04 13:00:00+00', 'after truncate'),
             (2002, 2, 2.00, 'NEW', '2026-11-04 13:00:00+00', 'after truncate');
      COMMIT;
    3. After a few seconds read SELECT id FROM public.orders ORDER BY id; on PostgreSQL and look for the warning in the Core Hub log:

      docker logs <core hub container> 2>&1 | grep "cannot be read as changes"
    Expected result

    The target table holds only 2001 and 2002. The log shows a warning such as Entity sales.orders cannot be read as changes from its last position: the table held 0 rows at epoch 18, not the 104 it was left with (it was truncated or its history was purged). The table is reloaded as of epoch 20. The agent emptied the target table once and reloaded every row of the source.

    Each TRUNCATE on a replicated table makes the agent reload that table, and the target is short of rows until the reload completes. If a job truncates the table every day, delete rows instead. The setting Reload the table when its history is gone, instead of failing turns the reload off: the entity then fails with an error and you reload it with a snapshot. See When the history is lost.

    Tune the snapshot read

    UI

    The snapshot reads a table in hash buckets, and each bucket has its own Vertica session. The default is four. To change it:

    1. In the pipeline list, select the Vertica icon of the pipeline. The Edit source page opens. Select Advanced settings;

    2. Select Add property and open the Key list. It offers these settings:

      • Source change retention in hours (not used by Vertica);

      • Technical fields;

      • Snapshot read parallelism (parallel buckets per table, 1 disables);

      • JDBC result buffer size in bytes;

      • Delete detection bucket size (removed keys held in memory per bucket);

      • Reload the table when its history is gone, instead of failing;

      • Minimum history retention in seconds required by the setup check;

    3. Choose Snapshot read parallelism. The Value starts at 4. Enter 8 and select Test and continue.

    Edit source page with the Advanced settings tab and the property Snapshot read parallelism with value 8
    Figure 4. The Advanced settings of the Vertica source with the snapshot read parallelism set to 8.
    Expected result

    The pipeline goes on running. The next snapshot reads each table in the number of buckets you set: the log at DEBUG level shows Migration of sales.payments of 100 rows starting in 8 buckets. Keep MaxClientSessions above the sessions that all pipelines use at the same time, plus the parallelism of each table that is read at once.

    For a large table, the options of the snapshot dialog matter as much as the source. Under Speed, Custom shows Read in parallel, which splits the table into ranges when you define partitions, and the number of writers, from 1 to 10. The write method of the target, Standard or Bulk load, is a separate switch for the copy and for live changes. Read Tune replication throughput and Filter rows with snapshot WHERE and target filters.

    Fix the errors you are most likely to meet

    SQL + UI

    Each error below was reproduced on the test lab.

    1. Set HistoryRetentionTime to 60, add an entity and select Confirm and continue. The page shows The entity could not be saved. and the notification Entity Upsert Failed with The history of the database spans 60 seconds (HistoryRetentionTime), too little to read changes from: Gluesync needs at least 3600. A superuser sets it with ALTER DATABASE DEFAULT SET HistoryRetentionTime = 3600; Fix: run the statement of step 1;

    2. Add a table with an array column, such as ARRAY[VARCHAR(20)], and let Gluesync create the PostgreSQL table. The Field mapping dialog proposes the type array, and Create table fails with Create target table execution error: ERROR: syntax error at or near "array". Fix: create the table yourself with the array type, tags text[], and choose it in Target Table. The mapping then shows array on both sides and the values arrive as {x,y};

    3. Change the password in Edit source to a wrong value and select Test and continue. The page shows Failed to save: The connection was saved, but agent <id> of pipeline <id> did not accept the new credentials: Failed to initialize pool: [Vertica][VJDBC](2248) FATAL: Authentication failed for username "gluesync". The wrong password is stored. Fix: enter the right one and select Test and continue again. The pipeline recovers by itself.

    Expected result

    After each fix the entity saves, the table is created or the credentials are accepted, and the pipeline returns to HEALTHY.

    Data types

    Vertica reports every integer type as int and every float type as float, both 64 bit. In the Field mapping dialog for a PostgreSQL target:

    Vertica type PostgreSQL type proposed Note

    int

    bigint

    64 bit

    numeric(10,2)

    numeric(10,2)

    The precision and the scale are kept

    varchar(20), long varchar(2000)

    varchar(20), varchar(2000)

    The length is kept

    timestamp

    timestamp(6) without time zone

    timestamptz

    timestamp with time zone, precision 6

    Read at UTC

    array[…​]

    array

    Create the target table yourself, with a type such as text[]

    The agent reference lists the mapping of every Vertica type, including intervals, geospatial types, ROW and SET: Data types.

    Verify the result

    You have set up the Vertica source if:

    • HistoryRetentionTime is at least 3600, and MaxClientSessions leaves room for the parallel buckets;

    • The login can SELECT from the schemas and has USAGE on them and on public;

    • The pipeline is HEALTHY and every entity has a key;

    • The row counts of the snapshot are equal on both sides;

    • A CDC insert, update, delete and key change reach the target;

    • You know that a TRUNCATE reloads the table.

    Troubleshooting

    Problem Possible cause What to do

    The entity could not be saved. with The history of the database spans <n> seconds (HistoryRetentionTime), too little to read changes from

    HistoryRetentionTime is below the minimum of the setup check (3600 seconds)

    As a superuser run ALTER DATABASE DEFAULT SET HistoryRetentionTime = 86400; and save the entity again

    Failed to save with Authentication failed for username

    The user or the password is wrong. The connection is saved anyway

    Enter the right credentials and select Test and continue again

    A schema or a table is missing from the Object browser

    The login has no USAGE on the schema, or no SELECT on the table

    Run GRANT USAGE ON SCHEMA <schema> TO <user>; and GRANT SELECT ON SCHEMA <schema> TO <user>;, then open Add entities again

    Confirm is disabled and the banner says There are multiple errors, for a table without a primary key

    CDC needs a key and Vertica does not require one

    Select the key checkbox of the columns that identify a row in the Field mapping dialog

    Failed to create table with ERROR: syntax error at or near "array"

    The table has an array column and the proposed PostgreSQL type is array, which PostgreSQL does not accept

    Create the target table with an array type such as text[] and choose it in Target Table

    The target has fewer rows than the source after the snapshot

    The snapshot did not deliver every bucket

    Run the snapshot of the entity again with Keep and merge, then compare the counts again

    The log shows cannot be read as changes from its last position and the target table was emptied and filled again

    A TRUNCATE, a purge or a long pause removed the history that CDC compares. The agent reloads the table

    Expected behavior. Avoid truncating replicated tables, or keep HistoryRetentionTime long enough for the longest pause of the pipeline

    Contact MOLO17 Support if you need assistance with a Vertica source.