Search docs.molo17.com
    Configuration guide Connectors

    Set up a GridGain target with the thin client

    GridGain keeps its data in caches, and its SQL layer shows a cache as a table. The Gluesync GridGain agent talks to that SQL layer over the thin client: it reads the catalog, and it writes each batch of changes with one multi-row MERGE INTO statement. This guide prepares a GridGain 8 cluster, creates a user, creates the target table, connects Gluesync, copies a PostgreSQL table with a snapshot and live changes, secures the connection, and explains the settings and the errors that you are most likely to meet.

    In this guide

    You will activate a cluster that uses native persistence, turn authentication on and create a user for Gluesync, and create the target table with the cache settings that you want. You will create the GridGain target connection, map a PostgreSQL table to the table, run a snapshot and CDC, and check the rows with SQL. You will also connect over TLS, read how the write batch settings work, and recover from the failures that are most likely on a new setup.

    Quick information

    Goal

    Replicate a PostgreSQL table to a GridGain SQL table with a snapshot, then keep it in sync with change data capture (CDC)

    Audience

    Data engineers and GridGain or Apache Ignite administrators

    Level

    Intermediate

    Time required

    About 60 minutes

    Area

    GridGain → SQL and control utility; Control Plane → Pipelines → GridGain target → Base settings and Advanced settings; entity → Edit settings

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a GridGain 8 cluster. This guide was tested with GridGain Community Edition 8.9, one node, native persistence on, authentication on. The agent reference lists GridGain 8.8 and later in GridGain agent for Gluesync. GridGain is a target only: you need another database as the source;

    2. Core Hub can open a connection to the thin client port of a node, 10800 by default;

    3. You can run SQL on the cluster as an administrator, with sqlline or any JDBC tool, and you can run the control utility control.sh on a node;

    4. You can sign in to the Control Plane with a role that can create pipelines;

    5. You have a source that Core Hub can read with CDC. This guide uses PostgreSQL with wal_level = logical, prepared as in Set up a PostgreSQL WAL source, and a table shop.orders with 5,000 rows: id bigint primary key, customer varchar(40), amount numeric(10,2), qty integer, paid boolean, created_at timestamp, and note text.

    The messages quoted in this guide were captured in the Control Plane and in the Core Hub log of Gluesync 2.3, against GridGain Community Edition 8.9. A cluster that is not on native persistence starts active, so step 1 does not apply to it.

    Guide steps

    Activate the cluster

    CLI

    A node that uses native persistence starts in the state INACTIVE, and it refuses every SQL statement until you activate the cluster. This is the usual first failure of a new setup: the connection test of the Control Plane fails with the message shown in the troubleshooting table.

    1. Check that the thin client port is enabled in the node configuration. The clientConnectorConfiguration bean of the lab node is:

      <property name="clientConnectorConfiguration">
    <bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
        <property name="host" value="0.0.0.0"/>
        <property name="port" value="10800"/>
        <property name="thinClientEnabled" value="true"/>
        <property name="jdbcEnabled" value="true"/>
    </bean>
</property>

    2. Activate the cluster once, from any node:

      /opt/gridgain/bin/control.sh --set-state ACTIVE --yes

      When authentication is on (step 2), add --user and --password, or the utility waits for them at a prompt.

    Expected result

    The utility prints Cluster state changed to ACTIVE. A cluster that you re-create from empty storage starts inactive again, and you activate it again.

    Turn authentication on and create the Gluesync user

    SQL

    1. Turn authentication on in the node configuration, next to the native persistence settings, and restart the node:

      <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="authenticationEnabled" value="true"/>
    ...
</bean>

    2. Activate the cluster as in step 1, with --user ignite --password ignite. These are the credentials of the default administrator.

    3. Connect as that administrator, change its password, and create the user that Gluesync uses. Quote the user name, so that GridGain stores it exactly as gluesync, the way you type it in the form.

      ALTER USER "ignite" WITH PASSWORD '<a strong password>';
CREATE USER "gluesync" WITH PASSWORD '<a strong password>';
    Expected result

    Both statements run without an error, and a connection with the new user and password opens. In our lab the user gluesync wrote to a table that the administrator had created, and we granted nothing.

    Create the target table

    SQL

    Create the table yourself, with unquoted names, before you map the entity. The Gluesync MERGE INTO statement writes the column names without quotes, so GridGain looks for them in upper case. A table that has lower-case column names, like the one that the Create table option of the field mapping dialog proposes, fails at the first write (see the troubleshooting table).

    1. Create the table and choose its cache settings in the WITH clause:

      CREATE TABLE PUBLIC.ORDERS (
  ID BIGINT PRIMARY KEY,
  CUSTOMER VARCHAR(40),
  AMOUNT DECIMAL(10,2),
  QTY INT,
  PAID BOOLEAN,
  CREATED_AT TIMESTAMP,
  NOTE VARCHAR(1000)
) WITH "CACHE_NAME=orders,TEMPLATE=PARTITIONED,BACKUPS=1";

      CACHE_NAME is the name of the cache that stores the rows. Without it, GridGain names the cache SQL_PUBLIC_ORDERS. TEMPLATE=PARTITIONED spreads the rows over the nodes, and BACKUPS=1 keeps one copy of each partition on another node. The primary key of the table is the key that Gluesync uses to find a row for an update and a delete, and it must be the same set of columns as the primary key of the source.

    2. Look at the result in the catalog. This is the query that Gluesync uses to list the tables, so a cache that has no SQL table does not appear in the Control Plane:

      SELECT TABLE_NAME, CACHE_NAME FROM SYS.TABLES WHERE SCHEMA_NAME = 'PUBLIC';
    Expected result

    The query returns one row, ORDERS with the cache name orders.

    Create the pipeline and the GridGain target connection

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

    2. In 1. Setup source, choose the PostgreSQL CDC agent tile and fill Host name / Address, Port, Database name, Username and Password. Select Test and continue, then Next;

    3. In 2. Setup target(s), search for gridgain and select the GridGain agent tile under RDBMS. Fill Base settings:

      • Host name / Address: a node of the cluster, as Core Hub sees it;

      • Port: 10800, the default;

      • Database name: a value that the form requires. The connection does not use it to choose a schema: you choose the schema, for example PUBLIC, when you map each entity;

      • Username and Password: the user of step 2;

    4. Keep isThin on. It is the thin client, which the agent reference recommends. Keep Partition Awareness Enabled on, its default: the agent adds partitionAwareness=true to the connection;

    5. Leave Disable auth off for a cluster with authentication. For a cluster without it, turn it on, and still enter a user name and a password: the form refuses an empty value, and the cluster ignores it;

    6. Select Test and continue, then Next, enter a Pipeline name and select Finish.

    Edit target page of the GridGain agent with host
    Figure 1. The GridGain target connection: host, port, database name, user, isThin and Partition Awareness Enabled on, and the Security switches.
    Expected result

    The target card shows a green check mark and <user>@<host>:10800, and the pipeline opens with the message No entities found. The test opens a pool of connections, so a wrong address, a wrong password, an inactive cluster and a TLS mismatch fail here, with the message of the driver in a Failed to save banner. The connection is saved even when the test fails: fix the value and select Test and continue again. A pipeline that shows the banner Error: Unknown error after the fix can need a minute or two to turn healthy.

    When you open Edit target for a target whose first test failed in the wizard, the form can be empty. Fill the host, the port and the other values again before you select Test and continue.
    Add the entity and map it to the table

    UI

    1. In the pipeline, select Entities. The Add entities page opens;

    2. In the Object browser, expand the schema shop and tick orders. The Field mapping dialog opens;

    3. In Target Schema, choose PUBLIC. In Target Table, choose ORDERS;

    4. Check the links. Gluesync links the source fields to the target columns by name, without regard to the case, and takes the key from the primary key of the table. Select Confirm, then Confirm and continue.

    Field mapping dialog with the source schema shop and table orders on the left
    Figure 2. The field mapping of shop.orders to the existing table PUBLIC.ORDERS: every field is linked by name, and the key is ID.
    Expected result

    The entity shop.orders is in the Hold group. The dialog can show the warning Data type mismatch detected for CREATED_AT, because the source type is timestamp without time zone and the target type is timestamp. It is a warning, and Confirm stays available. See Data types for the types that the dialog proposes when you let Gluesync create a table.

    Choose a primary key for tables that have none before you add the entity. Read Choose keys for tables without a primary key and Choose a key strategy.
    Start the snapshot and CDC

    UI

    1. In the row of the entity, select the start icon (Start). The dialog Sync changes (CDC) opens;

    2. In Start from, keep Full copy, marked Recommended for the first start;

    3. In Data already in the target, keep Keep and merge. Rows with the same key are overwritten, and the others stay. Do not choose Empty the table first on GridGain without reading the next paragraph and the troubleshooting table;

    4. Select Start entity.

    Pipeline Orders to GridGain with the Healthy group
    Figure 3. The pipeline after the snapshot: the entity is Active, the pipeline is Healthy, and the last batch of rows is shown.

    Empty the table first sends TRUNCATE TABLE to GridGain, and GridGain 8 has no such statement. The entity goes to the Error group with the notification SQL and the text shown in the troubleshooting table. Empty the table yourself with DELETE FROM PUBLIC.ORDERS; and keep Keep and merge, or turn on the target property that skips the truncate, see Snapshot options.

    Expected result

    The entity moves to the Active group, and the pipeline shows the Healthy group. The 5,000 rows of the lab table were written in a few seconds.

    Check the rows in GridGain

    SQL

    1. Count the rows and compare with the source:

      SELECT COUNT(*) FROM PUBLIC.ORDERS;

    2. Change the source, then read the same rows on the target. On PostgreSQL:

      INSERT INTO shop.orders VALUES (5001, 'new customer', 99.90, 3, true, '2026-11-09 10:15:30.123456', 'inserted');
UPDATE shop.orders SET qty = 777, note = 'updated' WHERE id = 1;
DELETE FROM shop.orders WHERE id = 2;

    3. On GridGain, read the rows after a few seconds. Cast the timestamp to text to see every digit:

      SELECT ID, QTY, NOTE, CAST(CREATED_AT AS VARCHAR) AS TS FROM PUBLIC.ORDERS WHERE ID IN (1, 2, 5001);
    Expected result

    The first count is 5000. After the three changes, the count is still 5000: one row more, one row less. The row 1 shows 777 and updated, the row 2 is gone, and the row 5001 is there with 2026-11-09 10:15:30.123456, so the microseconds arrive intact.

    Connect over TLS

    CLI + UI

    Skip this step for a cluster that accepts plain connections. The thin client needs three things on the server and two on the client:

    1. On the server, create a key store and, for the client, a trust store and a key store. The Ignite thin driver does not initialize without a client key store, even when the server does not ask for a client certificate. In a test setup:

      keytool -genkeypair -alias gridgain -keyalg RSA -keysize 2048 -validity 365 -dname "CN=<node host name>" -ext "SAN=dns:<node host name>" -keystore server.jks -storepass <password> -keypass <password>
keytool -exportcert -alias gridgain -keystore server.jks -storepass <password> -file server.cer
keytool -importcert -noprompt -alias gridgain -file server.cer -keystore trust.jks -storepass <password>
keytool -genkeypair -alias gluesync -keyalg RSA -keysize 2048 -validity 365 -dname "CN=gluesync" -keystore client.jks -storepass <password> -keypass <password>

    2. Turn TLS on for the thin client in the node configuration, with the key store of the server, and restart the node. Replace STORE_PASSWORD with the password of the key store:

      <bean class="org.apache.ignite.configuration.ClientConnectorConfiguration">
    <property name="port" value="10800"/>
    <property name="thinClientEnabled" value="true"/>
    <property name="sslEnabled" value="true"/>
    <property name="useIgniteSslContextFactory" value="false"/>
    <property name="sslContextFactory">
        <bean class="org.apache.ignite.ssl.SslContextFactory">
            <property name="keyStoreFilePath" value="/tls/server.jks"/>
            <property name="keyStorePassword" value="STORE_PASSWORD"/>
        </bean>
    </property>
</bean>

    3. Copy trust.jks and client.jks to a folder that Core Hub can read, for example /opt/gluesync/data/tls/;

    4. In the Control Plane, select the gear of the pipeline, then Edit target. Leave Enable TLS off and paste a Connection string with the plain paths of the two files:

      jdbc:ignite:thin://<node host name>:10800?sslMode=REQUIRE&sslTrustCertificateKeyStoreUrl=/opt/gluesync/data/tls/trust.jks&sslTrustCertificateKeyStorePassword=<password>&sslClientCertificateKeyStoreUrl=/opt/gluesync/data/tls/client.jks&sslClientCertificateKeyStorePassword=<password>&partitionAwareness=true

      The Username and Password of the form still apply. Select Test and continue.

    Expected result

    The target is healthy, and an insert in the source reaches the table over the TLS connection.

    In our test of Gluesync 2.3, Enable TLS with a trust store and a key store, uploaded or given as a path, could not connect to GridGain. The agent adds the prefix file: to the path of a store, and the thin driver does not accept it. The message is the one in the troubleshooting table. The Connection string in this step avoids the prefix. The passwords of the stores are saved in the connection string: limit who can edit pipelines.

    Snapshot options and write batches

    Two groups of settings change how Gluesync writes to GridGain.

    On the target, under Advanced settings of Edit target, Add property offers two keys:

    Advanced settings of the GridGain target with the property Disable truncate on target table before snapshot inserts and its switch
    Figure 4. The two advanced properties of the GridGain target: Cache Template and Disable truncate on target table before snapshot inserts.

    • Disable truncate on target table before snapshot inserts: turn it on to skip the TRUNCATE TABLE statement. With the switch on, a start with Empty the table first runs without the error and does not empty the table. We saw an entity that was in error with the truncate message resume and load all its rows by itself after we turned this property on and selected Test and continue;

    • Cache Template: the cache template name, PARTITIONED by default. We did not see an effect on the table that the field mapping dialog creates, whose statement has no WITH clause. Set the cache options in your own CREATE TABLE statement, as in step 3.

    On the entity, select the three-dot menu, Edit settings, then the Entity tab:

    • Max Write Batch Size: 1000 by default. The number of rows in one MERGE INTO statement;

    • Max Delete Batch Size: 1000 by default. The number of keys in one delete batch.

    Data types

    When you map to an existing table, the target types are the types of your columns. When you let Gluesync create the table, the dialog proposes a type for each source column. The table below is what the dialog proposed for a PostgreSQL source in our test. GridGain has these types: varchar, char, varbinary, timestamp, time, date, numeric, smallint, int, bigint, float, double and boolean.

    PostgreSQL type Proposed GridGain type Note

    smallint, integer, bigint

    smallint, int, bigint

    Values written unchanged

    real, double precision

    float, double

    numeric(12,4)

    numeric with precision 12 and scale 4

    Keep the precision and the scale of the source

    char(5), varchar(20)

    varchar(5), varchar(20)

    text, uuid, jsonb

    varchar, with a length of 0

    The dialog shows Missing required field until you enter a length, for example 1000

    boolean

    boolean

    date

    date

    time

    date

    Change it to time yourself

    timestamp

    time

    Change it to timestamp, with a precision of 6, yourself. With time the date is lost

    timestamptz

    timestamp

    bytea, text[]

    varbinary, with a length of 0

    The dialog asks for a length. An array shows a warning, because it has no matching type

    Set the types in the field mapping dialog before you confirm it, or create the table yourself as in step 3. The dialog also shows the statement that it will run, in a read-only Create table window:

    Create table ORDERS dialog with a read-only CREATE TABLE statement whose column names id
    Figure 5. The Create table window of the field mapping: the column names are quoted and in lower case, which the write statement of the agent does not match.

    Bulk and throughput settings

    Setting Default When to change it

    Max Write Batch Size (entity)

    1000

    The rows of one MERGE INTO statement. In our single-node lab, a snapshot of 105,000 rows took 2.7 seconds with 1000 and 4.2 seconds with 5000, one run each. Keep the default unless a measurement on your cluster shows a gain

    Min connections count and Max connections count (target, Settings)

    5 and 25

    The pool of connections of the agent. Raise the maximum when many entities write at the same time

    Partition Awareness Enabled (target)

    On

    Adds partitionAwareness=true to the connection. We could not measure an effect on a one-node cluster

    Speed in the start dialog

    Standard, one reader and two writers

    More writers copy faster but load GridGain more, see the Tune replication throughput guide

    The Core Hub log shows each write at debug level, for example Insert batch for entity shop.orders from null to null took 155ms for a snapshot batch. Read Read Gluesync logs and diagnose errors to find them.

    Verify the result

    You have set up the GridGain target if:

    • The cluster is active, and the Gluesync user can connect and read the target table;

    • The target table exists with unquoted column names and the same primary key as the source;

    • The target of the pipeline has a green check mark, and the entity is in the Active group;

    • The row count of the target equals the source, and a CDC insert, update and delete reach the table;

    • The timestamp columns keep the date and the microseconds;

    • You chose how to empty the table before a new snapshot: DELETE yourself, or the target property that skips the truncate.

    Troubleshooting

    Symptom Cause Fix

    Test and continue fails with Failed to save: Can not perform the operation because the cluster is inactive. Note, that the cluster is considered inactive by default if Ignite Persistent Store is used to let all the nodes join the cluster. To activate the cluster call Ignite.active(true). A Host Credentials Update Failed notification repeats it

    The cluster uses native persistence and nobody activated it.

    Run control.sh --set-state ACTIVE --yes (step 1), then select Test and continue again.

    The user name and the password fields show Required with Disable auth on

    The form requires both values, even for a cluster without authentication.

    Enter any user name and password. The cluster ignores them.

    Failed to save: …​ Failed to initialize pool: Authentication doesn’t support by remote server[…​err=The user name or password is incorrect [userName=<user>] …​]

    A wrong user or password, or a user that does not exist. The agent sends the user name even when Disable auth is on.

    Create the user (step 2) or fix the password, leave Disable auth off for a cluster with authentication, and select Test and continue again. The pipeline recovers without a restart.

    Failed to save: …​ Failed to initialize pool: Failed to connect to Ignite cluster [url=jdbc:ignite:thin://<host>:10800]

    The node is not reachable, or the cluster uses TLS for the thin client and the connection of Gluesync does not. We reproduced the second case.

    Check the host and the port from Core Hub. For TLS, follow the TLS step.

    Failed to save: …​ Failed to initialize pool: Failed to connect to server [url=jdbc:ignite:thin://<host>:10800], and the Core Hub log shows Ouch! Argument cannot be null: keyStoreFilePath

    Enable TLS has a trust store and no key store. The thin driver needs a key store.

    Add a client key store, as in the TLS step.

    Failed to save: …​ Failed to initialize pool: Failed to initialize key store (key store file was not found): [path=file:/opt/gluesync/data/connections/<id>/key-store.jks, msg=File file:/opt/gluesync/data/connections/<id>/key-store.jks not found]. The file exists

    The agent puts the prefix file: before the path of the store, and the thin driver does not accept it. The same message appears for a store that you give with Use a path.

    Leave Enable TLS off and use the Connection string of the TLS step.

    Notification Syntax Error when the entity starts: Failed to parse query. Column "ID" not found; SQL statement: MERGE INTO PUBLIC.ORDERS (id, customer, amount, …​

    The table has lower-case, quoted column names, for example because the Create table option of the field mapping created it. The statement of the agent uses unquoted names, which GridGain reads as upper case.

    Create the table with unquoted names (step 3) and map the entity to it. In our test the entity resumed and loaded the rows by itself after we replaced the table. To point an entity at another table, pause it, delete it and add it again.

    Notification SQL and the entity in the Error group after a start with Empty the table first: SQL Error: Code 1002, State 0A000, Message Unsupported statement: TRUNCATE TABLE PUBLIC.ORDERS

    GridGain 8 does not support TRUNCATE TABLE.

    Use Keep and merge, empty the table with DELETE FROM, or turn on Disable truncate on target table before snapshot inserts under Advanced settings of the target.

    The field mapping shows Missing required field and Confirm is not available

    A varchar or varbinary column that Gluesync proposes has a length of 0.

    Enter a length for each marked column, for example 1000.

    The target table has a time column where the source has a timestamp

    The dialog proposes time for timestamp.

    Change the Target type to timestamp with a precision of 6, or map to a table that you created.

    The pipeline shows Error: Unknown error. Please check agent gridgain (<id>) logs in order to find more details after you fixed the connection

    The banner stays until the next health check of the agent.

    Wait a minute or two and reload the page. Check the Core Hub log if it stays.

    The Control Plane does not list a table or a cache

    The agent lists only the tables of the SQL catalog (SYS.TABLES). A cache that has no SQL table is not listed.

    Create the table with CREATE TABLE, and select Reload in the object browser.

    Related guides