Search docs.molo17.com
    Configuration guide Connectors

    Set up an Elasticsearch or OpenSearch target

    Elasticsearch and OpenSearch are targets of the same Gluesync agent. Gluesync writes each source row as one document of an index, over the HTTP _bulk API, for the snapshot and for live changes. This guide shows how to prepare the cluster and a user with the rights Gluesync needs, create the connection and the pipeline, copy a PostgreSQL table, check the documents and the mapping, and recognize the errors you get when the address, the credentials or the rights are wrong.

    In this guide

    You will create a role and an API key on Elasticsearch, connect Gluesync with a user name and password or with the API key, copy a table with a snapshot and CDC, read the documents back with the search API, see how the index name and the field types are chosen, and tune the index name, the replicas and the bulk size. The same steps connect OpenSearch. A symptom, cause and fix table closes the guide.

    Quick information

    Goal

    Replicate a relational table to an Elasticsearch or OpenSearch index and confirm inserts, updates and deletes

    Audience

    Data engineers and cluster administrators

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Control Plane → Data pipelines → Entities; Elasticsearch → Dev Tools or curl

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a source that Core Hub can read with CDC. This guide uses PostgreSQL with wal_level = logical (see PostgreSQL CDC with WAL) and a table shop.orders;

    2. You have an Elasticsearch or OpenSearch cluster that Core Hub can reach over HTTP or HTTPS, on port 9200 by default. This guide was tested with Elasticsearch 8.15 with security on, and OpenSearch 2.17 with the security plugin off;

    3. You can sign in to the Control Plane with a role that can create pipelines, and you can call the cluster API as an administrator;

    4. On Linux, the host of a self-managed cluster needs vm.max_map_count of at least 262144. If it is lower, the cluster process stops during start.

    Gluesync only writes to the cluster. The agent has no source role, so it cannot read changes from Elasticsearch or OpenSearch. See Elasticsearch agent for Gluesync for what is supported.

    Guide steps

    Create the example source table

    SQL

    Skip this step if you already have a table to copy. It creates a table with the types that matter for a document store.

    1. On the source PostgreSQL database, run:

      CREATE SCHEMA shop;
      CREATE TABLE shop.orders (
        id integer PRIMARY KEY, customer varchar(100) NOT NULL, status varchar(20),
        amount numeric(12,2), quantity integer, paid boolean,
        created_at timestamp, shipped_at timestamptz, ship_date date,
        note text, meta jsonb, tags text[]);
      INSERT INTO shop.orders VALUES
       (1,'Acme Spa','new',1500.50,3,true,'2026-11-01 10:00:00',
        '2026-11-02 08:30:00+01','2026-11-02','first order','{"tier":"gold"}','{a,b}'),
       (2,'Globex','paid',99.99,1,false,'2026-11-01 11:15:00',NULL,NULL,NULL,NULL,NULL);
    Expected result

    SELECT count(*) FROM shop.orders returns 2.

    Give Gluesync a role on the cluster

    CLI

    Gluesync needs to bulk-index, update and delete documents, to create an index when it is missing, and to list the indexes and read their mappings when you add an entity. The role below worked in our test. Run the calls as an administrator, for example in Kibana Dev Tools or with curl.

    1. Create the role. It covers the indexes whose names start with shop-, and the index name orders, because the entity picker reads the mapping of an index named like the source table before you choose the target name:

      PUT _security/role/gluesync_writer
      {
        "cluster": ["monitor"],
        "indices": [{
          "names": ["shop-*", "orders"],
          "privileges": ["create_index", "write", "delete", "view_index_metadata", "read", "monitor"]
        }]
      }
    2. Create a user for Gluesync:

      PUT _security/user/gluesync
      { "password": "<a strong password>", "roles": ["gluesync_writer"] }
    3. Or create an API key instead of a user:

      POST _security/api_key
      {
        "name": "gluesync",
        "role_descriptors": {
          "gluesync_writer": {
            "cluster": ["monitor"],
            "indices": [{
              "names": ["shop-*", "orders"],
              "privileges": ["create_index", "write", "delete", "view_index_metadata", "read", "monitor"]
            }]
          }
        }
      }

      Keep the value of encoded from the answer. You will paste it into the Password field.

    Expected result

    The answers are {"role":{"created":true}} and {"created":true} for the user, or an object with id, api_key and encoded for the key.

    OpenSearch has its own security plugin and role format. Create a role there with the same rights: write, delete and create index on the indexes that Gluesync uses. The calls above are Elasticsearch calls and were not run against OpenSearch.

    Create the pipeline and the connection

    UI

    1. Open Data pipelines;

    2. Select the + button (Create pipeline), or Create your first pipeline on a new installation. The Agents setup wizard opens on 1. Setup source;

    3. Select the PostgreSQL source and enter its connection, then select Test and continue and Next;

    4. On 2. Setup target(s), search for elastic and select the Elasticsearch agent tile under NoSQL. The form that opens is titled Elasticsearch / OpenSearch;

      Setup target wizard step with the search field filled with elastic and the Elasticsearch agent tile under NoSQL
    5. Enter Node address (a node of the cluster) and Port (9200);

    6. For a user, enter Username and Password. For an API key, leave Username empty, paste the encoded value in Password and set Authentication mode to API_KEY. The other values of Authentication mode are BASIC (the default) and BEARER. Turn on Disable auth only for a cluster that allows anonymous access;

      Elasticsearch / OpenSearch connection form with the node address
    7. Turn on Enable TLS when the cluster answers on HTTPS. Under Upload 'certificate store' file, select Upload files or Use a path when the cluster uses its own certificate authority;

    8. Optional: set Server flavor to ELASTICSEARCH or OPENSEARCH. The default AUTO recognized both clusters in our test. Under Advanced connection properties, Add property offers Path prefix, Connect timeout (seconds), Request timeout (seconds) and Strict host name check. Additional nodes takes more nodes of the same cluster;

    9. Select Test and continue, then Next;

    10. On 3. Ready, enter a Pipeline name and select Finish.

    Expected result

    The target card shows a green check mark and the connection (<user>@<host>:9200, or @<host>:9200 for an API key), and the pipeline opens with the message No entities found. The test calls GET / on the node, so a wrong address, a wrong password or a TLS mismatch fails here, with the message in a Failed to save banner of the form (see the troubleshooting table). A cluster with security off, such as the OpenSearch of our test, passes with empty credentials and without Disable auth.

    localhost in the connection means the machine that runs Core Hub. If Core Hub runs in a container, enter the name that the container can resolve, for example the service name of the cluster in the same Docker network.

    Choose the table and the index

    UI

    In the field mapping, the target Target Index namespace is an optional name part of the index, and the target Target Index is the index itself.

    1. In the pipeline, select Entities. The Add entities page opens;

    2. In the Object browser, expand the schema shop and tick orders;

    3. Select Confirm and continue. The entity shop.orders is in the Hold group;

    4. Optional: open the three-dot menu of the entity and select Edit mapping. The Field mapping dialog fills Target Index namespace with shop and Target Index with orders, links every field, and shows the field types that Gluesync will create. Select Confirm, then Confirm and continue.

    Field mapping dialog with the source schema shop and table orders on the left
    Expected result

    The entity shop.orders is in the Hold group, and the cluster has no index yet: Gluesync creates the index when the entity starts, so a cluster that refuses the creation fails in the next step (see the troubleshooting table). The entity picker needs the monitor and view_index_metadata rights of step 2: without them a Tables Fetch or Tables Columns Fetch notification appears and Confirm and continue stays disabled.

    The index name is the prefix, the schema and the table joined with hyphens, in lower case. With no prefix and the schema shop, the index of orders is shop-orders. Each document gets the identifier shop::orders::<key>. See Index names for the rules and for how to change them.

    Start the snapshot and CDC

    UI

    1. In the row of the entity, select the start icon (Start). The dialog Sync changes (CDC) opens;

    2. In Start from, keep Full copy (marked Recommended for the first start) and keep Keep and merge under Data already in the target;

    3. Select Start entity.

    Expected result

    The entity moves to the Active group, the notifications Snapshot finished for shop.orders and CDC started for shop.orders appear, and the pipeline shows the Healthy group with a green check mark.

    Pipeline page with the Healthy pipeline in the list and the entity shop.orders in the Active group with its last rows
    Verify the documents and the mapping

    CLI

    1. Count and read the documents:

      curl -s 'http://<node>:9200/shop-orders/_count'
      curl -s 'http://<node>:9200/shop-orders/_search?sort=id&pretty'
    2. Read the mapping that Gluesync created:

      curl -s 'http://<node>:9200/shop-orders/_mapping?pretty'
    3. Change the source and read again. These statements update, insert and delete a row:

      UPDATE shop.orders SET status = 'shipped', amount = 1600.75 WHERE id = 2;
      INSERT INTO shop.orders (id, customer, status, amount, quantity, paid, created_at)
        VALUES (3, 'Initech', 'new', 10.10, 2, false, now());
      DELETE FROM shop.orders WHERE id = 1;
    Expected result

    After the snapshot, _count returns 2 and the document shop::orders::1 holds "amount": 1500.5, "created_at": "2026-11-01T10:00:00", "shipped_at": "2026-11-02T07:30:00Z" and "tags": ["a", "b"]. A few seconds after the three statements, the document of id 2 has the status shipped and the amount 1600.75, the document of id 3 exists, and the document of id 1 is gone.

    Optional: change the index name, replicas and bulk size

    UI

    These settings are on the target. Change them before you start the entity: Gluesync creates an index once and leaves an existing index, with its mapping and its replicas, alone. A new prefix on an entity that already runs makes the next write go to a new index and leaves the old index as it is.

    1. Open the gear menu of the pipeline toolbar, select Edit target and open the Advanced settings tab;

    2. Select Add property and pick a key from Key. The keys are Index prefix, Index name template, Insert action, Refresh policy, Truncate strategy, Create index if missing, Explicit mapping, Number of shards, Number of replicas, Document id field, Decimals as strings, Bulk batch size and Bulk request size limit (bytes);

    3. To add a prefix, set Index prefix (for example Acme Shop). The index becomes acme_shop-shop-orders: the name is lower-cased and the space becomes an underscore;

    4. On a one-node cluster, set Number of replicas to 0. The default is 1 shard and 1 replica, and a replica has no node to live on, so the index stays yellow;

    5. To write bigger batches, raise Bulk batch size (default 1000) or Bulk request size limit (bytes) (default 5242880);

    6. Optional: Refresh policy is NONE (default), WAIT_FOR or IMMEDIATE, Insert action is INDEX (default) or CREATE, and Truncate strategy is DELETE_BY_QUERY (default) or SKIP;

    7. Select Test and continue.

    Edit target page on the Advanced settings tab with the Index prefix key set to Acme Shop and the Number of replicas key set to 0

    For one entity, Edit settings has an Entity tab with Index name override, Ingest pipeline and Bulk batch size override.

    Expected result

    GET _cat/indices?v lists the next index created under the new name, and its health is green with 0 replicas on one node.

    Index names

    Gluesync picks the name in this order:

    1. Index name override of the entity, when set. In our test, the value Orders Override/1 gave the index orders_override_1;

    2. otherwise <prefix>-<scope>-<collection>, where the prefix is Index prefix, the scope is Target Index namespace and the collection is Target Index. An empty part is left out.

    The name is then made valid: it is lower-cased, and characters that an index name cannot hold, such as a space or a slash, become an underscore. Index name template in the Advanced settings of the target is a third way to build the name.

    Data type notes

    Explicit mapping is on by default, so Gluesync creates one field per column. What we saw with the PostgreSQL table above:

    Source column Field type Notes

    varchar

    keyword

    Exact match. Use text in the field mapping when you need full-text search.

    text

    text with a keyword sub-field

    The sub-field has ignore_above 256.

    integer, boolean

    integer, boolean

    numeric(12,2)

    double

    Pick scaled_float or keyword in the field mapping when you need exact decimals, or turn on Decimals as strings in the Advanced settings of the target.

    timestamp, date

    date

    Written as ISO-8601 text, for example 2026-11-01T10:00:00.

    timestamptz

    date

    Written in UTC, for example 2026-11-02T07:30:00Z.

    jsonb

    keyword

    The document holds the JSON as a string. Choose object in the field mapping to store it as a nested object.

    text[] (array)

    dynamic, shown by the cluster as text

    The document holds a JSON array. An array has no native mapping type: choose keyword, object or nested in the field mapping to control it.

    NULL

    JSON null

    A partial update with null clears the field.

    The target type list of the field mapping offers keyword, text, boolean, short, byte, integer, long, float, double, scaled_float, date, date_nanos, binary, dynamic, object, nested and flattened. An update of a document that is missing creates it: in our test, deleting a document in the cluster and then updating its source row wrote the whole row again.

    Troubleshooting

    Problem Possible cause What to do

    Test and continue fails with None of the nodes http://<host>:9200 could be reached: <host>: Name or service not known

    The host name does not resolve from Core Hub.

    Use a name or address that Core Hub can resolve. Inside a container, localhost is the container.

    Test and continue fails with None of the nodes …​ could be reached: Failed to connect to …​

    Nothing listens on that port, or a firewall blocks it.

    Check the port (default 9200) with curl from the Core Hub host. Add other nodes in Additional nodes.

    None of the nodes https://<host>:9200 could be reached: Unsupported or unrecognized SSL message

    Enable TLS is on, but the cluster answers in plain HTTP.

    Turn off Enable TLS, or turn on TLS on the cluster. For a private certificate authority, use Upload 'certificate store' file.

    GET / failed with HTTP 401 (security_exception): unable to authenticate user [<user>] for REST request [/]

    The user name or the password is wrong.

    Check them with curl -u <user> http://<node>:9200.

    GET / failed with HTTP 401 (security_exception): missing authentication credentials for REST request [/]

    Disable auth is on, or the credentials are empty, but the cluster requires them.

    Turn off Disable auth and enter Username and Password, or the API key.

    GET / failed with HTTP 401 …​ unable to authenticate with provided credentials and anonymous access is not allowed for this request

    The API key or the token is wrong, or Authentication mode does not match what you entered.

    Use API_KEY with the encoded value in Password and an empty Username, or BEARER with a valid token.

    Tables Fetch notification: GET /_cat/indices failed with HTTP 403 (security_exception): action [indices:monitor/stats] is unauthorized. Confirm and continue stays disabled and the page keeps loading

    The role has no monitor index privilege.

    Add monitor to the index privileges of the role (see step 2), then reload the page.

    Tables Columns Fetch notification: GET /orders/_mapping failed with HTTP 403 (security_exception): action [indices:admin/mappings/get] is unauthorized

    The role does not cover the index that is named like the source table.

    Add the table name to names in the role (see step 2), with view_index_metadata.

    Permission Denied notification: PUT /shop-orders failed with HTTP 403 (security_exception): action [indices:admin/create] is unauthorized. The entity is in the Error group after you start it

    The role has no create_index on that index.

    Add the privilege to the role (see step 2). The entity resumes by itself. You can also create the index yourself and turn off Create index if missing.

    The index is yellow and unassigned_shards is above 0

    The default 1 replica has no second node.

    Set Number of replicas to 0 on a one-node cluster. For an index that exists, use PUT shop-orders/_settings.

    The index has an unexpected name

    The name was lower-cased and normalized, or a prefix or an Index name override applies.

    See Index names. Search GET _cat/indices?v.

    A search on a keyword field with part of a word finds nothing

    varchar columns are mapped as keyword, which matches the whole value only.

    Map the field as text before the first write, or create a new index with the mapping you want.

    Requests to the cluster time out or the cluster answers with HTTP 429 or 5xx

    The cluster is busy or a node is down.

    Lower Bulk batch size or Bulk request size limit (bytes), raise Request timeout (seconds) under Advanced connection properties, and add nodes in Additional nodes.