Elasticsearch glyph Elasticsearch agent

Target Elasticsearch

OpenSearch is a flavor of this same agent (module elasticsearch). There is no second OpenSearch product and no source role.

Prerequisites

  • Tested against Elasticsearch 8.15.5 and OpenSearch 2.17.1. See Supported versions.

  • A cluster that answers the Elasticsearch HTTP JSON API. That is Elasticsearch, OpenSearch, or another product that speaks the same API.

  • Permission to bulk-index, update, and delete documents, to run _delete_by_query when snapshot deletion is left at its default, and to create an index when create-if-missing is on.

  • Credentials that match the authentication mode, or anonymous access when disableAuth is on.

Setup via Web UI

  • Hostname / IP address: DNS name or IP address of one cluster node.

  • Port: defaults to 9200.

  • TLS: off by default. When on, the node URLs use https:// and the server certificate is always validated. There is no trust-server-certificate option.

  • CA certificate path and password: a PKCS12 or JKS store, or a PEM file, holding the cluster CA (http_ca.crt on Elasticsearch, root-ca.pem on OpenSearch). The certificate can also be uploaded through Core Hub. Blank means the JVM default trust store. A cluster that generated its own CA needs this.

  • Authentication: defaults to BASIC (username and password).

    • API_KEY: put id:api_key, or an already encoded key, in the password field. The username is ignored.

    • BEARER: the password field is the token.

    • disableAuth: allows anonymous access.

  • Server flavor: AUTO (default), ELASTICSEARCH, or OPENSEARCH. AUTO calls GET /. version.distribution = opensearch means OpenSearch. An Elasticsearch response has no distribution field. The client does not change with the flavor. The flavor only decides the product name shown with the version in the UI and the logs; the version itself is read from the cluster.

  • Connect timeout (seconds): defaults to 30.

  • Request timeout (seconds): defaults to 120. A timeout is retried as a transient failure.

  • Refresh: defaults to NONE. The confirmed transaction id does not wait for a refresh. It advances when the cluster acknowledges the bulk request, whatever this policy is.

  • Bulk batch: defaults to 1000 documents.

  • Bulk bytes: defaults to 5242880. One document larger than the limit still goes out alone.

  • Decimals as strings: off by default, so decimals are sent as JSON numbers in plain notation with every digit. On, they are sent as strings, which a keyword field keeps verbatim.

  • Document id field: blank means the Gluesync document id is stored only as _id. Otherwise the id is copied into that field of the document too.

  • Additional nodes: host1:port1,host2:port2, tried in order when a node is down. A node without a port takes the main one.

  • Path prefix: the path the cluster answers under when it sits behind a reverse proxy, for example /es.

  • Index prefix and Index name template: both blank by default. See Index names.

  • Insert action: INDEX (default) or CREATE. See Insert.

  • Truncate strategy: DELETE_BY_QUERY (default) or SKIP.

  • Create index if missing, Explicit mapping, Number of shards, and Number of replicas: see Index names.

  • Bulk request size limit (bytes): the Bulk bytes limit above; a request is split when it would exceed it.

Entity settings

In the entity UI, schema is the optional index namespace and table is the index.

  • Index name override: when set, this name is used as the index. See Index names.

  • Ingest pipeline: optional ingest pipeline name sent with the bulk request.

  • Bulk batch size override: 0 means the agent default (1000).

Index names

The index name is chosen in this order:

  1. The entity index name override, when it is set.

  2. Otherwise a template that substitutes {prefix}, {scope}, and {collection}.

  3. Otherwise <prefix>-<scope>-<collection>.

The name is then normalized:

  • It is lowercased.

  • A backslash, slash, asterisk, question mark, double quote, <, >, pipe, space, comma, #, and colon each become an underscore.

  • A leading hyphen, underscore, or plus sign is dropped.

  • An empty name, ., .., or a name longer than 255 bytes is refused.

Create-if-missing defaults to on. New indexes default to 1 shard and 1 replica. Explicit mapping defaults to on and generates one property per column from the data type table below. With it off, the index is created empty and the cluster infers every field. A text field also gets a keyword sub-field (ignore_above 256) so it can still be sorted and aggregated. A scaled_float field uses a scaling factor of 10 to the power of the source column scale, or 2 when the scale is unknown. ARRAY is left out of the explicit mapping unless you pick keyword, object, or nested for the field. ARRAY has no native mapping type of its own.

Create table sends PUT /<index> and leaves an existing index alone, including its mapping. When you discover an existing index, its mapping types are read back through the same table; object, nested, and flattened fields appear as MAP, and fields of a type this agent does not know are skipped.

Writes

_id is the Gluesync document id of the row: by default <collection>::<key>, or <scope>::<collection>::<key> when the entity has a scope. Snapshots and CDC both use POST /_bulk. allowedBulkMode is NONE: there is no separate Gluesync bulk-loader mode.

A _bulk request is not atomic. The cluster can accept some items and reject others. Replay from the last confirmed transaction id writes the same document ids again. The confirmed transaction id advances only after the cluster acknowledges the bulk request. Refresh policy does not gate that acknowledgement.

Insert

The default insert action is INDEX, which overwrites the document with that _id.

CREATE reports a version conflict as a duplicate key, so Core Hub insert-conflict strategies apply. insertConflictSupport is DEPENDS_ON_TARGET only for CREATE.

Update

An update is a bulk update with doc and doc_as_upsert set to true. A JSON null clears that field on a partial update.

Delete

A delete is a bulk delete by _id. A missing document is not a failure.

Truncate and snapshot deletion

The default is DELETE_BY_QUERY with match_all and conflicts=proceed. SKIP leaves the documents in place. The index and its mapping always stay. _delete_by_query on a large index is slow.

Data types

This is the mapping decision for this agent. The first type is what explicit mapping emits. Alternatives are what you can select for that field.

Gluesync type Default mapping Alternatives

STRING

keyword

text

BOOLEAN

boolean

SHORT

short

byte, integer, long

INT

integer

long

LONG

long

FLOAT

float

double. NaN and Infinity are written as strings.

DOUBLE

double

float. NaN and Infinity are written as strings.

BIG_DECIMAL

double

scaled_float, keyword

LOCAL_DATE

date

keyword. Written as 2024-01-31.

LOCAL_DATE_TIME

date

date_nanos, keyword. Written in ISO-8601 without an offset.

OFFSET_DATE_TIME

date

date_nanos, keyword. Written in ISO-8601 with the offset.

LOCAL_TIME

keyword

Written as an ISO time string. A mapping has no time-of-day type.

OFFSET_TIME

keyword

Written as an ISO time string with the offset.

BYTE_ARRAY

binary

Base64

ARRAY

dynamic

keyword, object, or nested. Omitted from the explicit mapping unless one of those three is selected. There is no native ARRAY mapping type.

MAP

object

nested, flattened, dynamic. flattened is an Elasticsearch field type; OpenSearch names its equivalent flat_object.

Temporal values are ISO-8601 strings, which the default date format of both products parses. They never go through the JVM default time zone. null is JSON null, so a partial update clears the field.

HTTP errors

The agent titles cluster and bulk failures as follows.

Condition Title

No node answers

Cluster Unreachable

Connect or request timeout

Request Timeout

HTTP 401

Authentication Failed

HTTP 403

Permission Denied

HTTP 404

Index Not Found

HTTP 400

Request Rejected

HTTP 429

Cluster Overloaded

HTTP 5xx

Cluster Error

Bulk request whose only failures are create version conflicts

Document Already Exists

Other rejected bulk items

Bulk Items Rejected

Limitations

  • No source role, and none is planned.

  • A _bulk request is not atomic.

  • ARRAY has no native mapping type.

  • _delete_by_query on a large index is slow.

  • A cluster that is neither Elasticsearch nor OpenSearch is accepted when it answers the same HTTP API.

  • The integration test suite runs with security disabled and plain HTTP. TLS and the API_KEY and BEARER modes are not covered by it.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

port defaults to 9200. enableTls defaults to false. Server flavor, authentication mode, timeouts, refresh, bulk size, path prefix, and additional nodes are set on the connection in the Control Plane.

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "host-address",
        "port": 9200,
        "username": "",
        "password": "",
        "enableTls": false
      }
}'