Greenplum agent

Target Greenplum

Greenplum speaks the PostgreSQL wire protocol, so this agent is the PostgreSQL target with Greenplum’s connection defaults, a bulk path built for Greenplum’s segments, and table creation that sets the distribution policy and the storage of the tables.

Prerequisites

To have Gluesync writing into your Greenplum cluster you will need:

The host, port and database of the coordinator, and a user that can log in to it from Core Hub: the coordinator’s pg_hba.conf must allow the Core Hub host;
A user with these privileges:
- CREATE on the database, so that the agent creates its GLUESYNC helper schema (schema of the staging tables of bulk operations). Without it, create the schema yourself and turn on Disable automatic schema creation;
- CREATE on the schema of the tables the agent creates, or, for tables that already exist, SELECT, INSERT, UPDATE, DELETE and TRUNCATE on them;
- The CREATEEXTTABLE role attribute for readable gpfdist external tables, if the bulk path loads through gpfdist (bulk load method AUTO or GPFDIST). With the COPY method it is not needed.
The agent does not need a superuser;
Network access from Core Hub to the coordinator and, for the gpfdist load, from every segment host to Core Hub. See Network requirements of gpfdist.

CREATE ROLE gluesync LOGIN PASSWORD '<password>';
GRANT CONNECT, CREATE ON DATABASE <database> TO gluesync;
GRANT USAGE, CREATE ON SCHEMA <schema> TO gluesync;

-- Only for the gpfdist load. Same syntax on Greenplum 6 and 7.
ALTER ROLE gluesync CREATEEXTTABLE (type='readable', protocol='gpfdist');

Without the attribute, creating a readable gpfdist external table fails with permission denied: no privilege to create a readable gpfdist(s) external table. With the AUTO method the agent then loads with COPY, see Troubleshooting.

Setup via Web UI

Hostname / IP Address: DNS record or IP address of the coordinator;
Port: Optional, defaults to 5432;
Database name: Name of your target database, defaults to postgres;
Username: User with the privileges above, defaults to gpadmin;
Password: Password belonging to the given username;
Enable TLS: off by default. See TLS;
Trust server certificate: on by default.

Specific configuration

The agent has two groups of settings: the settings of the connection (host-level), which the setup wizard asks for when you connect the agent, and the settings of the agent (agent-level), which apply to every table of the pipeline’s target.

Host-level settings

Setting Default Meaning

Setting	Default	Meaning
Use connection pool (`useConnectionPool`)	`true`	Uses a pool of connections to the coordinator.
Gluesync schema name (necessary for bulk operations) (`gluesyncSchemaName`)	`GLUESYNC`	Schema of the staging tables of bulk operations.
Disable automatic schema creation (`disableAutomaticSchemaCreation`)	`false`	Turn it on when the connection user must not create schemas, and create the Gluesync schema yourself.
Bulk load method (`bulkLoadMethod`)	`AUTO`	How the bulk path loads the spool file into the staging table: `AUTO`, `GPFDIST` or `COPY`. `AUTO` uses `gpfdist` when the segments can reach Core Hub and the flush is at least Minimum bytes of a flush loaded through gpfdist, and `COPY` otherwise. `GPFDIST` always uses `gpfdist` and fails the setup when a segment cannot reach Core Hub. `COPY` always streams the rows through the coordinator. See Bulk path.
gpfdist port (`gpfdistPort`)	`8580`	Port of the embedded `gpfdist` server, from `1` to `65535`. The segments connect to this port at the advertised host.
gpfdist bind address (`gpfdistBindAddress`)	`0.0.0.0`	Address the embedded `gpfdist` server listens on.
gpfdist advertised host (`gpfdistAdvertisedHost`)	empty	Host name or address the segments use to reach the `gpfdist` server. Empty means the host name of the Core Hub machine. It must resolve to Core Hub from every segment host.
Minimum bytes of a flush loaded through gpfdist by the AUTO method (`gpfdistMinFlushBytes`)	`8388608` (8 MiB)	With `AUTO`, a flush smaller than this is loaded with `COPY`: for a small load, the round trip of the segments costs more than it saves. `0` or more.
Spool directory of the bulk path (`spoolDirectory`)	empty	Directory of the spool files. Empty means `gluesync-greenplum` in the temporary directory of the Core Hub machine.
gpfdist TLS keystore path (`gpfdistTlsKeystorePath`)	empty	Path of a PKCS12 keystore with the certificate of the `gpfdist` server. When set, the server speaks `gpfdists`. See TLS of gpfdist.
gpfdist TLS keystore password (`gpfdistTlsKeystorePassword`)	empty	Password of the keystore. Required when the keystore path is set.
gpfdist TLS truststore path (`gpfdistTlsTruststorePath`)	empty	Path of a PKCS12 truststore with the CA that signs the client certificates of the segments. When set, every segment must present a client certificate signed by that CA; when empty, the segments are not asked for a certificate.
gpfdist TLS truststore password (`gpfdistTlsTruststorePassword`)	empty	Password of the truststore.
gpfdist write timeout in seconds (`gpfdistWriteTimeoutSeconds`)	`300`	A segment that reads nothing for this long loses its connection to the `gpfdist` server.

Use connection pool (useConnectionPool)

true

Uses a pool of connections to the coordinator.

Gluesync schema name (necessary for bulk operations) (gluesyncSchemaName)

GLUESYNC

Schema of the staging tables of bulk operations.

Disable automatic schema creation (disableAutomaticSchemaCreation)

false

Turn it on when the connection user must not create schemas, and create the Gluesync schema yourself.

Bulk load method (bulkLoadMethod)

AUTO

How the bulk path loads the spool file into the staging table: AUTO, GPFDIST or COPY. AUTO uses gpfdist when the segments can reach Core Hub and the flush is at least Minimum bytes of a flush loaded through gpfdist, and COPY otherwise. GPFDIST always uses gpfdist and fails the setup when a segment cannot reach Core Hub. COPY always streams the rows through the coordinator. See Bulk path.

gpfdist port (gpfdistPort)

8580

Port of the embedded gpfdist server, from 1 to 65535. The segments connect to this port at the advertised host.

gpfdist bind address (gpfdistBindAddress)

0.0.0.0

Address the embedded gpfdist server listens on.

gpfdist advertised host (gpfdistAdvertisedHost)

empty

Host name or address the segments use to reach the gpfdist server. Empty means the host name of the Core Hub machine. It must resolve to Core Hub from every segment host.

Minimum bytes of a flush loaded through gpfdist by the AUTO method (gpfdistMinFlushBytes)

8388608 (8 MiB)

With AUTO, a flush smaller than this is loaded with COPY: for a small load, the round trip of the segments costs more than it saves. 0 or more.

Spool directory of the bulk path (spoolDirectory)

empty

Directory of the spool files. Empty means gluesync-greenplum in the temporary directory of the Core Hub machine.

gpfdist TLS keystore path (gpfdistTlsKeystorePath)

empty

Path of a PKCS12 keystore with the certificate of the gpfdist server. When set, the server speaks gpfdists. See TLS of gpfdist.

gpfdist TLS keystore password (gpfdistTlsKeystorePassword)

empty

Password of the keystore. Required when the keystore path is set.

gpfdist TLS truststore path (gpfdistTlsTruststorePath)

empty

Path of a PKCS12 truststore with the CA that signs the client certificates of the segments. When set, every segment must present a client certificate signed by that CA; when empty, the segments are not asked for a certificate.

gpfdist TLS truststore password (gpfdistTlsTruststorePassword)

empty

Password of the truststore.

gpfdist write timeout in seconds (gpfdistWriteTimeoutSeconds)

300

A segment that reads nothing for this long loses its connection to the gpfdist server.

A value that is not valid fails the connection with a message that names the setting. A change of the gpfdist settings applies at the next load.

Agent-level settings

Setting Default Meaning

Setting	Default	Meaning
Storage type of the tables the agent creates (`storageType`)	`HEAP`	`HEAP`, `AO_ROW` (append-optimized, row oriented) or `AO_COLUMN` (append-optimized, column oriented).
Compression of the append-optimized tables the agent creates (`compressType`)	`ZSTD`	`ZSTD`, `ZLIB` or `NONE`. Used only with `AO_ROW` and `AO_COLUMN`.
Compression level (`compressLevel`)	`1`	`1` to `19` for `ZSTD`, `1` to `9` for `ZLIB`. Ignored for `NONE` and for heap tables. A level out of range fails the connection with a message that names the setting.
Disable truncate on target table before snapshot inserts (`disableTruncateOnSnapshot`)	`false`	Keeps the rows of the target table when a snapshot starts. See Target advanced settings.
State Preservation Table Name Prefix (`statePreservationTableNamePrefix`)	empty	Prefix of the state preservation tables. See PostgreSQL target advanced settings.

Storage type of the tables the agent creates (storageType)

HEAP

HEAP, AO_ROW (append-optimized, row oriented) or AO_COLUMN (append-optimized, column oriented).

Compression of the append-optimized tables the agent creates (compressType)

ZSTD

ZSTD, ZLIB or NONE. Used only with AO_ROW and AO_COLUMN.

Compression level (compressLevel)

1

1 to 19 for ZSTD, 1 to 9 for ZLIB. Ignored for NONE and for heap tables. A level out of range fails the connection with a message that names the setting.

Disable truncate on target table before snapshot inserts (disableTruncateOnSnapshot)

false

Keeps the rows of the target table when a snapshot starts. See Target advanced settings.

State Preservation Table Name Prefix (statePreservationTableNamePrefix)

empty

Prefix of the state preservation tables. See PostgreSQL target advanced settings.

The storage settings apply to the tables the agent creates from now on. They do not change tables that exist.

Entity settings

The entities of the pipeline have the PostgreSQL target’s write settings:

Setting Default Meaning

Setting	Default	Meaning
Max Write Batch Size (`maxWriteBatchSize`)	`1000`	Rows of an `INSERT` or upsert batch.
Max Delete Batch Size (`maxDeleteBatchSize`)	`1000`	Rows of a `DELETE` batch.
Max Write Bulk Size (`maxWriteBulkSize`)	`1000000`	Rows a snapshot spools before the bulk path loads them. During CDC every call loads what it received.
SQL Pre-Snapshot start command, SQL Post-Snapshot stop command, SQL Pre-CDC start command, SQL Post-CDC stop command	empty	SQL that runs on the target at the start and at the end of the snapshot and of CDC. See Target advanced settings.

Max Write Batch Size (maxWriteBatchSize)

1000

Rows of an INSERT or upsert batch.

Max Delete Batch Size (maxDeleteBatchSize)

1000

Rows of a DELETE batch.

Max Write Bulk Size (maxWriteBulkSize)

1000000

Rows a snapshot spools before the bulk path loads them. During CDC every call loads what it received.

SQL Pre-Snapshot start command, SQL Post-Snapshot stop command, SQL Pre-CDC start command, SQL Post-CDC stop command

empty

SQL that runs on the target at the start and at the end of the snapshot and of CDC. See Target advanced settings.

Write paths

Batch path

With bulk operations off, the agent writes the changes with JDBC batches: INSERT batches, DELETE batches, and for an upsert an UPDATE … FROM followed by an INSERT of the keys that were not updated. The statements never use ON CONFLICT or MERGE, so they run on Greenplum 6 and 7.

Whether a duplicate key is rejected, and so whether the On duplicate key strategy of the entity applies, depends on the table: it needs a unique index, which the tables the agent creates have, except the append-optimized tables on Greenplum 6. See Insert conflict strategy.

Bulk path

The bulk path is on per entity: Use bulk operations during Snapshot and Use bulk operations during CDC, which you toggle from the play/pause panel of the entity. It works as follows:

The agent encodes each row as a line of tab-delimited text and appends it to a spool file in the spool directory of Core Hub (UTF-8, one row per line);
During CDC the spool is flushed on every call. During a snapshot it is flushed when it holds Max Write Bulk Size rows, and when the snapshot ends;
The flush loads the spool file into a staging table in the Gluesync schema, in one transaction, by the Bulk load method:
- GPFDIST: the agent registers the file in its embedded gpfdist server under a random path, creates a temporary readable external table over gpfdist://<advertised host>:<port>/<path> and runs INSERT INTO <staging table> SELECT … FROM <external table>. Every primary segment pulls its share of the file in parallel, so the load grows with the number of segments;
- COPY: the agent streams the file through COPY … FROM STDIN on the coordinator, so every row passes through the coordinator;
- AUTO: GPFDIST when the reachability check passed and the flush is at least Minimum bytes of a flush loaded through gpfdist, COPY otherwise.
The staging table is merged into the target table in the same transaction (DELETE … USING the staging table, then INSERT … SELECT of the inserted and updated rows), and committed. The spool file is deleted after the commit. If the load fails, the transaction is rolled back, the staging table is truncated and the batch is retried, without writing the rows twice.

The staging table is a heap table in the Gluesync schema, distributed by the old key columns of the staging rows (<key>_OLD), so that the merge joins co-located rows when the target table is distributed by its primary key. A staging table of an entity without a key is distributed randomly.

When the target table has a valid B-tree index that starts with the key columns, the agent turns off hash and merge joins for the transaction of the merge (SET LOCAL), so that the merge uses the key index and its cost follows the size of the flush, not the size of the target table. The setting does not outlive the transaction.

The agent logs each flush at INFO level, with the rows, the bytes, the load method, the load time and the merge time.

Direct copy during snapshot

When bulk operations are on during the snapshot, the agent loads the rows of a flush straight into the target table, without a staging table, when both of these are true:

Every row of the flush is a full-row write (an insert, or an upsert under the Snapshot write method), not a delete;
The target table has an enforced unique key: a valid unique B-tree index on exactly the key columns of the entity, such as a primary key.

Every row is then written once. The load runs in one transaction with the load method of the flush. A flush that holds a delete goes through the staging table. Append-optimized tables created on Greenplum 6 have no primary key, so their snapshots always go through the staging table.

If the target table already holds a row with the same key, for example when Disable truncate on target table before snapshot inserts is on or when a snapshot resumes, Greenplum rejects the load with a duplicate key error. The agent rolls the transaction back, loads the same rows through the staging table, which replaces the rows of the same key (a delete and an insert), and keeps the entity on the staging table for the rest of the snapshot. It logs this at INFO level.

Network requirements of gpfdist

With the gpfdist load, the segments are the clients: every primary segment host opens a connection to Core Hub, to the advertised host and the gpfdist port (8580 by default). Core Hub does not connect to the segments. A coordinator that reaches Core Hub is not enough.

Open the gpfdist port between every segment host and Core Hub, in both firewalls (host and network);
The gpfdist advertised host must resolve, from every segment host, to the machine that runs Core Hub. When it is empty, the agent uses the host name of the Core Hub machine, which segment hosts often cannot resolve (a container’s host name never resolves outside it). Set it to a name or address of the Core Hub machine that the segment hosts can reach;
The port the segments connect to is the gpfdist port. A mapping that changes the port between the segments and Core Hub does not work;
The server listens on gpfdist bind address, 0.0.0.0 by default. Narrow it to one interface if the machine has several and only one is reachable from the segments;
Two Greenplum agents in the same Core Hub with the same bind address and port share one server, each with its own paths.

The server serves only the spool files that a load has registered, each under a random path, and removes the path when the load ends. The traffic of plain gpfdist is not encrypted: on a network you do not trust, use gpfdists.

Docker

Publish the gpfdist port of the Core Hub container, and set gpfdist advertised host to the address of the Docker host that the segment hosts can reach:

services:
  core-hub:
    ports:
      - "8580:8580"

The published port must be the same as the gpfdist port, as the segments connect to the Docker host at that port.

Kubernetes

Expose the gpfdist port of the Core Hub pod with a Service (type LoadBalancer, or NodePort with the same port number as the gpfdist port), or with a hostPort, and set gpfdist advertised host to the address that the segment hosts use to reach it. Allow the port in the network policies between the segment hosts and the pod.

Reachability check

When an entity uses bulk operations and the method is AUTO or GPFDIST, the agent checks at setup that every primary segment can reach the server: it registers a one-row file, reads it through a temporary external table, and requires a request from every primary segment within 30 seconds.

With GPFDIST, a segment that does not reach the server fails the setup with a message that names the advertised host and port;
With AUTO, the check ends with a warning and a notification, "gpfdist is not reachable", and the bulk path loads with COPY until the next setup that passes the check.

With COPY the check is skipped.

TLS of gpfdist

To encrypt the load, both sides need certificates signed by the same CA. verify_gpfdists_cert stays at its default (on): the segments verify the server, and the server verifies the segments.

On the segments. In every data directory listed by SELECT datadir FROM gp_segment_configuration, create a gpfdists/ directory owned by gpadmin with these files (the segments read them; the copy in the coordinator’s data directory is harmless):

root.crt: the certificate of the CA, in PEM format;
client.crt: a client certificate signed by that CA, in PEM format;
client.key: the unencrypted PKCS#8 private key of client.crt, in PEM format, with mode 0600.

In Gluesync. Set gpfdist TLS keystore path (gpfdistTlsKeystorePath) to a PKCS12 keystore that holds the server certificate and its private key, and gpfdist TLS keystore password (gpfdistTlsKeystorePassword). The certificate must be signed by the same CA, and its CN and SAN must match gpfdist advertised host. The agent then serves gpfdists, and the external tables use a gpfdists:// location. To require the client certificates of the segments, also set gpfdist TLS truststore path (gpfdistTlsTruststorePath) to a PKCS12 truststore that holds the CA, and its password; without a truststore the server does not ask the segments for a certificate.

When the load fails on TLS, the error of the agent names the gpfdists files, the keystore setting and the advertised host.

TLS of the connection

TLS is off by default, as Greenplum clusters often run without SSL. When Enable TLS is off, the connection does not use SSL.

With Enable TLS on and Trust server certificate on (default), the JDBC sslmode is require: the connection is encrypted and the certificate is not checked against a CA;
With Enable TLS on and Trust server certificate off, the JDBC sslmode is verify-ca: the server certificate must be signed by a CA found in the trust store you provide in the connection settings (trust store path and password).

Tables and schemas

Gluesync can create the target tables, as it does for any SQL target (see Target table creation), and writes any table of the database that the connection user can write to. A table you create yourself is written as it is, with the distribution policy and the storage it has.

The schemas that Greenplum keeps for its own tools (gp_toolkit, pg_aoseg, pg_bitmapindex, pg_ext_aux) and the external and foreign tables are not listed in the discovery, and the full-catalog walk of Core Hub (see PII discovery and masking) treats those schemas as system schemas.

Table creation

The statement that Core Hub proposes for a new table has the form:

CREATE TABLE <schema>.<table> (
  ...,
  PRIMARY KEY ("id")
) WITH (appendoptimized=true, orientation=column, compresstype=zstd, compresslevel=1)
DISTRIBUTED BY ("id")

Distribution: DISTRIBUTED BY the primary key columns, or DISTRIBUTED RANDOMLY for a table without a primary key. The agent works with whatever distribution policy a table has, including the tables it did not create. To distribute a table by another key, edit the generated statement in the table creation screen before you run it: Core Hub runs the statement as you leave it;
Storage: the WITH (…) clause comes from the agent-level settings Storage type, Compression and Compression level. It is left out for HEAP tables. The clause is the same for every table the agent creates;
Primary key on Greenplum 6: Greenplum 6 does not allow a primary key or a unique index on an append-optimized table. For AO_ROW and AO_COLUMN the agent leaves the PRIMARY KEY out of the statement and logs a warning, once per table. It then keeps the keys unique by itself: the bulk path replaces the rows of a key (a delete and an insert), and the batch path updates the row, or inserts it when it is missing. Greenplum 7 takes the primary key on append-optimized tables;
macaddr8: Greenplum 6 does not have the type, and the agent refuses to create a table with a macaddr8 column on it, with a message that suggests macaddr or a text type.

Data types

The mapping is the PostgreSQL target’s, narrowed to the types Greenplum has. The default is the type the agent creates for a Gluesync type. A column can be mapped to the alternatives:

Gluesync type Default Greenplum type Alternatives

Gluesync type	Default Greenplum type	Alternatives
`SHORT`	`smallint`	`smallserial`
`INT`	`integer`	`int`, `serial`
`LONG`	`bigint`	`bigserial`, `oid`
`BOOLEAN`	`boolean`	`bool`
`BIG_DECIMAL`	`decimal`	`numeric`
`FLOAT`	`real`	`float`
`DOUBLE`	`double precision`
`LOCAL_DATE`	`date`
`LOCAL_TIME`	`time without time zone`	`time`
`LOCAL_DATE_TIME`	`timestamp without time zone`	`timestamp`
`OFFSET_DATE_TIME`	`timestamp with time zone`	`timestamptz`
`OFFSET_TIME`	`time with time zone`	`timetz`
`STRING`	`varchar`	`char`, `character`, `character varying`, `text`, `json`, `jsonb`, `xml`, `uuid`, enum types, `money`, `interval`, `inet`, `cidr`, `macaddr`, `macaddr8`, `bit`, `bit varying`, `point`, `lseg`, `line`, `circle`, `box`, `path`, `polygon`, `int4range`, `int8range`, `numrange`, `tsrange`, `tstzrange`, `daterange`, `tsvector`, `tsquery`, `txid_snapshot`, `pg_lsn`, `hstore`
`BYTE_ARRAY`	`bytea`
`ARRAY`	`array`	Arrays of the types above

SHORT

smallint

smallserial

INT

integer

int, serial

LONG

bigint

bigserial, oid

BOOLEAN

boolean

bool

BIG_DECIMAL

decimal

numeric

FLOAT

real

float

DOUBLE

double precision

LOCAL_DATE

date

LOCAL_TIME

time without time zone

time

LOCAL_DATE_TIME

timestamp without time zone

timestamp

OFFSET_DATE_TIME

timestamp with time zone

timestamptz

OFFSET_TIME

time with time zone

timetz

STRING

varchar

char, character, character varying, text, json, jsonb, xml, uuid, enum types, money, interval, inet, cidr, macaddr, macaddr8, bit, bit varying, point, lseg, line, circle, box, path, polygon, int4range, int8range, numrange, tsrange, tstzrange, daterange, tsvector, tsquery, txid_snapshot, pg_lsn, hstore

BYTE_ARRAY

bytea

ARRAY

array

Arrays of the types above

macaddr8 exists on Greenplum 7 only. See Table creation;
hstore needs the extension in the target database: run CREATE EXTENSION hstore; before the pipeline writes the column;
Greenplum does not have the multirange types, pg_snapshot, geometry, geography, raster, vector and tinyint: they are not offered in the mapping.

Updates with bulk operations during CDC

When Use bulk operations during CDC (useBulkOperationsDuringCDC) is on for an entity, the agent does not apply an update as an UPDATE statement. It loads the change into a staging table, deletes the existing row from the target table and inserts the new version of it. With the option off, updates are applied as UPDATE statements.

For the target table this has the following effects:

Columns that Gluesync does not write, such as a column with a default value, are set again as for a new row;
Every update leaves the old version of the row in the table. Heap tables keep it as a dead row until VACUUM, and append-optimized tables mark it deleted until VACUUM compacts the table, so a table that receives many updates needs regular vacuuming.

Troubleshooting

Issue Solution

Issue	Solution
`The server is not a Greenplum Database or an Apache Cloudberry`	The connection reached a PostgreSQL server, or another product. The agent reads `SELECT version()` and `gp_segment_configuration` when it connects. Point it at the coordinator of a Greenplum cluster, or use the PostgreSQL agent for a PostgreSQL server. Greenplum versions older than 6 are rejected too.
`The Greenplum segments cannot all reach the gpfdist server of the agent` (setup fails with the `GPFDIST` method)	A segment host cannot open a connection to the advertised host and port. Check the firewall between every segment host and Core Hub, that gpfdist advertised host resolves to Core Hub from the segment hosts, and the gpfdist port. In Docker and Kubernetes the port must be published or exposed, see Network requirements of gpfdist. The message ends with the reason: `Only N of M primary segments reached the server` names how many segments connected.
`gpfdist is not reachable` (warning, with the `AUTO` method)	The same cause. The agent keeps working and loads with `COPY`, which is slower on a large cluster because every row passes through the coordinator. Fix the network and run the setup again, for example by restarting the entity. Set Bulk load method to `GPFDIST` to have the setup fail instead of falling back.
The probe fails with `permission denied: no privilege to create a readable gpfdist(s) external table`	The connection user lacks the external table privilege. Run `ALTER ROLE <user> CREATEEXTTABLE (type='readable', protocol='gpfdist');`.
`append-only tables do not support unique indexes`	You added a primary key or a unique index to an append-optimized table on Greenplum 6. Remove it: the agent works without it.
`Column … has the type macaddr8, which Greenplum 6 does not have`	Map the column to `macaddr` or to a text type.
`hstore` column fails	Install the extension in the target database with `CREATE EXTENSION hstore;`.
The connection fails with a message that names `gpfdistPort`, `gpfdistMinFlushBytes`, `gpfdistTlsKeystorePassword`, `storageType`, `compressType` or `compressLevel`	The value of that setting is not valid, see the ranges in Specific configuration. The message states the expected range.
Segments reject `gpfdists`	Check, on every segment data directory, the `gpfdists/` files `root.crt`, `client.crt` and `client.key` (owner `gpadmin`, key mode `0600`, unencrypted PKCS#8), that the server certificate in the keystore of gpfdist TLS keystore path is signed by the same CA, and that its CN and SAN match gpfdist advertised host.
The disk of Core Hub fills during a large load	The spool file holds the rows of a flush as text until the load commits. Point Spool directory of the bulk path to a volume with space for Max Write Bulk Size rows, or lower the setting. The agent deletes the spool files of a run that was interrupted when it starts again.

The server is not a Greenplum Database or an Apache Cloudberry

The connection reached a PostgreSQL server, or another product. The agent reads SELECT version() and gp_segment_configuration when it connects. Point it at the coordinator of a Greenplum cluster, or use the PostgreSQL agent for a PostgreSQL server. Greenplum versions older than 6 are rejected too.

The Greenplum segments cannot all reach the gpfdist server of the agent (setup fails with the GPFDIST method)

A segment host cannot open a connection to the advertised host and port. Check the firewall between every segment host and Core Hub, that gpfdist advertised host resolves to Core Hub from the segment hosts, and the gpfdist port. In Docker and Kubernetes the port must be published or exposed, see Network requirements of gpfdist. The message ends with the reason: Only N of M primary segments reached the server names how many segments connected.

gpfdist is not reachable (warning, with the AUTO method)

The same cause. The agent keeps working and loads with COPY, which is slower on a large cluster because every row passes through the coordinator. Fix the network and run the setup again, for example by restarting the entity. Set Bulk load method to GPFDIST to have the setup fail instead of falling back.

The probe fails with permission denied: no privilege to create a readable gpfdist(s) external table

The connection user lacks the external table privilege. Run ALTER ROLE <user> CREATEEXTTABLE (type='readable', protocol='gpfdist');.

append-only tables do not support unique indexes

You added a primary key or a unique index to an append-optimized table on Greenplum 6. Remove it: the agent works without it.

Column … has the type macaddr8, which Greenplum 6 does not have

Map the column to macaddr or to a text type.

hstore column fails

Install the extension in the target database with CREATE EXTENSION hstore;.

The connection fails with a message that names gpfdistPort, gpfdistMinFlushBytes, gpfdistTlsKeystorePassword, storageType, compressType or compressLevel

The value of that setting is not valid, see the ranges in Specific configuration. The message states the expected range.

Segments reject gpfdists

Check, on every segment data directory, the gpfdists/ files root.crt, client.crt and client.key (owner gpadmin, key mode 0600, unencrypted PKCS#8), that the server certificate in the keystore of gpfdist TLS keystore path is signed by the same CA, and that its CN and SAN match gpfdist advertised host.

The disk of Core Hub fills during a large load

The spool file holds the rows of a flush as text until the load commits. Point Spool directory of the bulk path to a volume with space for Max Write Bulk Size rows, or lower the setting. The agent deletes the spool files of a run that was interrupted when it starts again.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "greenplum-coordinator-address",
        "port": 5432,
        "databaseName": "db_name",
        "username": "gluesync",
        "password": "",
        "maxConnectionsCount": 100,
        "enableTls": false,
        "trustServerCertificate": true
      },
      "customHostCredentials": {
        "bulkLoadMethod": "AUTO",
        "gpfdistPort": 8580,
        "gpfdistAdvertisedHost": "core-hub.example.com"
      }
}'

The keys under customHostCredentials are the host-level settings, named as in parentheses in Specific configuration. A setting you leave out has its default.