HTTP Source and HTTP Target agent for Gluesync: overview

Index

Core principles

Gluesync’s HTTP agent (courier-agent) connects pipelines to HTTP in both directions, through the API definitions of Courier, the REST editor of the Core Hub web UI:

as HTTP Source it serves Courier inbound pipeline endpoints on its own listener: every authenticated request becomes one pipeline row, written durably to disk and answered 202, that the pipeline delivers to any target (see Source setup);
as HTTP Target it turns the change feed of any supported source into calls toward a REST API (see Target setup).

An agent instance is assigned to a pipeline either as source or as target; the two roles share the code, not the state.

HTTP Target

The HTTP Target turns the change feed of any supported source into HTTP calls toward a REST API: one request per inserted, updated or deleted row, built from the row itself. It is the delivery side of Courier, the REST editor of the Core Hub web UI, where the API is described once as a collection (base URL and authentication) of endpoints (HTTP verb, path template, headers and bindings).

The pipeline treats a collection as a schema, an endpoint as a table and the bindings as its columns, so an entity toward an API is configured exactly like an entity toward a database: pick the source table, pick the endpoint, map the columns. At run time each mapped value fills its binding, the binding fills the path, the query string, a header or the body, and the request is sent with the credentials of the collection. The operation that produced the row (Insert, Update or Delete) is part of every request, as a body field and as a {__operation} placeholder usable in headers and paths. Every request also carries an Idempotency-Key header that stays the same when the same change is sent again.

The agent ships inside Core Hub, runs in its process and needs no connection parameters of its own: it reads the Courier definitions from Core Hub directly, with no token.

HTTP Source

The HTTP Source receives data instead of reading it: clients push records to Courier inbound write endpoints (CREATE, UPDATE, DELETE) that have bindings and are assigned to the HTTP Source of a pipeline. The pipeline treats the inbound collection as a source schema, each such endpoint as a source table and its bindings as the columns, so the entity is configured like any other: pick the endpoint, pick the target table, map the columns.

Each request is matched, authenticated with a Personal API Token, bound into one row (Insert, Update or Delete) and appended to a queue on the local disk; the client gets 202 Accepted with the row’s sequence as soon as the row is on disk, forced past the operating system’s cache, never after the target. Core Hub then polls the queue like any other source and the target writes the rows, at least once and in order per endpoint. A stopped or failing target only makes the queue grow, within configurable limits.

Courier inbound endpoints that read (LIST, READ), and write endpoints without bindings, keep running SQL through Core Hub against a SQL-capable source agent.

Supported data types

HTTP Source

Every binding of an inbound pipeline endpoint becomes a column with a fixed type; a value that does not fit is refused with 400:

Binding type Column type Accepted values

Binding type	Column type	Accepted values
STRING	`VARCHAR(255)`	Text up to 255 characters; objects and arrays as raw JSON text, which must fit too.
NUMBER	`NUMERIC(18,2)`	Up to 16 integer digits and 2 decimal places once trailing zeros are removed, never rounded; text of more than 64 characters is refused.
BOOLEAN	`BOOLEAN`	`true` or `false`.

STRING

VARCHAR(255)

Text up to 255 characters; objects and arrays as raw JSON text, which must fit too.

NUMBER

NUMERIC(18,2)

Up to 16 integer digits and 2 decimal places once trailing zeros are removed, never rounded; text of more than 64 characters is refused.

BOOLEAN

BOOLEAN

true or false.

An endpoint without a key gets a synthetic _seq key column (NUMERIC(18,0)) holding the request sequence.

HTTP Target

Every column value is rendered as text and placed where its binding says. The binding type decides how a JSON body encodes it:

Binding type Encoding

Binding type	Encoding
STRING	A JSON string, escaped. Strings, date and time types (ISO-8601), UUIDs and any other value are sent as they print.
NUMBER	A bare JSON number in plain notation (`0.0000000001`, never `1E-10`) with every digit it has, nothing rounded; a value that is not a number is sent as a string. Only a number too long to write out (more than 1,000 characters of text or 10,000 digits) fails the write.
BOOLEAN	A bare `true` or `false`; a value that is not a boolean is sent as a string.

STRING

A JSON string, escaped. Strings, date and time types (ISO-8601), UUIDs and any other value are sent as they print.

NUMBER

A bare JSON number in plain notation (0.0000000001, never 1E-10) with every digit it has, nothing rounded; a value that is not a number is sent as a string. Only a number too long to write out (more than 1,000 characters of text or 10,000 digits) fails the write.

BOOLEAN

A bare true or false; a value that is not a boolean is sent as a string.

Binary columns are sent in Base64, arrays and collections as JSON arrays. In a form-encoded body, in the query string and in the path every value is text, URL-encoded. A binding without value is null in a JSON body and omitted from a form body.

Platform supported features

Feature Supported Notes

Feature	Supported	Notes
Target role	✅	HTTP Target: one HTTP request per row toward a Courier outbound endpoint.
Source role	✅	HTTP Source: one row per request on a Courier inbound pipeline endpoint, queued on disk and answered `202`.
Snapshots	✅	HTTP Source: the snapshot is empty, then the queued rows flow. HTTP Target: every row of the snapshot is delivered sequentially, with operation `Update` (a snapshot row is an upsert).
CDC	✅	HTTP Source: every accepted request is a change, delivered at least once and in order per endpoint. HTTP Target: inserts, updates and deletes are delivered with the configured verb of the endpoint; the operation travels in the body field and in `{__operation}`.
Bulk load	❌	One request per row; no batching.
Database pooling	N/A	No database connection. One HTTP client per agent configuration, with connection and request timeouts.
Internal caches	✅	Courier definitions (and, on the HTTP Source, token verdicts for at most a minute) are cached on the agent and refreshed on every change made in Courier. The HTTP Source keeps its queue on the local disk until the target confirms each row.
UDF	❌	Field functions of the pipeline apply before the row reaches the agent.
Custom document keys	N/A	The resource is identified by the PATH bindings of the endpoint.
Before & after images	❌	The request carries the row as it is after the change. A delete is addressed with the before-image of its PATH and QUERY bindings.
Recursion protection	N/A
Logical partitioning	❌
Allowed operations forwarding	✅	Operations filtered out at entity level are not sent.
Snapshot write method	N/A	Insert and upsert are the same request. Every request carries an `Idempotency-Key` header, the same when the row is sent again; honouring it is up to the receiving API.
Data filtering	✅	Entity filters apply before delivery.
Truncate	❌	Ignored: there is no remote truncate.
Error handling	✅	A non-2xx answer (a 3xx too, redirects are never followed), a connection error or a timeout is a writing error: the entity reports it and the change is retried from the checkpoint. A `DELETE` answered 404 or 410 counts as delivered.

Target role

✅

HTTP Target: one HTTP request per row toward a Courier outbound endpoint.

Source role

✅

HTTP Source: one row per request on a Courier inbound pipeline endpoint, queued on disk and answered 202.

Snapshots

✅

HTTP Source: the snapshot is empty, then the queued rows flow. HTTP Target: every row of the snapshot is delivered sequentially, with operation Update (a snapshot row is an upsert).

CDC

✅

HTTP Source: every accepted request is a change, delivered at least once and in order per endpoint. HTTP Target: inserts, updates and deletes are delivered with the configured verb of the endpoint; the operation travels in the body field and in {__operation}.

Bulk load

❌

One request per row; no batching.

Database pooling

N/A

No database connection. One HTTP client per agent configuration, with connection and request timeouts.

Internal caches

✅

Courier definitions (and, on the HTTP Source, token verdicts for at most a minute) are cached on the agent and refreshed on every change made in Courier. The HTTP Source keeps its queue on the local disk until the target confirms each row.

UDF

❌

Field functions of the pipeline apply before the row reaches the agent.

Custom document keys

N/A

The resource is identified by the PATH bindings of the endpoint.

Before & after images

❌

The request carries the row as it is after the change. A delete is addressed with the before-image of its PATH and QUERY bindings.

Recursion protection

N/A

Logical partitioning

❌

Allowed operations forwarding

✅

Operations filtered out at entity level are not sent.

Snapshot write method

N/A

Insert and upsert are the same request. Every request carries an Idempotency-Key header, the same when the row is sent again; honouring it is up to the receiving API.

Data filtering

✅

Entity filters apply before delivery.

Truncate

❌

Ignored: there is no remote truncate.

Error handling

✅

A non-2xx answer (a 3xx too, redirects are never followed), a connection error or a timeout is a writing error: the entity reports it and the change is retried from the checkpoint. A DELETE answered 404 or 410 counts as delivered.

Security

The HTTP Source accepts only requests carrying a Personal API Token with the HTTPS agent permission scope (gluesync.https_agent), scoped to its pipeline and agent. A collection can also keep a token allowlist, and always a blocklist that wins over everything else. The owner of the token must have a role that may write: a token of a Viewer or a Monitor is refused with 403. A token revoked, or whose owner is deleted or changes role, is refused within about a second; the HTTP Source otherwise remembers a token verdict for at most a minute, and never past the token’s expiry. The listener serves plain HTTP only when a TLS keystore is not set and you explicitly allow it, or when it listens on the loopback address: see Security on the source page.

On the HTTP Target, collection credentials (basic password, bearer token, API key value) and endpoint header values are encrypted at rest in Core Hub, masked in every API response and redacted from the test history and from write errors; they are sent only to the origin (scheme, host and port) of the base URL they were entered for. Calls to loopback, link-local, cloud metadata, multicast addresses and to Core Hub’s own port are refused unless GS_COURIER_ALLOW_LOCAL_EGRESS=true, and redirects are never followed: see Egress rule.

Supported version

HTTP Source: any HTTP/1.1 client able to send a bearer token. HTTP Target: any HTTP/1.1 API reachable over http:// or https://, answering 2xx to accepted deliveries. Both require Core Hub 2.3.2.1 or above.

Q&A

Is the client answered after the target wrote the row?
No. The HTTP Source answers 202 as soon as the row is in its queue on disk; delivery to the target follows within the polling interval. Before answering, the HTTP Source forces the row to the disk (an fsync, shared by the requests that arrive together), so a 202 survives a restart of Core Hub and a crash or a power loss of the host. When the disk cannot take the row the client gets 503 and retries.

What happens to requests while the target is down?
They keep being accepted and queued, and are delivered when the target comes back. Beyond the configured backlog limits the HTTP Source answers 503 with Retry-After until the target catches up.

Does the agent send a different HTTP verb for inserts, updates and deletes?
No. One verb is configured per endpoint and used for every operation; the operation name is sent in the body field (operation by default) and is available as the {__operation} placeholder in header value templates and in the path.

What happens when the API is down or answers an error?
The write fails: the transaction is stored as a writing error with its metrics, the entity shows the error and Core Hub retries the same change until the API accepts it. No change is skipped. The error never shows a credential: URLs are cut of their query values and user:pass@, and the start of the answer has the collection’s secret and the values of the endpoint’s credential headers replaced by *. The only non-2xx answer taken as delivered is a DELETE answered 404 or 410.

Can the API tell a retried row from a new one?
Yes: every row request carries an Idempotency-Key, a SHA-256 of the change’s source position, table, operation and values, the same each time Core Hub hands the change over again, after a retry or a resume. An API that remembers the keys it has seen can answer a repeated one without applying the row again. Its limits, where a new read gives a row a new key, are listed in Idempotency key.

Why is my call to localhost or to a private address refused?
Loopback, link-local, cloud metadata, multicast addresses and Core Hub’s own port are refused, and so are names that do not resolve; private ranges such as 10.0.0.0/8 are allowed. Start Core Hub with GS_COURIER_ALLOW_LOCAL_EGRESS=true to lift the rule, for a mock server on the same host.

Can I send the same row to two APIs?
Yes: add one HTTP Target per collection to the pipeline (or one entity per endpoint), each with its own column mapping.

How do I see the exact request the agent sends?
Use the Send test request button of the endpoint in Courier for sample values, or point the collection at a request recorder such as WireMock, as the pgsqlwal-http-target-integration-test demo kit does.