HTTP Target agent

Target HTTP Target (Courier outbound)

The HTTP Target agent (courier-agent) delivers every change captured on the source as one HTTP call toward a REST API of your choice. What it calls is not configured on the agent: it is described in Courier, the REST editor of the Core Hub web UI, as a collection (the API: base URL and authentication) holding endpoints (HTTP verb, path, headers and bindings). The pipeline then maps source columns onto the bindings of an endpoint, the way it maps columns onto the columns of a database table.

Prerequisites

To have Gluesync delivering changes to your API you will need to have:

Core Hub 2.3.2.1 or above, which ships the HTTP Target and the Courier module;
A Core Hub user who can manage connections (a Manager or a Super Admin), to define and test the API in Courier;
An HTTP or HTTPS endpoint reachable from the host running Core Hub, where the HTTP Target runs, and not on one of the addresses the egress rule refuses;
The credentials the API expects, if any: a basic username and password, a bearer token or an API key, which are stored on the Courier collection.

How a row becomes a request

For every row the pipeline emits the HTTP Target resolves the endpoint of the target entity, fills its bindings with the mapped column values and sends:

URL: the base URL of the collection, followed by the endpoint path template with every {placeholder} replaced by the value of the PATH binding of the same name (URL-encoded), followed by one query parameter per QUERY binding;
Headers: the endpoint headers, whose value templates may reference {binding} names and {__operation}, followed by the authentication header of the collection (Authorization: Basic …, Authorization: Bearer … or the API key header) and by the Idempotency-Key of the row (see Idempotency key);
Body, only for POST, PUT and PATCH with a body type other than NONE: the operation field first (Insert, Update or Delete), then one field per BODY binding named after it. With body type JSON the binding type is kept (NUMBER and BOOLEAN values unquoted, a missing value becomes null); with FORM_URLENCODED the same fields are form-encoded and missing values are skipped.

One HTTP verb is configured per endpoint and is used for inserts, updates and deletes alike: the operation travels in the body field and in the {__operation} placeholder, so the receiving API can tell them apart. Rows delivered by a snapshot carry Update, because a snapshot row is an upsert; Insert is what a captured insert carries.

A binding takes its default value only when the row does not carry its column, for example when no source column is mapped onto it. A column the source sets to NULL is sent as null, not as the default, and a NULL in a required binding fails the write (the entity goes into error), as a missing value without default does. Truncate operations are ignored.

A delete needs only its PATH and QUERY bindings, which identify the resource: its BODY and HEADER bindings are optional and take no default, so a header built only from a binding is sent empty. The PATH and QUERY values come from the row as it was before the delete, by the same NULL rule, so that the delete goes where the insert of the row went. When that before-image holds the key columns only (a PostgreSQL table with the default replica identity logs nothing else) the other columns were not logged, so their PATH and QUERY bindings keep their defaults. Values Gluesync generates instead of reading them do not count as logged columns: the TRANSACTION* technical fields of the source and the field functions of the entity on a column no source column feeds. A Custom field function on a column a source column feeds, which computes its value from the key, makes the delete read as a full before-image: its PATH and QUERY bindings then take no default.

The collection’s credentials, and the header values of its endpoints, are sent to the origin (scheme, host and port) of the base URL they were entered for only: a path template that would move the request to another host (for example @other.host/x) fails the write instead of sending them there.

Idempotency key

Every row request carries an Idempotency-Key header: a SHA-256, in hexadecimal, of what the source keeps across a new read of the change. That is its position in the source (without the time and the place in the agent’s cache that a reader stamps on it at each read; for a snapshot row, the position the snapshot started from and the place of its page in the snapshot), its table and operation, every value and before-image of the row, and how many identical rows came before it in the same batch. So the key is the same when Core Hub hands the change over again, after a failed call, a restart or a resume, and two different changes to one row never share it. An API that remembers the keys it has seen can answer a repeated one without applying the row twice. A row with no source position gets no key. The key is not part of the endpoint definition and is not shown by the Send test request button.

The key has limits, where a new read of the same change gets a new key and the API takes it for a new change (a duplicate, never a lost row):

a change whose position holds nothing but a time, where the reader stamps that time at each read;
a value made at read time, such as a field function that stamps the current time;
a snapshot row whose key holds its page, when a resume cuts the pages elsewhere (a keyed row of a resumable snapshot leaves the page out);
a CouchDB source, whose rows carry the sequence their feed page ended at, and a Vertica source, whose rows carry the epoch window of their read: a new read that ends elsewhere changes the key of every row it reads again.

Conversely, two identical changes at the same source position handed over in two batches share a key, so an API that deduplicates applies one of them; and a new snapshot of a source that has not changed since the last one gives an unchanged row the key it had then, so an API that still remembers it skips it.

Answers

A 2xx answer delivers the row.
A DELETE row answered 404 or 410 counts as delivered: the resource is already gone, which is what the delete wanted.
Any other answer fails the write, a 3xx too: redirects are never followed, so a credential cannot travel to a host nobody configured.
A connection error, a refused destination (see Egress rule) or a timeout fails the write as well. The agent waits 10 seconds for the connection and 30 seconds for the call.

A failed write stores the transaction as a writing error, the entity reports it and Core Hub retries the change from its checkpoint instead of skipping it. The error shows the status, the verb, the URL without its query values and without any user:pass@, and the first 200 characters of the answer, with every credential replaced by *: the secret of the collection, a Basic credential’s header token, and the value of every endpoint header whose name says it carries a credential (auth, key, token, secret, signature, password, passwd, session, cookie, credential), whole and word by word.

Egress rule

Every outbound call, from the Send test request button, from a saved endpoint’s test and from the HTTP Target writes, goes through the egress rule. Gluesync runs on your network, so private ranges (RFC 1918) are allowed. The host name is resolved first and the call is refused when any of its addresses is:

loopback (127.0.0.0/8, ::1) or the unspecified address (0.0.0.0, ::);
link-local (169.254.0.0/16, which holds most cloud metadata services, and fe80::/10);
a cloud metadata service outside link-local: AWS over IPv6 (fd00:ec2::254) and Alibaba Cloud (100.100.100.200);
multicast;
any of the above behind the NAT64 prefix 64:ff9b::/96;
an address of the Core Hub host on Core Hub’s own port (GLUESYNC_PORT, 1717 by default).

The HTTP client then connects to the addresses that were checked, so a DNS answer that changes in between cannot move the call; the last rule is a check of the URL, made on a resolution of its own. A name that does not resolve is refused too. A refused call is a failure with a fixed message and no stack trace. Behind a JVM proxy (-Dhttps.proxyHost) every rule is a check of the URL only, because the proxy resolves the name again: what the proxy may reach is the proxy’s own rule to set.

To call one of these addresses on purpose, for example a mock server on the same host, start Core Hub with the environment variable GS_COURIER_ALLOW_LOCAL_EGRESS=true. The rule is then off altogether, for the test button and the HTTP Target writes alike, and Core Hub logs a warning.

Setup via Web UI

Courier: describe the API

In Courier Outbound create a collection and its endpoints.

Collection
- Name: the name of the API. It is the target schema the pipeline will see.
- Base URL: http:// or https:// URL every endpoint path is appended to, without user:pass@: credentials belong in Authentication.
- Authentication: None, Basic (Username and Password), Bearer (Token) or API key (Send API key in a header or in the query string, Parameter name and API key). Secrets are encrypted at rest and never returned by the API; leaving the secret blank when editing a collection keeps the stored one, unless the base URL now points at another origin (scheme, host or port; another path on the same host keeps it) or the API key changed name or location: then enter it again.
Endpoint
- Name: the target table the pipeline will see.
- Method: GET, POST, PUT, PATCH, DELETE, HEAD or OPTIONS.
- Path: path appended to the base URL, with {name} placeholders for PATH bindings, e.g. /customers/{id}.
- Body type: NONE, JSON or FORM_URLENCODED; only POST, PUT and PATCH send a body.
- Operation field name: (optional, defaults to operation) name of the body field carrying Insert, Update or Delete.
- Headers: name and value template, e.g. X-Operation = {__operation}. Host, Content-Length, Transfer-Encoding and Connection are set by the HTTP client and cannot be defined. A header value can hold a credential (an API key, a token in a scheme the collection’s authentication does not cover), so it is treated as a secret: encrypted at rest, shown as ******** (only a Super Admin can read it, through the REST API), and sent only to the origin of the base URL it was entered for. Leaving ******** when editing keeps the stored value.
- Bindings: the values the request is built from, one target column each:
  - Name: the column name the pipeline maps onto, and the placeholder name;
  - Location: PATH (fills the path), QUERY (becomes a query parameter), BODY (becomes a body field), HEADER (a value for header templates);
  - Type: STRING, NUMBER or BOOLEAN (drives the JSON encoding);
  - Required: (defaults to true) a missing or NULL required value fails the write;
  - Default value: used when the row does not carry the binding’s column, for example when no column is mapped onto it.

The Send test request button sends the endpoint with the sample values of its bindings, after asking you to confirm, and shows the status, the headers, the start of the answer and the duration. A 3xx is shown as it is, with its Location header. Saved test history keeps the latest 50 results of each saved endpoint, older ones are deleted. A result never repeats a credential: Authorization, Proxy-Authorization, Cookie, Set-Cookie, the API key header, the API key query value and every header the endpoint defines are shown as ••••, in the request and in the answer, and a credential the API echoes back in the body, in another header or in an error is replaced by *. The test goes through the egress rule.

Moving a collection to another origin (scheme, host or port) while its endpoints hold header values entered for the old one asks for a confirmation: those values are never sent to another host, so after the move each of those endpoints is refused (a writing error, a test that does not run) until its header values are entered again.

Agent: connect the HTTP Target to the pipeline

Add the HTTP Target agent to the pipeline as a target. The agent runs inside Core Hub and needs no host, port or credentials.

Outbound collection: (optional) the Courier collection this agent instance delivers to, by name or id. When set, discovery exposes only that collection; otherwise every outbound collection is exposed as a schema.

Entity: map columns onto bindings

Create the entity as for any other target: the collection is the schema, the endpoint is the table and the bindings are its columns. Map each source column onto the binding that has to carry it; source columns that are not mapped never reach the API. The primary key of the endpoint is its first required PATH binding; an endpoint without one exposes a synthetic _row key column. The Map to pipeline button of the endpoint in Courier builds the same mapping from the endpoint side: choose the pipeline with an HTTP Target, the source table and the column of each binding, and save the entity.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to describe the API in Courier and to connect the agent. Ids are chosen by the caller; posting an outbound definition with an id already stored updates it, with the same checks as a PUT.

Create the collection

curl --location --request POST 'http://core-hub-ip-address/api-agent/collections' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "id": "7c5f1d0e-2b7a-4e0a-9f3c-1a2b3c4d5e6f",
    "name": "demo-api",
    "baseUrl": "https://api.example.com",
    "auth": { "type": "Bearer", "token": "s3cr3t" }
}'

baseUrl must be an http:// or https:// URL without user:pass@. auth accepts {"type": "None"}, {"type": "Basic", "username": "…", "password": "…"}, {"type": "Bearer", "token": "…"} and {"type": "ApiKey", "location": "HEADER" | "QUERY", "name": "X-Api-Key", "value": "…"}.

Create an endpoint

curl --location --request POST 'http://core-hub-ip-address/api-agent/endpoints' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "id": "0f9e8d7c-6b5a-4433-9211-aabbccddeeff",
    "name": "customers",
    "collectionId": "7c5f1d0e-2b7a-4e0a-9f3c-1a2b3c4d5e6f",
    "httpVerb": "POST",
    "pathTemplate": "/customers/{id}",
    "bodyType": "JSON",
    "operationField": "operation",
    "headers": [
        { "name": "X-Operation", "valueTemplate": "{__operation}" },
        { "name": "X-Tenant", "valueTemplate": "{tenant}" }
    ],
    "bindings": [
        { "name": "id", "type": "NUMBER", "location": "PATH", "required": true },
        { "name": "name", "type": "STRING", "location": "BODY", "required": true },
        { "name": "email", "type": "STRING", "location": "BODY", "required": false },
        { "name": "active", "type": "BOOLEAN", "location": "BODY", "required": true },
        { "name": "credit", "type": "NUMBER", "location": "BODY", "required": true },
        { "name": "tenant", "type": "STRING", "location": "HEADER", "required": true, "defaultValue": "acme" }
    ]
}'

GET /api-agent/collections, GET /api-agent/endpoints?collectionId=…, PUT /api-agent/collections/{id}, PUT /api-agent/endpoints/{id} and the matching DELETE routes complete the API. Every change is pushed to the running HTTP Targets, which refresh their definitions without a restart. An update that leaves the secret blank keeps the stored one, unless the base URL now points at another origin or the API key changed name or location: then the request is refused with 400 until the secret is sent again.

Header values come back as ********; a Super Admin gets them in clear with ?include_secrets=true. Sending ******** back keeps the stored value, while the endpoint stays in its collection and the collection on the origin the value was entered for; otherwise the save is a 400 until the value is sent again. A collection save (POST or PUT) that moves the base URL to another origin while endpoints hold header values for the stored one is a 400 naming those endpoints; repeated with ?force=true it moves anyway, and the header values of those endpoints must then be entered again.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "hostCredentials": {
        "connectionName": "HTTP Target"
    },
    "customHostCredentials": {
        "outboundCollection": "demo-api"
    }
}'

Setup via Bootstrapper

The Bootstrapper creates the Courier definitions, the agent and the entities in one run: the target agent entry of config.json carries a courier block with the collections and endpoints above, and the table list maps source columns onto binding names.

{
  "agentType": "TARGET",
  "agentTag": "courier-agent",
  "hostCredentials": { "connectionName": "HTTP Target" },
  "customHostCredentials": { "outboundCollection": "demo-api" },
  "courier": {
    "collections": [
      {
        "name": "demo-api",
        "baseUrl": "https://api.example.com",
        "auth": { "type": "Bearer", "token": "s3cr3t" },
        "endpoints": [
          {
            "name": "customers",
            "httpVerb": "POST",
            "pathTemplate": "/customers/{id}",
            "bodyType": "JSON",
            "headers": [ { "name": "X-Operation", "valueTemplate": "{__operation}" } ],
            "bindings": [
              { "name": "id", "location": "PATH", "type": "NUMBER" },
              { "name": "name", "location": "BODY" },
              { "name": "email", "location": "BODY", "required": false },
              { "name": "tenant", "location": "HEADER", "defaultValue": "acme" }
            ]
          }
        ]
      }
    ]
  }
}

public:
  target: demo-api          # the Courier collection
  tables:
    whitelist:
      - customers           # the Courier endpoint
    custom:
      customers:
        keys:
          - id
        columns:            # source column -> binding
          - id: id
          - full_name: name
          - email: email

Set TARGET_TYPE=NoSQL and CREATE_TABLE_IF_NOT_EXISTS=false: the HTTP Target has no tables to create. A run that moves a collection to another origin needs "force": true in the courier block, see above.

Request examples

With the customers endpoint above and the source row id=2, full_name='Clark Kent', email='clark@dailyplanet.com', active=false, credit=99.99:

Insert or update of the row

POST /customers/2 HTTP/1.1
Host: api.example.com
Authorization: Bearer s3cr3t
X-Operation: Update
X-Tenant: acme
Idempotency-Key: 3f9a…c41e
Content-Type: application/json

{"operation":"Update","name":"Clark Kent","email":"clark@dailyplanet.com","active":false,"credit":99.99}

Delete of the row (only the key is known)

POST /customers/2 HTTP/1.1
Host: api.example.com
Authorization: Bearer s3cr3t
X-Operation: Delete
X-Tenant:
Idempotency-Key: 9b07…e2d5
Content-Type: application/json

{"operation":"Delete","name":null,"email":null,"active":null,"credit":null}

On the delete the HEADER binding tenant takes no default, so X-Tenant is sent empty: only the PATH and QUERY bindings of a delete keep their defaults.

An endpoint orders defined as PUT /orders/{id} with body type FORM_URLENCODED and bindings id (PATH), region (QUERY), status and amount (BODY) delivers the row id=2, region='US', status='shipped', amount=99.90 as:

PUT /orders/2?region=US HTTP/1.1
Host: api.example.com
Authorization: Bearer s3cr3t
Idempotency-Key: 51cd…08fa
Content-Type: application/x-www-form-urlencoded

operation=Update&status=shipped&amount=99.90

Values are rendered as text: strings as they are (JSON-escaped in a JSON body), numbers in plain notation (0.0000000001, never 1E-10) with every digit they have, booleans as true/false, date and time types in ISO-8601, binary columns in Base64, arrays as JSON arrays. A NUMBER is not rounded or limited to a column size: only a number too long to write out, more than 1,000 characters of text or more than 10,000 digits in plain notation, fails the write. A value that is not a number (or not a boolean) in a NUMBER (or BOOLEAN) binding is sent as a string. Path and query values are URL-encoded like JavaScript’s encodeURIComponent.

Guidelines

Design the endpoint for upserts. Insert and update share the verb and the path, so an idempotent PUT/POST on the resource identified by the PATH binding is the natural fit; let the receiving side branch on the operation field when inserts and deletes need different handling.
Use one collection per API, one endpoint per source table. The collection carries base URL and credentials once; endpoints with the same name in different collections are not ambiguous, because the entity schema scopes the lookup.
Map only what the API needs. Unmapped source columns never leave Gluesync; a binding with a default value is a cheap way to add a constant (tenant, source system, API version) to every call.
Make PATH bindings required and keyed. The first required PATH binding is the entity key; it is what identifies the resource on deletes, when the row carries nothing else. Put a constant that the resource address needs (a tenant in the path) in a PATH or QUERY binding: a delete keeps their defaults, not those of HEADER and BODY bindings.
Mind the volume. The agent sends one request per row, sequentially per entity, and bulk mode is not available: size the snapshot of large tables accordingly, or snapshot through another channel and let the HTTP Target handle the change feed.
Expect retries. A failed call is retried from the checkpoint until the API answers 2xx, with the same payload and the same Idempotency-Key; make the endpoint idempotent, or honour the key, and use the entity error state and notifications to see what the API answered.
Answer a delete of a missing resource with 404 or 410. The HTTP Target counts it as delivered, so a retried delete does not block the entity.
Test against a mock first. A request recorder such as WireMock shows the exact method, URL, headers and body the agent sends; the pgsqlwal-http-target-integration-test demo kit does exactly that. A mock on the same host needs GS_COURIER_ALLOW_LOCAL_EGRESS=true, see Egress rule.