HTTP Source agent

Core Hub 2.3.2.1 or above, which ships the HTTP agent and the Courier module;

A Core Hub user who can manage connections (a Manager or a Super Admin) to define the collections and endpoints in Courier;

A pipeline whose source is the HTTP agent (the HTTP Source) and whose target is any Gluesync target agent;

The HTTP Source listening port (8090 by default) reachable by the clients on the host that runs Core Hub; open it in the firewall, or publish it from the container, for example -p 8090:8090;

A TLS keystore for the listener, or a TLS terminator in front of it: without one the listener refuses to serve plain HTTP on any address but loopback, unless inboundAllowPlainHttp is set, see Security;

A Personal API Token for each client, owned by a Manager or a Super Admin, carrying the HTTPS agent permission scope (gluesync.https_agent) and either no agent scope or the one of the pipeline and the HTTP Source agent. Core Hub refuses to save a collection that no active token can call, so create the first token before the collection, see Personal API Token;

Enough local disk for the queue: up to maxTotalBacklogMb (8 GB by default) in the http_cache directory under the agent’s data directory, and minFreeDiskMb (1 GB by default, or 5% of the volume if that is more) kept free.

it has the shape of a Personal API Token, gsp_ followed by 64 lowercase hexadecimal digits: any other value is refused at once, without asking Core Hub, and never cached;

it is valid (not revoked, not expired) and its owner still exists;

the role of its owner may write, as in Query Studio: a Manager or a Super Admin. A token of a Viewer or a Monitor is refused with 403, whatever its scopes;

it carries the HTTPS agent permission scope (gluesync.https_agent), or no permission restriction at all;

its agent scopes include the endpoint’s pipeline and HTTP Source agent, or it has no agent scope at all;

the collection’s blocklist and allowlist admit it.

The blocklist (blockedTokenIds) holds the tokens that can never call the collection, whether the allowlist is on or off and whatever their scopes. It wins over everything else. Blocking a token leaves its other collections alone, and you can unblock it later. A token cannot be on the blocklist and on the allowlist at once: Core Hub refuses to save such a collection.

The allowlist (allowlistEnabled and allowedTokenIds) narrows the other tokens:

a token that is valid is remembered for at most one minute, and never past the token’s own expiry;

a token that is unknown, revoked or expired is remembered for 5 seconds, enough to spare Core Hub a client that retries a revoked token in a loop;

a refusal for a missing permission scope, an agent scope or the owner’s role is not remembered: the token is real, its owner may be granted what it lacks a moment later, and the answer is cheap to ask for again.

Body: at most maxBodyKb (1 MB by default), checked on the Content-Length before the body is read, and while reading a chunked body.

Concurrency: at most maxInFlight requests (64) are served at once; one more gets 503 at once instead of waiting on a pool. The server uses its own serverThreads (16), separate from Core Hub’s.

Time: a request is served for 60 seconds at most, its body included, and a body that arrives slower than 128 bytes per second is dropped, so a client that trickles its body cannot hold one of the maxInFlight slots for long. Both are answered 408.

Queue per endpoint: beyond maxBacklogMb (2 GB) of unconfirmed rows for one endpoint, that endpoint answers 503 with Retry-After: 5; the others keep accepting, so one stalled entity does not turn the others away.

Queue in total: beyond maxTotalBacklogMb (8 GB) of space in use by the queue files, every endpoint answers 503 with Retry-After: 5. A queue file reuses its space only once it has drained, so under a constant flow the limit can be reached for a moment even with a fast target: the 503s are brief, the file recycles and requests flow again.

Disk: while the free space of the queue’s directory is below minFreeDiskMb (1 GB by default) or 5% of the volume, whichever is more, every endpoint answers 503 with Retry-After: 5, so the queue never fails on a full disk halfway through a row.

Connections: acceptQueueSize (1024) connections wait in the operating system before being accepted by acceptors (2) threads; idle connections are closed after 30 seconds.

Delivery: one poll drains up to 10 full batches or one second of rows, and Core Hub polls every 500 ms by default, so delivery keeps up with sustained intake.

Plain HTTP only on loopback, or on your say-so. Without a keystore, tokens and data would travel unencrypted, so the listener does not start on an address other than loopback (localhost, ::1, 127.x.x.x) unless inboundAllowPlainHttp is true, and then it logs a warning. Either set inboundTlsKeystorePath and inboundTlsKeystorePassword (and inboundTlsKeyAlias if the keystore holds several keys) so the listener serves HTTPS only, or put the listener behind a TLS terminator (reverse proxy, load balancer) and set inboundBindHost to 127.0.0.1, or to a private interface together with inboundAllowPlainHttp. The certificate must name the host names or IP addresses the clients use: the server checks the requested host against it and answers 400 otherwise.

Bind host. The listener binds every interface (0.0.0.0) by default; restrict it with inboundBindHost and the firewall.

Credentials never reach the target. Authorization, Proxy-Authorization and Cookie cannot be bound, so the caller’s token or session never ends up in a table.

Least privilege. Give each client its own token with only the HTTPS agent permission and the agent scope of its HTTP Source, owned by a user whose role you would let write, with an expiry. Turn the collection allowlist on to restrict a collection to named tokens, and block a token you retire from one collection without revoking it everywhere.

Routes stay hidden. A request that matches no endpoint is answered 401 unless it carries a token the HTTP Source accepts, so the collections and endpoints cannot be listed before authenticating.

Nothing sensitive in logs or answers. Refused requests are logged without the body, the values or the token, with control characters stripped. Error answers never carry internal details: not Core Hub’s address, an exception message or a path on disk, and when Core Hub’s own answer cannot be used the client gets the fixed message Upstream error.

Slow clients. The time a request may take and the minimum speed of its body are bounded, so clients that trickle bytes cannot turn everyone else away.

Proxies. A JSON body that repeats a key is refused, so a proxy or WAF in front that checks the first occurrence cannot be bypassed with a second one.

http_source_requests_total{endpoint,status}: requests per endpoint name and status code (unmatched for requests matching no endpoint). The status tag tells 202, 400, 401, 403, 404, 408, 413 and 503 apart.

Entity lag: the rows and bytes queued and not yet confirmed, and the age of the oldest one, are reported as the entity lag (gluesync_cache_* metrics) on the entity monitoring.

Read checkpoint: the sequence counter is the agent’s read checkpoint (gluesync_read_checkpoint_* metrics), saved every second when it moves and every minute otherwise, so its age stays meaningful on an idle agent.

Logs: refused requests are not logged one by one. One WARN line per minute sums them up by status code, and a recurring failure is logged in full once a minute, then counted. Per-request details are at DEBUG level.

No deduplication of retried requests. The HTTP Source does not read an Idempotency-Key header: a client that retries a request it already got 202 for queues a second row. On an endpoint with a key the second row rewrites the same target row; on an append-only endpoint (_seq key) it is a duplicate. (The HTTP Target does send the header toward the APIs it calls.)

No delivery status endpoint. The 202 confirms the row is queued, not written: follow delivery through the entity monitoring.

Writes only. LIST and READ endpoints stay on Core Hub’s SQL path against a SQL-capable source agent; a source cannot answer a read.

One port per agent instance. Each HTTP Source instance in Core Hub listens on its own inboundPort.

Fixed column sizes. Strings up to 255 characters and numbers up to NUMERIC(18,2); larger values are refused with 400.

No partial updates. PATCH is not available: an Update replaces the whole row.

Ordering holds per endpoint, not across endpoints.

Give every endpoint a key when the target has one. A key binding (or a required PATH binding) makes updates, deletes and client retries address the same row; keep _seq for append-only targets.

Use one endpoint with an operation binding when a client sends inserts, updates and deletes of the same resource: its rows carry every column and keep the per-endpoint order.

Retry on 503, 408 and on timeouts, not on other 4xx. Honour Retry-After; a 400 does not change by repeating it. A request that got no 202 may still have been queued, so make the retry harmless: give the endpoint a key.

Turn on TLS or put a terminator in front before exposing the port outside a trusted network.

Size the disk for outages. The queue absorbs a target outage up to maxTotalBacklogMb; at your intake rate, that is how long the target can be down before clients get 503.

Test with Preview the row first, then with curl, before pointing production clients at the endpoint.