Apache CouchDB agent

Target Apache CouchDB

The Apache CouchDB target agent (agent tag couchdb) writes snapshots and CDC as JSON documents. One entity is one CouchDB database, under the fixed schema _default.

Prerequisites

To have Gluesync working on your Apache CouchDB instance as a target you will need:

Apache CouchDB 3.3, 3.4 or 3.5, a single node or a cluster;
A user that can write to the target databases. The agent reads the current revision of the documents it updates and writes, replaces and deletes documents, so the user needs read and write access to the database: it must be a member of it, or a database administrator;
A server administrator, to create databases. When Create database if missing is on (the default), the first write creates a database that does not exist, and CouchDB reserves that to server administrators. With a user that is not one, create the databases beforehand, or turn the setting off. The Truncate strategy RECREATE_DATABASE deletes and creates the database again, so it needs a server administrator too;
Network access from Core Hub to the HTTP port of the node (default 5984), or to the TLS port you use.

Setup via Web UI

Protocol: HTTP REST, the only one;
Node address: DNS record or IP address of one node of the server, without the scheme (couchdb.example.com, not couchdb.example.com);
Port: Optional, defaults to 5984;
Username: Username with the access listed in the prerequisites;
Password: Password belonging to the given username;
Disable authentication: Optional, defaults to false. Sends the requests with no authorization, for a server that allows anonymous access;
Enable TLS: Optional, defaults to false. When it is on, the node URLs use https:// and the server certificate is always validated. There is no option to trust it blindly;
CA certificate path and its password: Optional. A PKCS12 or JKS store, or a PEM file, that holds the certificate authority of the server. The certificate can also be uploaded through Core Hub. Blank means the default trust store of the JVM. A server with a certificate from a private CA needs this;
Additional nodes: Optional. host1:port1,host2:port2, the other nodes of a cluster, tried in order when a node is down. A node without a port takes the port above;
Databases (comma separated): Optional. The databases to offer when you browse the server with a user that may not list them, for example orders,customers.

Custom properties

Path prefix: (optional, defaults to empty) The path the server answers under, when it sits behind a reverse proxy, for example /couchdb;
Connect timeout (ms): (optional, defaults to 10000) How long a node may take to accept a connection before the next one is tried;
Request timeout (ms): (optional, defaults to 300000) How long a request may run, big _bulk_docs requests included.

Specific configuration

Setting Default Description

Setting	Default	Description
Database name prefix (`databaseNamePrefix`)	empty	Prepended to the name of every database the agent writes. See Database names.
Create database if missing (`createDatabaseIfMissing`)	`true`	Creates the target database on the first write when it does not exist. When it is off, a missing database fails the write with "Database Not Found".
Shards (q) (`shards`)	`8`	Number of shards (`q`) of a database that the agent creates. The main lever of the write throughput: see Throughput and shards. `0` keeps the default of the server.
Replicas (n) (`replicas`)	`0`	Number of copies of every shard (`n`) of a database that the agent creates. `0` keeps the default of the server.
Write batch size (`writeBatchSize`)	`1000`	Documents in one `_bulk_docs` request.
Write request size limit (bytes) (`writeMaxBytes`)	`8388608`	Size limit of the JSON of one `_bulk_docs` request (8 MiB). A document that is bigger than the limit goes alone.
Write concurrency (`writeConcurrency`)	`8`	`_bulk_docs` requests in flight at the same time. It also sizes the connection pool of the agent.
Compress requests (`compressRequests`)	`true`	Sends the request bodies gzip compressed.
Conflict retries (`conflictRetries`)	`5`	Rounds of lookup and write for the documents that changed under a write. See Write path.
Truncate strategy (`truncateStrategy`)	`DELETE_DOCUMENTS`	What the agent does when Core Hub truncates an entity: `DELETE_DOCUMENTS`, `RECREATE_DATABASE` or `SKIP`. See Truncate and snapshot deletion.

Database name prefix (databaseNamePrefix)

empty

Prepended to the name of every database the agent writes. See Database names.

Create database if missing (createDatabaseIfMissing)

true

Creates the target database on the first write when it does not exist. When it is off, a missing database fails the write with "Database Not Found".

Shards (q) (shards)

8

Number of shards (q) of a database that the agent creates. The main lever of the write throughput: see Throughput and shards. 0 keeps the default of the server.

Replicas (n) (replicas)

0

Number of copies of every shard (n) of a database that the agent creates. 0 keeps the default of the server.

Write batch size (writeBatchSize)

1000

Documents in one _bulk_docs request.

Write request size limit (bytes) (writeMaxBytes)

8388608

Size limit of the JSON of one _bulk_docs request (8 MiB). A document that is bigger than the limit goes alone.

Write concurrency (writeConcurrency)

8

_bulk_docs requests in flight at the same time. It also sizes the connection pool of the agent.

Compress requests (compressRequests)

true

Sends the request bodies gzip compressed.

Conflict retries (conflictRetries)

5

Rounds of lookup and write for the documents that changed under a write. See Write path.

Truncate strategy (truncateStrategy)

DELETE_DOCUMENTS

What the agent does when Core Hub truncates an entity: DELETE_DOCUMENTS, RECREATE_DATABASE or SKIP. See Truncate and snapshot deletion.

Entity settings

Database name override: (optional, defaults to empty) Replaces the table name as the database of the entity. The agent-wide Database name prefix still applies.

"Create table" in the UI generates the request that creates the database, PUT /<database>?q=<shards>&n=<replicas>, as a JSON statement. A q or n of 0 is left out, which keeps the default of the server.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "agentType": "TARGET",
    "agentTag": "couchdb",
    "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "couchdb-host-address",
        "port": 5984,
        "username": "gluesync",
        "password": "your_password",
        "disableAuth": false,
        "enableTls": false,
        "additionalHosts": "couchdb-node2:5984,couchdb-node3:5984"
    },
    "customHostCredentials": {
        "pathPrefix": "",
        "connectTimeoutMilliseconds": 10000,
        "requestTimeoutMilliseconds": 300000
    }
}'

certificatePath and certificatePassword go in hostCredentials when TLS needs a private certificate authority. additionalHosts is optional.

Setup specific configuration

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/specific' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "configuration": {
      "databaseNamePrefix": "",
      "createDatabaseIfMissing": true,
      "shards": 8,
      "replicas": 0,
      "writeBatchSize": 1000,
      "writeMaxBytes": 8388608,
      "writeConcurrency": 8,
      "compressRequests": true,
      "conflictRetries": 5,
      "truncateStrategy": "DELETE_DOCUMENTS"
    }
}'

The entity setting is databaseName in the configuration of the entity.

Database names

The database of an entity is named, in this order:

the Database name prefix of the agent, followed by
the Database name override of the entity, or the name of the table when there is no override.

The name is then made valid for CouchDB:

It is lowercased;
Every character outside a-z, 0-9, _, $, (, ), +, / and - becomes _;
When it does not start with a letter, gs_ is put in front of it.

The first write creates a missing database with PUT /<database>?q=<shards>&n=<replicas>, when Create database if missing is on. A 412 file_exists (someone else created it in the meantime) is not an error.

Document ids

When the entity has a key mapping, the id is the one that mapping composes. See Custom document keys;
Otherwise, when the key is a single column named _id, the document id is its value as it is;
Otherwise the id is made of the values of the key columns joined with ::, in key order, with no prefix: the database already is the collection (a key of 4 and eu gives 4::eu). A key column other than _id is also written as a field of the document.

CouchDB documents are replicated from a CouchDB source with their _id, so the ids stay the same on the target.

Field names

CouchDB reserves the names that start with _ for its own members (_id, _rev, _attachments, …). _bulk_docs rejects the whole request, not just the document, when a document has another top-level field that starts with _. For that reason the agent refuses a target column that starts with _ (other than _id) when the entity is set up, naming the column and suggesting another target column name. Rename the column in the entity, for example TRANSACTION_ID instead of _TRANSACTION_ID for a technical field.

Write path

The batch collapses to the last change of every document id, in the order of those last changes. Partial changes of the same document add up instead of replacing each other;
It is cut into requests of at most Write batch size documents and Write request size limit bytes of JSON;
At most Write concurrency requests are in flight. The ids are unique across them, so no request depends on another. The bodies are gzip compressed when Compress requests is on;
Each request is applied as follows:
- Insert. _bulk_docs without a revision. A document that already exists is a conflict, which is reported as a duplicate key, only when every rejected document is a conflict, so the insert conflict strategies of Core Hub apply. See Insert conflict strategy. Over a deleted document (a tombstone) the write succeeds: CouchDB accepts a write without a revision there;
- Update and upsert. The agent asks POST /<database>/_all_docs with the ids for their live revisions. A live document is written at its revision, and a tombstone or a missing document is written without one. Documents that come back conflict (they changed under the write) are looked up and written again, up to Conflict retries rounds;
- Delete. The same lookup. A document that is missing, or already a tombstone, is nothing to do: a delete is tolerant. The others are written as tombstones. Conflicts are retried like an upsert;
- Any other refusal of a document (forbidden from a validate_doc_update function, doc_validation, …) fails the batch with an error that names up to 10 ids and the reasons. The documents that were accepted stay written;
A server that is busy or restarting (408, 429, 500, 502, 503, 504), an unreachable node and a timeout are retried by the agent before the transaction is reported to Core Hub. Writes are idempotent by document id, so a retry is safe.

Optimistic snapshot upsert. When a snapshot writes with the upsert method into a database the agent created, or found empty at its first write, the lookup of revisions is skipped: the documents go out without a revision, and only the ones that conflict are looked up and written again. The mode ends for good, for that database in that process, when more than a tenth of a request conflicts. It is worth 50 to 80 percent of the throughput of an upsert snapshot.

Partial changes

A change that leaves a column out (changed columns only, a skipped null, a field that is absent) is merged into the stored document: the lookup returns the stored body, the written fields are laid over it, and the result is written at the stored revision. The fields that were left out keep their stored value, and so do the attachment stubs. When there is no stored document, it is created from the whole row image: the written fields plus the columns that were left out as unchanged, with the value the source holds.

A document written whole replaces the stored one, as in every document store, so its attachments are not kept.

Truncate and snapshot deletion

The Truncate strategy decides what happens when Core Hub truncates an entity, for example before a snapshot:

Strategy What happens

Strategy	What happens
`DELETE_DOCUMENTS` (default)	Pages `_all_docs` and deletes every document in bulk, except the design documents.
`RECREATE_DATABASE`	Reads the `_security` object, deletes the database, creates it again with the same `q` and `n`, and restores `_security`. The design documents are lost. It needs a server administrator.
`SKIP`	Logs it and leaves the data as it is.

DELETE_DOCUMENTS (default)

Pages _all_docs and deletes every document in bulk, except the design documents.

RECREATE_DATABASE

Reads the _security object, deletes the database, creates it again with the same q and n, and restores _security. The design documents are lost. It needs a server administrator.

SKIP

Logs it and leaves the data as it is.

Throughput and shards

The number of shards (q) of the target database is the main lever of the write throughput. A database that the agent creates gets Shards (q), 8 by default. The second lever is the number of requests in flight, Write concurrency.

Measured on CouchDB 3.5.1, on a single node (n=1), in a laboratory on Apple silicon with Docker, with 1,000 documents per _bulk_docs request:

Shards (`q`)	Requests in flight	Documents per second
2 (the server default)	1	7,200
2	4	13,300
2	8	15,000
8	8	41,000
16	16	55,000
16	32	64,000
32	32	81,000

Gzip on the request body raised the 8 requests in flight on 2 shards from 15,000 to 18,300 documents per second.

Writing 200,000 documents of about 0.9 KB through the agent into a database with 8 shards, 8 requests in flight, gives about 34,000 documents per second for inserts, 34,500 for an upsert snapshot in optimistic mode and 19,000 for an upsert that looks up the revisions first.

The numbers depend on the machine and on what else it runs. To write faster, create the target databases with more shards, or create them beforehand with the q you want, and raise Write concurrency together with the shards.

Data types

CouchDB stores JSON, so the type of a field is a JSON shape. The first column below is the type of the source column, the second one the type Gluesync gives a new field by default, and the third the alternatives you can choose for a column of that Gluesync type. A column whose type was never set is written in the natural form of its value. null is written as an explicit JSON null.

CouchDB stores numbers as IEEE doubles. An integer keeps every digit up to what a JSON reader reads back. A decimal does not: 12345678901234567890.123456789 is read back as 12345678901234567000.0. For that reason a BIG_DECIMAL is written as a string by default.

Gluesync type Default Alternatives Value written

Gluesync type	Default	Alternatives	Value written
`STRING`	`string`	`json`	The text. With `json`, the text is parsed and the JSON value is written: a text that is not JSON fails the row and names the column.
`BYTE_ARRAY`	`base64`		Base64 text.
`SHORT`, `INT`, `LONG`	`integer`	`string`	A number, or its text.
`FLOAT`, `DOUBLE`	`number`	`string`	A number. `NaN` and the infinities are always written as text, because JSON has no literal for them.
`BIG_DECIMAL`	`string`	`number`	As a string, `toPlainString()`. As a number, `BigDecimal.toString()` with the exponent kept, which CouchDB stores as a double: about 15 significant digits.
`BOOLEAN`	`boolean`		`true` or `false`.
`LOCAL_DATE`	`string`	`epoch_millis`	`2026-10-21`, or the milliseconds of midnight UTC.
`LOCAL_TIME`, `OFFSET_TIME`	`string`		ISO text.
`LOCAL_DATE_TIME`	`string`	`epoch_millis`	ISO text, or the milliseconds read as UTC.
`OFFSET_DATE_TIME`	`string`	`epoch_millis`	ISO text with the offset, or the milliseconds.
`ARRAY`	`array`	`string`	A JSON array, element by element, or its JSON text.
`MAP`	`object`	`string`	A JSON object, or its JSON text.

STRING

string

json

The text. With json, the text is parsed and the JSON value is written: a text that is not JSON fails the row and names the column.

BYTE_ARRAY

base64

Base64 text.

SHORT, INT, LONG

integer

string

A number, or its text.

FLOAT, DOUBLE

number

string

A number. NaN and the infinities are always written as text, because JSON has no literal for them.

BIG_DECIMAL

string

number

As a string, toPlainString(). As a number, BigDecimal.toString() with the exponent kept, which CouchDB stores as a double: about 15 significant digits.

BOOLEAN

boolean

true or false.

LOCAL_DATE

string

epoch_millis

2026-10-21, or the milliseconds of midnight UTC.

LOCAL_TIME, OFFSET_TIME

string

ISO text.

LOCAL_DATE_TIME

string

epoch_millis

ISO text, or the milliseconds read as UTC.

OFFSET_DATE_TIME

string

epoch_millis

ISO text with the offset, or the milliseconds.

ARRAY

array

string

A JSON array, element by element, or its JSON text.

MAP

object

string

A JSON object, or its JSON text.

The types the agent reads from a CouchDB source are in Data types of the source.

HTTP errors

The agent titles the failures of the server as follows:

Condition Title

Condition	Title
No node answers, after every node was tried	Server Unreachable
HTTP 401	Authentication Failed
HTTP 403	Permission Denied
HTTP 404	Database Not Found
HTTP 400	Request Rejected
HTTP 409	Document Conflict
HTTP 429	Server Overloaded
HTTP 5xx	Server Error
A `_bulk_docs` write whose only rejections are conflicts	Document Already Exists
Any other rejection in a `_bulk_docs` write	Documents Rejected (with the first ids and reasons)

No node answers, after every node was tried

Server Unreachable

HTTP 401

Authentication Failed

HTTP 403

Permission Denied

HTTP 404

Database Not Found

HTTP 400

Request Rejected

HTTP 409

Document Conflict

HTTP 429

Server Overloaded

HTTP 5xx

Server Error

A _bulk_docs write whose only rejections are conflicts

Document Already Exists

Any other rejection in a _bulk_docs write

Documents Rejected (with the first ids and reasons)

Notes

Attachments are neither read nor written, and a document written whole replaces the stored one, attachments included. A partial change keeps them;
A purged document (_purge) leaves no change in a CouchDB source, so a pipeline from CouchDB does not delete it on the target;
The agent has no recursion protection for a CouchDB to CouchDB pipeline;
Partitioned databases are written as normal ones: the agent uses no partition-specific API;
RECREATE_DATABASE loses the design documents of the database, and DELETE_DOCUMENTS leaves them;
A _bulk_docs request is not atomic: CouchDB accepts some documents and rejects others, and the accepted ones stay written.

Troubleshooting

The server answers 401 or 403

401 is shown as "Authentication Failed": check the username and the password, or turn Disable authentication on for a server that allows anonymous access. 403 is shown as "Permission Denied": the user cannot write to the database, or cannot create it. Add the user to the members of the database, or use a server administrator to create the databases.

The database is not found

404 is shown as "Database Not Found". Turn Create database if missing on, with a user that is a server administrator, or create the database beforehand with the name the agent computes (see Database names). Check the Path prefix when the server sits behind a reverse proxy.

A column holds text that is not valid JSON

A column whose type is json was given a text that is not JSON, and the error names the column. Fix the value in the source, or choose string for that column. When CouchDB itself refuses a whole request (400, which it reports as invalid UTF-8 JSON for any body it cannot parse), the error names the database and points at the columns of type json.

The write refuses a field name

A target column that starts with _ (other than _id) is refused when the entity is set up, because _bulk_docs would reject every request that carries it. Rename the target column.

Conflicts are still changing after the retries

The error says that documents were still changing after the Conflict retries rounds. Another writer is updating the same documents while Gluesync writes them. Stop that writer, or raise Conflict retries.

Documents are rejected

The error names up to 10 ids and the reasons. forbidden comes from a validate_doc_update function of the database that refuses the document. doc_validation means that the document is not valid for CouchDB, for example for a field that starts with _.

The writes are slow

Create the target databases with more shards (Shards (q)), and raise Write concurrency with them. See Throughput and shards. The shards of a database cannot be changed by the agent once the database exists: create it beforehand with the q you want. Check that Compress requests is on, and that the server is not busy with other work.