Apache CouchDB Agent for Gluesync: Features Overview

Index

Source data from Apache CouchDB

Target Apache CouchDB

Agent changelog

Apache CouchDB source advanced settings

Apache CouchDB target advanced settings

Core principles

Gluesync’s Apache CouchDB agent talks to CouchDB over its HTTP API with JSON bodies. It uses no vendor SDK, the same approach as the Elasticsearch agent. One agent works as a source and as a target, on a single node or on a cluster. When you list the other nodes of a cluster, the agent tries them in order if the node it talks to is down.

One entity is one CouchDB database. CouchDB has no level above the database, so every entity sits under one fixed schema called _default. The fields of a database are not declared anywhere, so the agent infers them from a sample of its documents.

Change data capture

The source agent copies a database with a snapshot and then follows its _changes feed. It creates nothing in your databases and needs no change to your applications.

The snapshot lists the document ids from the changes feed and reads the documents with many parallel _bulk_get requests. It never pages _all_docs, which is bound to one stream on the server and does not scale. An interrupted snapshot resumes from the feed position it had safely reached.
CDC polls the changes feed of every entity from the position saved by Core Hub, in long poll mode. Changes made while the snapshot ran are replayed from the position the snapshot started at.
A deletion reaches the target as a delete that carries only the _id of the document, so the entity key must be _id.

To read more, see How it works.

Writing to CouchDB

The target agent writes every batch with parallel _bulk_docs requests. It looks up the current revision of the documents it updates, retries the documents that changed under the write, writes deletions as tombstones and merges partial changes into the stored document. It creates a missing database with the number of shards you choose, which is the main lever of the write throughput. See Throughput and shards.

Security

The agent connects with HTTP Basic authentication, or anonymously when you disable authentication. With TLS on, the server certificate is always validated: there is no option to trust it blindly. A server with a certificate issued by a private certificate authority is trusted by giving the agent that CA certificate. Turn TLS on for any CouchDB that is not on a private network.

Measured performance

Measured on CouchDB 3.5.1 on a single node, in a laboratory on Apple silicon with Docker:

The snapshot read reaches 94,000 documents per second with 16 parallel _bulk_get requests, against 28,000 per second for _all_docs?include_docs=true, which does not scale with parallel requests;
The write reaches 15,000 documents per second on a database with 2 shards and 81,000 on one with 32 shards. See Throughput and shards;
A whole 10-million-document database was copied from CouchDB to PostgreSQL, end to end, in 274 seconds.

The numbers depend on the hardware and on the load of the server.

Supported versions

Gluesync works on Apache CouchDB 3.3, 3.4 and 3.5, on a single node or on a cluster. The agent checks the server when it connects and refuses a server that is not CouchDB 3.x. The integration tests run against CouchDB 3.5.1.

Requirements

Network access from Core Hub to the HTTP port of the CouchDB node, 5984 by default, or to the TLS port you use;
For a source, a user that is a member of every database to replicate, with read access to its documents. Listing the databases with _all_dbs needs a server administrator in CouchDB 3.x: otherwise name the databases in the connection;
For a target, a user that can write to the target databases. Creating a missing database needs a server administrator. See Prerequisites.

Platform supported features

Feature Support Notes

Feature	Support	Notes
Target role	✅	Writes documents with parallel `_bulk_docs` requests over the HTTP JSON API.
Source role	✅	Reads the databases with `_changes` and `_bulk_get`.
Snapshots	✅	Ids from the changes feed, documents from parallel `_bulk_get` requests. The snapshot resumes from the feed position it had safely reached.
CDC	✅	Inserts, updates and deletes of documents, read from the `_changes` feed.
CDC type	Changes feed (long poll)	The position is the opaque `seq` of the feed of each database. Delivery is at least once, and the target applies a change twice without harm.
User defined data types	N/A	CouchDB documents are JSON: a field is a string, a number, a boolean, an object or an array.
Bulk load	N/A	There is no separate bulk mode: every batch already travels as one `_bulk_docs` request.
Database pooling	✅	One HTTP client for the whole agent. The size of its connection pool follows Write concurrency.
Source persisted internal cache	N/A	The agent reads the changes feed of each entity directly, and Core Hub keeps the confirmed position.
Target persisted internal cache	N/A	Writes go straight to CouchDB.
Driver / connector	HTTP JSON	Ktor HTTP client with JSON. Not a vendor SDK.
Write policies	✅	A change that leaves a column out is merged into the stored document. See Partial changes.
Before image	❌	The changes feed carries the current document only.
Insert conflict strategy	✅	A document that already exists is reported as a duplicate key. See Insert conflict strategy.
Chains	N/A	Not supported being it a NoSQL database.
Truncate operation support (reading)	N/A	CouchDB has no truncate: a deleted document is a delete change.
Truncate operation support (writing)	✅	Honors the Core Hub “TRUNCATE before snapshot” option with the Truncate strategy of the agent: `DELETE_DOCUMENTS` (default), `RECREATE_DATABASE` or `SKIP`.

Target role

✅

Writes documents with parallel _bulk_docs requests over the HTTP JSON API.

Source role

✅

Reads the databases with _changes and _bulk_get.

Snapshots

✅

Ids from the changes feed, documents from parallel _bulk_get requests. The snapshot resumes from the feed position it had safely reached.

CDC

✅

Inserts, updates and deletes of documents, read from the _changes feed.

CDC type

Changes feed (long poll)

The position is the opaque seq of the feed of each database. Delivery is at least once, and the target applies a change twice without harm.

User defined data types

N/A

CouchDB documents are JSON: a field is a string, a number, a boolean, an object or an array.

Bulk load

N/A

There is no separate bulk mode: every batch already travels as one _bulk_docs request.

Database pooling

✅

One HTTP client for the whole agent. The size of its connection pool follows Write concurrency.

Source persisted internal cache

N/A

The agent reads the changes feed of each entity directly, and Core Hub keeps the confirmed position.

Target persisted internal cache

N/A

Writes go straight to CouchDB.

Driver / connector

HTTP JSON

Ktor HTTP client with JSON. Not a vendor SDK.

Write policies

✅

A change that leaves a column out is merged into the stored document. See Partial changes.

Before image

❌

The changes feed carries the current document only.

Insert conflict strategy

✅

A document that already exists is reported as a duplicate key. See Insert conflict strategy.

Chains

N/A

Not supported being it a NoSQL database.

Truncate operation support (reading)

N/A

CouchDB has no truncate: a deleted document is a delete change.

Truncate operation support (writing)

✅

Honors the Core Hub “TRUNCATE before snapshot” option with the Truncate strategy of the agent: DELETE_DOCUMENTS (default), RECREATE_DATABASE or SKIP.

Q&A

Is CouchDB 2.x supported? No. The agent is built and tested for CouchDB 3.3, 3.4 and 3.5, and refuses a server that is not CouchDB 3.x.

I have got Couchbase, is it supported? Couchbase is a different product and has its own agents. See Couchbase.

Does the agent change my database? As a source, no: it only reads databases and documents. As a target it creates a database that does not exist, when Create database if missing is on, and writes the documents.

Can I replicate from CouchDB to CouchDB? Yes, a pipeline can have a CouchDB source and a CouchDB target. A technical field such as _TRANSACTION_ID starts with an underscore, which CouchDB refuses in a document, so map it to a name that does not.

Can I use two-way sync between two CouchDB databases? The agent has no recursion protection yet, so two-way sync between CouchDB databases would send each change back and forth. Use one direction.

Are attachments replicated? No. Attachments are neither read nor written. See Notes.