Couchbase DCP source agent

Source data from Couchbase (DCP)

The Couchbase DCP agent is a source-only connector. It reads Couchbase’s native Database Change Protocol (DCP) stream through dcp-client. The parent module supplies shared configuration, snapshot, and discovery behavior.

The connection fields below mirror the shared Couchbase connection form. The DCP behavior flag is verified from the shared CouchbaseConfiguration.

Prerequisites

To have Gluesync perform CDC from Couchbase via DCP you will need to have:

Valid user credentials that can open a DCP stream on the source bucket. Unlike the Eventing source, this agent does not require the Couchbase Eventing service, does not deploy Eventing functions, and writes nothing to the source bucket.
- Enterprise Edition: assign the Data DCP Reader role (data_dcp_reader) on the source bucket, or on the scope/collection you stream. Examples: data_dcp_reader[myBucket], data_dcp_reader[myBucket:myScope], or data_dcp_reader[myBucket:myScope:myCollection]. That role can start DCP streams and read data/XATTRs; it cannot write.
- Community Edition: granular roles such as data_dcp_reader are Enterprise-only. CE users can only be assigned bucket_full_access, admin, or ro_admin. Use bucket_full_access on the source bucket (or a broader admin role) so the agent can open the DCP stream.
Couchbase Server reachable from the agent:
- With scopes and collections: Couchbase 7.0+;
- Without scopes and collections: Couchbase 6.x is supported.
Community Edition (CE) and Enterprise Edition (EE) are supported; DCP is the native stream and works without the Eventing service.
TLS, Capella, rebalance, and failover behavior have been verified by the agent test harness.
Following ports opened on your Couchbase Server cluster for DCP/KV connectivity (see Couchbase Server Ports). Do not assume the Eventing service port (8096 / 18096):
- 8091 (management / bootstrap REST API); use 18091 when TLS or Capella is enabled;
- 11210 (Data Service / KV, unencrypted) on every Data Service node the agent discovers;
- 11207 (Data Service / KV over TLS) when TLS is enabled.
  
  The connection form defaults the management port to 8091 (18091 for TLS / Capella). Capella or NAT deployments may advertise alternate KV ports; confirm the live map with /pools/default/nodeServices before opening firewalls.

Setup via Web UI

The following fields mirror the existing Couchbase Eventing source connection form. Confirm in the Control Plane that the agent picker shows Couchbase DCP agent (couchbase-dcp) as a source only.

Hostname / IP Address: DNS SVR record of your Couchbase Server cluster or IP Address of one of the nodes (automatic discovery of all other nodes is then applied). For Capella users, put the given hostname like cb.XYZ.cloud.couchbase.com;
Port: Optional, defaults to 8091, use 18091 for Couchbase Capella DBaaS and secured connections;
Database name: Name of your source bucket;
Username: Username with Data DCP Reader access on Enterprise Edition, or bucket_full_access (or equivalent CE role) on Community Edition;
Password: Password belonging to the given username;
Enable Tls: (optional, defaults to false) Enable or disable the usage of TLS encryption;
Tls certificates: (optional) File browser to let you upload your certificates;
Use Scope and Collections (useScopesAndCollections): Controls how the DCP stream is filtered:
- On: Applies a server-side collection filter. Requires Couchbase 7.0+;
- Off: Reads the whole-bucket stream. Couchbase 6.x is supported.

Custom host credentials

No DCP-specific custom host credentials are documented. The agent uses the shared Couchbase host credentials shown above.

Specific configuration

The key behavioral flag is Use Scope and Collections (useScopesAndCollections):

true: Filter DCP on the server by scope and collection (Couchbase 7.0+);
false: Read the whole bucket (Couchbase 6.x supported).

Do not copy Eventing-only keys (scriptTimeoutSeconds, Eventing worker counts, Eventing language compatibility) onto this agent.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

The hostCredentials object below mirrors the shared Couchbase source connection.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "host-address",
        "port": 8091,
        "databaseName": "myBucket",
        "username": "",
        "password": "",
        "enableTls": true,
        "certificatePath": "/myPath/cert.pem"
        }
}'

Setup specific configuration

Set useScopesAndCollections according to the Couchbase version and stream behavior described above. Eventing settings such as workersNumber and scriptTimeoutSeconds do not apply.

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/specific' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "configuration": {
    "useScopesAndCollections": true
  }
}'

Native DCP stream (no Eventing)

This agent consumes Couchbase’s native DCP change stream. You do not deploy the Eventing service or Gluesync Eventing functions.

Works on Community Edition and Enterprise Edition.
Scopes and collections require Couchbase 7.0+ and use a server-side collection filter.
Bucket-only (no scopes/collections) reads the whole-bucket stream and is supported on Couchbase 6.x.
Checkpoint sizing comes from the Couchbase cluster map. The vbucket count is not hard-coded to 1024; for example, a Couchbase 8.x Magma bucket can report 128 vbuckets.
TLS, Capella, rebalance, and failover have been verified.

Entity and deletion matching without collections

When Use Scope and Collections is off, the DCP agent reads a whole-bucket stream:

Documents are matched to entities through the document type field. Snapshot matching uses the equivalent WHERE type == "…" condition.
Deletion events have no document body, so they are matched through the document id/key pattern.
A deletion is admitted only when its key is the entity metaId or can be decomposed using documentKeyResolution.
A multi-entity setup that uses a plain metaId key is rejected during setup by CheckDcpKeyResolutionComponentSetup, because deletion events cannot otherwise be assigned safely to an entity.

For Eventing-based CDC, see Change data capture (Eventing).

Troubleshooting

`Could not open the Couchbase DCP stream` / `not using TLS`

Symptom: the DCP source fails while opening the stream. The log line looks like:

Could not open the Couchbase DCP stream on <source-host> (bucket myBucket), not using TLS. A bootstrap failure looks the same whether the KV port is unreachable or the cluster certificate is not trusted, so check both.

The phrase "not using TLS" is the Plaintext mode label from DcpClientSecurity (dcpTrustSource / Plaintext.describe() = "not using TLS"). It reports that the DCP client is in plaintext mode. It does not mean TLS is forced, and it does not mean the cluster rejected a non-TLS connection.

Work through the causes in this order. Missing privileges on the source bucket are the usual real cause; they often surface looking like a TLS or bootstrap failure.

RBAC / permissions on the source bucket. The user must be able to open a DCP stream and read data.
- Enterprise Edition: data_dcp_reader (Data DCP Reader) on the streamed bucket, or on the scope/collection.
- Community Edition: bucket_full_access (or admin). Granular DCP roles are EE-only.
  
  See Couchbase roles. Re-check after any role change; a leftover bootstrap error can still mention "not using TLS".
Reachability from Core Hub to Couchbase. Confirm KV 11210 and management 8091 from the host that runs the agent (use 11207 / 18091 only when TLS is actually enabled). Eventing ports (8096 / 18096) are not required. See Couchbase Server Ports.
Leftover cert / truststore forcing TLS. For on-prem plaintext, the host must be an IP or couchbase://…, not couchbases://. Drop leftover certificates / truststore material from a previous TLS setup if Enable Tls is off.
- Multi-entity setup with a plain metaId key is rejected: Configure a decomposable document key through documentKeyResolution, or use an entity layout in which deletion keys can be matched unambiguously.