Static Couchbase DCP source agent

Source data from Couchbase (DCP)

The Couchbase DCP agent is a source-only connector. It reads Couchbase’s native Database Change Protocol (DCP) stream through dcp-client. The parent module supplies shared configuration, snapshot, and discovery behavior.

The connection fields below mirror the shared Couchbase connection form. The DCP behavior flag is verified from the shared CouchbaseConfiguration.

Prerequisites

To have Gluesync perform CDC from Couchbase via DCP you will need to have:

  • Valid user credentials that can open a DCP stream on the source bucket. Unlike the Eventing source, this agent does not require the Couchbase Eventing service, does not deploy Eventing functions, and writes nothing to the source bucket.

    • Enterprise Edition: assign the Data DCP Reader role (data_dcp_reader) on the source bucket, or on the scope/collection you stream. Examples: data_dcp_reader[myBucket], data_dcp_reader[myBucket:myScope], or data_dcp_reader[myBucket:myScope:myCollection]. That role can start DCP streams and read data/XATTRs; it cannot write.

    • Community Edition: granular roles such as data_dcp_reader are Enterprise-only. CE users can only be assigned bucket_full_access, admin, or ro_admin. Use bucket_full_access on the source bucket (or a broader admin role) so the agent can open the DCP stream.

  • Couchbase Server reachable from the agent:

    • With scopes and collections: Couchbase 7.0+;

    • Without scopes and collections: Couchbase 6.x is supported.

  • Community Edition (CE) and Enterprise Edition (EE) are supported; DCP is the native stream and works without the Eventing service.

  • TLS, Capella, rebalance, and failover behavior have been verified by the agent test harness.

  • Following ports opened on your Couchbase Server cluster for DCP/KV connectivity (see Couchbase Server Ports). Do not assume the Eventing service port (8096 / 18096):

    • 8091 (management / bootstrap REST API); use 18091 when TLS or Capella is enabled;

    • 11210 (Data Service / KV, unencrypted) on every Data Service node the agent discovers;

    • 11207 (Data Service / KV over TLS) when TLS is enabled.

      The connection form defaults the management port to 8091 (18091 for TLS / Capella). Capella or NAT deployments may advertise alternate KV ports; confirm the live map with /pools/default/nodeServices before opening firewalls.

Setup via Web UI

The following fields mirror the existing Couchbase Eventing source connection form. Confirm in the Control Plane that the agent picker shows Couchbase DCP agent (couchbase-dcp) as a source only.

  • Hostname / IP Address: DNS SVR record of your Couchbase Server cluster or IP Address of one of the nodes (automatic discovery of all other nodes is then applied). For Capella users, put the given hostname like cb.XYZ.cloud.couchbase.com;

  • Port: Optional, defaults to 8091, use 18091 for Couchbase Capella DBaaS and secured connections;

  • Database name: Name of your source bucket;

  • Username: Username with Data DCP Reader access on Enterprise Edition, or bucket_full_access (or equivalent CE role) on Community Edition;

  • Password: Password belonging to the given username;

  • Enable Tls: (optional, defaults to false) Enable or disable the usage of TLS encryption;

  • Tls certificates: (optional) File browser to let you upload your certificates;

  • Use Scope and Collections (useScopesAndCollections): Controls how the DCP stream is filtered:

    • On: Applies a server-side collection filter. Requires Couchbase 7.0+;

    • Off: Reads the whole-bucket stream. Couchbase 6.x is supported.

Custom host credentials

No DCP-specific custom host credentials are documented. The agent uses the shared Couchbase host credentials shown above.

Specific configuration

The key behavioral flag is Use Scope and Collections (useScopesAndCollections):

  • true: Filter DCP on the server by scope and collection (Couchbase 7.0+);

  • false: Read the whole bucket (Couchbase 6.x supported).

Do not copy Eventing-only keys (scriptTimeoutSeconds, Eventing worker counts, Eventing language compatibility) onto this agent.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

The hostCredentials object below mirrors the shared Couchbase source connection.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "host-address",
        "port": 8091,
        "databaseName": "myBucket",
        "username": "",
        "password": "",
        "enableTls": true,
        "certificatePath": "/myPath/cert.pem"
        }
}'

Setup specific configuration

Set useScopesAndCollections according to the Couchbase version and stream behavior described above. Eventing settings such as workersNumber and scriptTimeoutSeconds do not apply.

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/specific' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "configuration": {
    "useScopesAndCollections": true
  }
}'

Native DCP stream (no Eventing)

This agent consumes Couchbase’s native DCP change stream. You do not deploy the Eventing service or Gluesync Eventing functions.

  • Works on Community Edition and Enterprise Edition.

  • Scopes and collections require Couchbase 7.0+ and use a server-side collection filter.

  • Bucket-only (no scopes/collections) reads the whole-bucket stream and is supported on Couchbase 6.x.

  • Checkpoint sizing comes from the Couchbase cluster map. The vbucket count is not hard-coded to 1024; for example, a Couchbase 8.x Magma bucket can report 128 vbuckets.

  • TLS, Capella, rebalance, and failover have been verified.

Entity and deletion matching without collections

When Use Scope and Collections is off, the DCP agent reads a whole-bucket stream:

  • Documents are matched to entities through the document type field. Snapshot matching uses the equivalent WHERE type == "…​" condition.

  • Deletion events have no document body, so they are matched through the document id/key pattern.

  • A deletion is admitted only when its key is the entity metaId or can be decomposed using documentKeyResolution.

  • A multi-entity setup that uses a plain metaId key is rejected during setup by CheckDcpKeyResolutionComponentSetup, because deletion events cannot otherwise be assigned safely to an entity.

For Eventing-based CDC, see Change data capture (Eventing).

Troubleshooting

  • Multi-entity setup with a plain metaId key is rejected: Configure a decomposable document key through documentKeyResolution, or use an entity layout in which deletion keys can be matched unambiguously.

  • Authentication or DCP stream open failures: On Enterprise Edition, confirm the user has data_dcp_reader on the streamed bucket/scope/collection. On Community Edition, confirm bucket_full_access (or admin). See Couchbase roles.

  • Connection timeouts after bootstrap: Confirm 11210 (or 11207 with TLS) is reachable on every Data Service node, not only the management port 8091 / 18091. Eventing ports are not required. See Couchbase Server Ports.