Azure Cosmos DB CDC with Gluesync: Setup & Configuration
How change data capture works
Starting from Gluesync 2.3, the Azure Cosmos DB agent reads the Cosmos DB change feed in all versions and deletes mode (AVAD). In this mode Cosmos DB itself reports every insert, update and delete that happens on the monitored container, so Gluesync does not need to infer deletes from the content of the documents.
-
Inserts, updates and deletes are captured natively, including the deletes performed by the Cosmos DB time-to-live (TTL) expiry.
-
Every intermediate change is delivered, in order. If a document is updated three times between two reads, the target receives three changes, not only the latest version. Changes to the same document are always delivered in the order they happened.
-
No change is lost. The agent saves its position in the change feed only after the target has confirmed the changes, see Delivery guarantees.
-
No application changes are needed. Your applications keep inserting, updating and deleting documents as usual.
-
No soft-delete field and no Azure Function. Documents are not required to carry a "deleted" flag, and nothing has to physically remove the flagged documents from the container.
| The all versions and deletes mode is a feature of the Azure Cosmos DB for NoSQL API and it must be enabled on your account before the agent can start the change data capture. See Enabling the all versions and deletes change feed mode below. |
If you are upgrading from Gluesync 2.2 and you used the soft-delete setting, read Upgrading from Gluesync 2.2 before upgrading.
Source data from Azure Cosmos DB
Prerequisites
To have Gluesync working on your Azure Cosmos DB instance as a source connector you will need to have:
-
an Azure Cosmos DB account that uses the API for NoSQL (other Cosmos DB APIs are not supported as a source);
-
valid URL and a Azure Cosmos DB service key;
-
continuous backup enabled on the account, with either the 7-day or the 30-day tier;
-
the All versions and deletes change feed mode feature enabled on the account, as described in Enabling the all versions and deletes change feed mode;
-
a lease container in the source database, see Setting up Gluesync’s CosmosDB agent;
-
Gluesync Azure Cosmos DB agent will connect to the URL provided to the port 443 through the HTTPS protocol.
| Without continuous backup and the all versions and deletes feature enabled, the agent cannot read the change feed and CDC does not start. Review the limitations before enabling the feature on a production account. |
Setup via Web UI
-
Hostname / IP Address: Hostname of your Azure Cosmos DB service;
-
Port: Optional, defaults to
443; -
Database name: Name of your source database.
Setup via Rest APIs
Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.
Connect the agent
curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
"hostCredentials": {
"connectionName": "myAgentNickName",
"host": "host-address",
"password": "your azure secret key",
"port": 443
},
"customHostCredentials": {
"lease": "lease_container_name"
}
}'
Enabling the all versions and deletes change feed mode
Follow these steps on the Azure Cosmos DB account that holds the source container. The steps are performed by an Azure user with permission to modify the account.
-
In the Azure portal open your Azure Cosmos DB account (API for NoSQL).
-
Make sure the account uses continuous backup. Under Settings, open Backup & Restore (or Backup Policy, depending on the portal version) and, if the account still uses periodic backup, migrate it to Continuous (7 days) or Continuous (30 days). See the Microsoft documentation on continuous backup for pricing and restrictions.
-
Under Settings, open the Features page.
-
Select All versions and deletes change feed mode.
-
Read the information in the pane that opens and select Enable.
-
Wait for the feature to be enabled. The operation can take up to 30 minutes and, while it runs, other changes to the account are blocked. Plan it outside of maintenance windows of the account.
-
Create (or verify) the lease container, then configure the Gluesync agent as described in the sections above.
The feature can also be enabled via REST API, by setting the account property enableAllVersionsAndDeletesChangeFeed to true using the Azure Resource Manager API version 2024-12-01-preview or later:
az rest --method patch \
--url "https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.DocumentDB/databaseAccounts/<account_name>?api-version=2024-12-01-preview" \
--body '{ "properties": { "enableAllVersionsAndDeletesChangeFeed": true } }'
| The feature can only be enabled on an existing account: it cannot be enabled while the account is created. The Gluesync agent does not change the account configuration on your behalf. |
To learn more about the change feed modes please refer to the Microsoft documentation.
Setting up Gluesync’s CosmosDB agent
First of all, we need to set up the lease container in the Azure Cosmos DB database. The Lease container is used by Cosmos DB to store the state of the Change Feed Processor. It can be created both manually and dinamically. The following example shows how it can be created manually through the Azure CLI, (at the moment Gluesync does not create it dinamically).
az cosmosdb sql container create -g development --account-name <account_name> --database-name <db_name> --name <lease_name> --partition-key-path /id
The lease container also coordinates the processing of the change feed across multiple workers and it can be stored in the same account as the monitored container or in a separate account.
The monitored container instead holds the data from which the change feed is generated. Every insert, update and delete performed on the monitored container, including the ones due to the TTL expiry, is reflected in the change feed read by Gluesync in all versions and deletes mode.
Lease container partition key
The lease container must use /id as its partition key, as in the example above. This is required by the Azure Cosmos DB change feed processor in all versions and deletes mode.
Starting from Gluesync 2.3 the Lease container partition key field setting is removed from the Advanced settings tab of the agent: the lease container is always read with the /id partition key. If the setting is present in a configuration restored from a backup or exported from Gluesync 2.2, it is ignored and, when it holds a value other than id, the agent logs a warning.
|
What the lease container stores
The lease container holds the position of the agent in the change feed of each monitored container. The leases of a monitored container are stored with the gs-avad- prefix followed by the container name, so one lease container can serve all the containers monitored by the same agent.
-
The position is kept across restarts: when a pipeline is stopped and started again, the agent resumes from where it stopped, as long as the position is still within the continuous backup retention window.
-
When an entity starts a new full snapshot, the leases of its container are reset, so that the change feed starts again together with the snapshot. The leases are kept when other entities of the agent read the same container.
-
Do not share the lease container with other change feed applications, and in particular with applications that read the change feed in latest version mode: the agent refuses to start when the lease container holds leases of another change feed mode for the same container.
Delete events
Deletes are read directly from the change feed: when a document is deleted from the monitored container, either by your application or by the Cosmos DB time-to-live (TTL) expiry, the agent receives a delete change and replicates it to the target as a delete.
There is nothing to configure for this to happen:
-
documents do not need any "deleted" flag;
-
no Azure Function (or any other process) is needed to remove documents from the monitored container;
-
you can freely use the TTL at the container or at the item level.
Delivery guarantees
The agent delivers every change at least once:
-
the change feed position is saved in the lease container only after the target has confirmed that it stored the changes;
-
if the target fails, the entity is stopped or the agent stops unexpectedly before the confirmation, the unconfirmed changes are read again from the change feed when the pipeline resumes;
-
during the snapshot, the change feed is already being read, so a change can be applied to the target both by the snapshot and by the change data capture.
Changes that are delivered twice are absorbed by the target: inserts and updates are written as upserts, and a delete of a document that no longer exists on the target is ignored. For this reason keep the insert conflict strategy of the pipeline set to UPSERT, which is the default.
| If the target stops accepting changes, the agent does not move its change feed position forward and the changes wait in Azure Cosmos DB. If the target is not available for longer than the continuous backup retention window, the entities must be re-snapshotted. |
Troubleshooting
The agent checks the change feed before it starts the change data capture of a container and stops with one of the following errors when a requirement is not met:
| Error | What to do |
|---|---|
All Versions And Deletes Change Feed Not Available |
Enable continuous backup and the all versions and deletes change feed mode on the account, see Enabling the all versions and deletes change feed mode. |
Change Feed Resume Point Outside The Retention Window |
The pipeline was stopped for longer than the continuous backup retention window. Run a new snapshot of the affected entities. |
Partition Merge Not Supported By The Change Feed |
The account has partition merge enabled, or it merged partitions in the past. The all versions and deletes mode cannot be used on this account. |
Change Feed Lease Mode Conflict |
The lease container holds leases written by an application that reads the change feed in another mode. Use a dedicated lease container for the agent. |
Change Feed Leases Not Initialized |
The agent could not initialize its leases in time. Check that the lease container exists, uses the |
Limitations
-
Only the Azure Cosmos DB for NoSQL API is supported.
-
Accounts that have partition merge enabled, or that have ever merged partitions, are not supported by the all versions and deletes mode.
-
Changes can only be read within the continuous backup retention window (7 or 30 days, depending on the tier you chose). If a pipeline stays stopped for longer than that, the missing changes can no longer be read from the change feed and the pipeline must be re-snapshotted.
-
The change feed cannot be started from a point in the past. To make sure that no change is lost, Gluesync starts reading the change feed before it starts the snapshot, so changes that happen while the snapshot runs are replicated once the snapshot is completed.
-
Previous images (the content of a document before it was updated or deleted) are not available in the change feed.
-
Changes to the same document are delivered in order. Changes to documents stored in different physical partitions of the container can be delivered in a different order than the one in which they happened.
-
Requests in all versions and deletes mode are always sent through the Azure Cosmos DB Gateway connection mode. You may notice a slightly higher latency compared to the direct connection mode and you should take this into account when sizing the provisioned throughput (RU/s) of the account.
-
The feature must be enabled on an existing account and, until the enablement is completed (up to 30 minutes), other changes to the account are blocked.
Upgrading from Gluesync 2.2
Breaking change. Gluesync 2.3 removes the support for soft-delete on the Azure Cosmos DB source agent. Up to Gluesync 2.2 the agent detected deletes through a "soft-delete" flag on the documents (the deleted field, configurable through the Soft delete document delete field setting) and an Azure Function removed the flagged documents. This mechanism is replaced by the all versions and deletes change feed mode.
|
If you used soft-delete on Gluesync 2.2, follow these steps:
-
Before upgrading, enable continuous backup and the all versions and deletes change feed mode on the Azure Cosmos DB account, as described in Enabling the all versions and deletes change feed mode. Check the limitations, especially the one on partition merge.
-
Upgrade Gluesync to 2.3.
-
The Soft delete document delete field setting is removed. If it is present in a configuration restored from a backup or exported from Gluesync 2.2, it is ignored.
-
Remove or decommission the Azure Function that was used to set or process the deleted flag, and any other process that physically deletes the flagged documents on behalf of Gluesync. Deletes are now captured when the documents are really deleted.
-
Start the pipelines. On the first start of an existing pipeline the agent removes the change feed leases written by Gluesync 2.2 from the lease container and starts reading the change feed from the current moment, because the all versions and deletes mode cannot resume from a 2.2 position or from a past timestamp. The agent logs a warning with the time span that is not covered.
Changes made to the monitored containers between the last change processed by Gluesync 2.2 and the first start on 2.3 are not captured. Either stop writing to the monitored containers for the duration of the upgrade, or run a new snapshot of the affected entities after the upgrade. -
Verify on the target that deletes are replicated as expected.
Documents that still carry the old deleted: true flag are now replicated as normal updates: the flag is treated as any other field of the document and it does not produce a delete on the target. If you want to get rid of those documents, delete them from the monitored container.
|
To learn more about the Azure Cosmos DB change feed processor please refer to the following link from the Microsoft’s learning portal.
For further details, please refer to the official Azure Cosmos DB documentation at this link: learn.microsoft.com/en-us/azure/cosmos-db/introduction/