Before & After Images

Before & after images provide a comprehensive view of data changes by capturing the state of records both before and after modifications, enabling precise synchronization and data validation.

Overview

Before & after images support is a powerful feature that enhances data synchronization accuracy and efficiency. This feature is available for both RDBMS and NoSQL agents, with optimized implementations for each type of database system.

Implementation by database type

Database type Features

RDBMS Agents

* Field-level comparison at source agent level * Only changed columns are transmitted to target * Updates on non-monitored columns are automatically skipped * Rows with identical before & after values are excluded from replication

NoSQL Agents

* Row-level comparison for efficient synchronization * Automatic skipping of unchanged records * Bandwidth and processing optimization * Ongoing feature improvements and extensions

Key benefits

  • Bandwidth optimization: Only transmit data that has actually changed

  • Processing efficiency: Skip unnecessary updates when values remain unchanged

  • Precise change tracking: Field-level granularity for RDBMS systems

  • Resource conservation: Reduce processing overhead by filtering out unchanged data

Performance impact

Benefit Description

Reduced network traffic

By transmitting only changed columns, network bandwidth usage is significantly reduced

Lower processing overhead

Skipping unchanged rows eliminates unnecessary processing at both source and target

Optimized storage usage

Efficient handling of data changes reduces storage requirements for change tracking

Improved synchronization speed

Focusing only on actual changes leads to faster overall synchronization

Setup

While the Control plane UI determines the best option for your specific database automatically, you are required to check for its specific availability in your source database.

To further help you setting the before & after images for your database, we have implemented a set of instructions for each database type.

Check for the following section named Working with Before & After images in the source agent documentation for a guide on how to enable it.

Using Rest APIs

To setup the before & after images for a specific agent, you can use the following Rest API snippet:

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/entities/' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '[
  {
    "entity": {
      "entityId": "cb64f725",
      "entityName": "dbo.DRIVERS",
      "agentEntities": [
        {
          "type": "SingleTable",
          "entityId": "cb64f725",
          "entityName": "dbo.DRIVERS",
          "agentEntityId": "0b7d244d",
          "entityType": {
            "type": "Source",
            "maxFetchItemsCountPerIteration": 1000,
            "maxTransactionMessageKbSize": 1024,
            "pollingIntervalMilliseconds": 100,
            "unchangedDataFilterType": "ENTIRE_ROW"  // Can be set to ENTIRE_ROW, FIELDS or DISABLED
          },
          "agentId": "92f6e98d",
          "tablesProperties": {
            "dbo.DRIVERS": {}
          },
          "table": {
            "name": "DRIVERS",
            "schema": "dbo"
          },
          "columns": [
            {"name": "ID", "alias": "ID", "type": "int"},
            {"name": "FIRST_NAME", "alias": "FIRST_NAME", "type": "varchar"}
          ],
          "keys": [
            {"name": "ID", "alias": "ID", "type": "int"}
          ]
        },
        {
          "type": "NoSqlEntity",
          "entityId": "cb64f725",
          "entityName": "dbo.DRIVERS",
          "agentEntityId": "9dcc0fad",
          "entityType": {
            "type": "Target",
            "skipDeletion": false
          },
          "agentId": "a021dd8e",
          "customProperties": {},
          "tablesProperties": {
            "dbo.DRIVERS": {}
          },
          "entityObject": {
            "scope": "dbo",
            "collection": "DRIVERS"
          },
          "columns": [
            {"name": "ID", "alias": "ID", "type": "int"},
            {"name": "FIRST_NAME", "alias": "FIRST_NAME", "type": "string"}
          ],
          "keys": [
            {"name": "ID", "type": "int"}
          ]
        }
      ]
    }
  }
]'

Accepted params for the unchangedDataFilterType are:

  • ENTIRE_ROW: Filters data only if the entire row is unchanged;

  • FIELDS: Filters data based on changed columns/fields values;

  • DISABLED: Disables the functionality.