Static DynamoDB CDC with Gluesync: Streams Setup Guide

Source data from Amazon DynamoDB

Prerequisites

To have Gluesync working on your DynamoDB AWS account you will need to have:

  • valid user credentials with permissions: arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess;

  • IAM access key obtained through AWS IAM console;

  • IAM secret key obtained through AWS IAM console;

  • Name of the region where the DynamoDB belongs to;

  • Each table you’d like to listen to changes from must be configured with DynamoDB streams ON, please refer to the chapter here below for a complete step-by-step guide.

The retention period of transaction logs in DynamoDB it is by default set to the maximum available value of 24 hours.

Setup via Web UI

  • Region: The name of the region where your DynamoDB belongs to;

  • Access Key: IAM Credential access key ID created under your AWS IAM console;

  • Secret Key IAM Credential secret created under your AWS IAM console.

Custom host credentials

This agent has no specific configuration properties.

Setup via Rest APIs

Here following an example of calling the CoreHub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "my-region-us",
        "username": "myAccessKeyId",
        "password": "mySecretAccessKey"
}'

Turning ON DynamoDB Streams on your table

This step is mandatory to be followed to enable CDC on your DynamoDB source table(s).

By default, DynamoDB table(s) has/have CDC turned off. To enable CDC on DynamoDB tables you are required to follow the steps listed hereafter:

  1. Connect to your AWS console;

  2. Enter inside the DynamoDB control plane;

  3. From the tables tab, select the table you’re looking to enable CDC onto;

  4. Once you’re inside the table details press on the Exports and streams tab;

  5. From the Exports and streams tab scroll down reaching DynamoDB stream details and then press Enable;

  6. A new window will appear telling you which of the 4 available options you would like to choose, select New and old images;

  7. Once selected, press the Enable streams button.

Perform this operation per each one of the tables that are meant to be replicated under Gluesync.

Additional charges by AWS may apply. Please refer to the documentation at this link to know more about billing details for DynamoDB Streams.
In further versions, we might support automatic CDC enablement via DynamoDB SDK.