Set up a Redis Streams target for CDC events

Create the user. Replace the password. The reset keyword starts from a user with no commands, no keys and no password:

redis-cli ACL SETUSER gluesync reset on '>change-me-strong' '~gs:*' +xadd +ping +info

Check what the user can and cannot do:

redis-cli --user gluesync --pass change-me-strong --no-auth-warning PING
redis-cli --user gluesync --pass change-me-strong --no-auth-warning XADD gs:probe '*' a 1
redis-cli --user gluesync --pass change-me-strong --no-auth-warning XADD other:probe '*' a 1

Remove the probe stream with an administrator account:

redis-cli DEL gs:probe

In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

In 1. Setup source, choose the PostgreSQL CDC agent tile and fill Base settings: Host name / Address, Port, Database name, Username and Password. Select Test and continue. The source appears with a green check mark. Select Next;

In 2. Setup target(s), choose the Redis Streams tile under Event streaming. In Base settings fill:

Open Advanced settings, select Add property, choose the key Stream key prefix and enter gs as the value;

Select Test and continue. The target appears with a green check mark. Select Next, enter a Pipeline name and select Finish.

In the pipeline, select Entities. The Add entities page opens with the Object browser. Expand the source schema public and select the table customers. The Field mapping dialog opens;

Look at the Target side. Target Stream is the name of the stream and it starts as the table name, customers. Target Key namespace (optional) is a scope that goes into the stream key. Leave both as they are;

Select Confirm, then Confirm and continue.

The entity Stream key override, if you set one, exactly as written;

The Stream key template, if you set one, with {prefix}, {scope}, {collection} and {entity} replaced;

Otherwise <prefix>:<scope>:<collection>, skipping every part that is empty.

In the pipeline, select the play button of the entity. The Sync changes (CDC) dialog opens. Start from offers Where it stopped, Now and Full copy. Keep Full copy: it copies all rows first and then follows the changes. The Speed section shows Standard, and its Writing part says that this target has no bulk load, so the agent writes with the pipelined XADD batches of step 8;

Select Start entity. The entity becomes Active and Last rows shows 20;

Read the stream:

redis-cli TYPE gs:customers
redis-cli XLEN gs:customers
redis-cli XRANGE gs:customers - + COUNT 1

Insert, update and delete one row, with a few seconds between the statements:

INSERT INTO customers (id, name, surname, address, gender, phone, email)
VALUES (1001, 'CDC', 'Insert', '1 Stream Lane', 'Female', NULL, 'cdc@example.com');
UPDATE customers SET name = 'UpdatedName' WHERE id = 1001;
DELETE FROM customers WHERE id = 1001;

Read the three newest entries, newest first:

redis-cli XREVRANGE gs:customers + - COUNT 3

Remove every row, then read the last entry:

TRUNCATE customers;

redis-cli XREVRANGE gs:customers + - COUNT 1

In the pipeline list, select the Redis icon of the pipeline. The Edit target(s) page opens. Select Advanced settings;

Select Add property and add Include before image. Switch it on. An update entry then carries the old column values, and a delete entry carries the row the source sent;

Select Add property and add Trim strategy. Choose MAXLEN. Add Trim MAXLEN (entries kept per stream) with the value 30. Add Approximate trimming (~) and switch it off, so Redis keeps exactly 30 entries and does not wait to trim in whole blocks;

Select Test and continue;

Insert 40 rows and update one of them on the source, then read the length and the last entry:

INSERT INTO customers
SELECT n, 'N' || n, 'S' || n, 'A' || n, 'Male', '+39-' || n, 'e' || n || '@example.com'
FROM generate_series(1, 40) AS n;
UPDATE customers SET phone = '+39-555-9999' WHERE id = 2;

redis-cli XLEN gs:customers
redis-cli XREVRANGE gs:customers + - COUNT 1

Enter a wrong password and select Test and continue. The page shows Failed to save with Unable to connect to redis/<unresolved>:6379 (with your host name and port). The text does not say why. The Core Hub log has the cause: RedisCommandExecutionException: WRONGPASS invalid username-password pair or user is disabled. Fix: enter the password of the Redis user;

Enter a host that does not exist. The page shows Unable to connect to <name>/<unresolved>:6379, and the log shows UnknownHostException: Failed to resolve '<name>'. Fix: enter a name that Core Hub resolves. Core Hub has to resolve it, not the machine of your browser;

Enter a port with no Redis behind it. The same message appears, and the log shows Connection refused. Fix: enter the port of your server;

Set the stream key prefix to gs while the user is limited to a different pattern, for example ~stream:*, and start the entity. The connection test passes, then the entity goes to Error and the notification Redis command execution error shows NOPERM No permissions to access a key. Fix: let the user write the key, for example redis-cli ACL SETUSER gluesync resetkeys '~gs:*';

Set Trim strategy to MAXLEN and Trim MAXLEN (entries kept per stream) to 0, and change a row on the source. The entity goes to Error with Trim MAXLEN must be greater than zero when the trim strategy is MAXLEN, got 0. Fix: enter a value greater than zero.