Set up an Oracle XStream source with CDC
The Oracle XStream agent reads changes through the XStream Out API, the interface that Oracle GoldenGate uses. Oracle itself captures the changes from the redo log and hands them to Gluesync through an outbound server. This guide shows how to prepare the database, create the XStream users and the outbound server, check that Oracle captures your tables, and connect Gluesync. It also helps you choose between XStream and LogMiner.
In this guide
You will decide whether XStream or LogMiner fits your database, turn on the GoldenGate replication parameter and ARCHIVELOG mode, add supplemental logging to the tables, create the XStream admin user and the connect user, create the outbound server, and read the Oracle views that tell you whether capture works. You will then create the source connection in the Control Plane. The last part lists the Oracle errors that you are most likely to meet.
Quick information
Goal |
Have Oracle capture the changes of your tables into an XStream outbound server that Gluesync reads |
Audience |
Oracle DBAs and Gluesync users who run Oracle Enterprise Edition |
Level |
Advanced |
Time required |
About 45 minutes, plus the restart of the database |
Area |
Oracle database → Control Plane → Pipelines → Oracle Database → Oracle Xstream agent |
Applies to |
Gluesync 2.3 |
Before you begin
Make sure that:
-
You can connect to Oracle as
SYSDBAand you can restart the instance. Turning on ARCHIVELOG mode needs a restart; -
Your database is Oracle Database Enterprise Edition and your licence covers the XStream APIs. The Oracle XStream reference lists Oracle GoldenGate (XStream APIs) licensing as a requirement. Check the terms with Oracle before you start;
-
You have the tables to replicate. The examples use the tables
CUSTOMERSandORDERSof the schemaSALES, in a pluggable database (PDB) calledORCLof the container database (CDB)ORCLCDB. Every table needs a primary key; -
Core Hub can reach the listener of the database;
-
You have a target connection ready, or you will create it in the pipeline wizard.
|
The database steps of this guide were run on Oracle Database 19c Enterprise Edition (CDB with a PDB). The Gluesync agent could not be started in the test setup: the published Core Hub image of Gluesync 2.3 that was used did not contain the XStream agent package, and the Control Plane answered |
Choose XStream or LogMiner
Both agents capture from the redo log and both support a CDB with PDBs. The difference is the way Oracle delivers the changes.
| XStream | LogMiner | |
|---|---|---|
How changes are read |
Oracle captures the changes in the database and a server-side outbound server hands them to Gluesync |
Gluesync starts LogMiner sessions and reads the mined redo records |
What you create in Oracle |
An admin user, a connect user and an outbound server, with |
A LogMiner user with grants, supplemental logging and ARCHIVELOG mode |
Editions and licence |
Enterprise Edition with the Oracle GoldenGate (XStream APIs) licence |
The LogMiner agent lists Oracle XE, SE and EE |
Retention of redo |
Oracle keeps the archived logs until the outbound server has processed them |
You keep the archived logs yourself until Gluesync has read them |
Choose XStream when you already license GoldenGate and you want Oracle to keep track of what Gluesync has processed. Choose LogMiner when you do not hold that licence, when you run another edition, or when you do not want an outbound server in the database. For LogMiner, follow Set up Oracle LogMiner CDC end to end.
Guide steps
SQL
Connect to the CDB root as SYSDBA. Set the size and the place of the fast recovery area to values that fit your system, turn on the parameter, and restart the database in MOUNT mode to enable ARCHIVELOG mode:
alter system set db_recovery_file_dest_size = 20G;
alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile;
alter system set enable_goldengate_replication = true;
shutdown immediate
startup mount
alter database archivelog;
alter database open;
alter pluggable database all open;
The directory of db_recovery_file_dest must exist before the restart. On a single-tenant (non-container) database you run the same statements and you skip the last one.
Check the result:
archive log list
select log_mode from v$database;
show parameter enable_goldengate
show pdbs
|
Archived logs fill the recovery area. Size it for the redo that your system writes between two reads of Gluesync, and watch it with |
SQL
The two XStream users need a tablespace each. Create them in the CDB root. On a multitenant database, create them in the PDB as well, because the users are common users with CONTAINER=ALL. Adapt the file paths to your system:
CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_adm_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
ALTER SESSION SET CONTAINER = ORCL;
CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCL/xstream_adm_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCL/xstream_tbs.dbf'
SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
ALTER SESSION SET CONTAINER = CDB$ROOT;
SQL
The admin user creates and manages the outbound server. Create it as a common user in the CDB root, then grant the XStream administrator privileges. Replace the password:
CREATE USER c##xstrmadmin IDENTIFIED BY "<admin password>"
DEFAULT TABLESPACE xstream_adm_tbs
QUOTA UNLIMITED ON xstream_adm_tbs
CONTAINER=ALL;
GRANT CREATE SESSION, SET CONTAINER TO c##xstrmadmin CONTAINER=ALL;
BEGIN
DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
grantee => 'c##xstrmadmin',
privilege_type => 'CAPTURE',
grant_select_privileges => TRUE,
container => 'ALL');
END;
/
On a single-tenant database, name the user xstreamadmin, drop the c## prefix and the CONTAINER clauses, and call GRANT_ADMIN_PRIVILEGE without the container argument, as in Oracle XStream Single-Tenant Setup.
SQL
The connect user is the one that Gluesync signs in with, and the one the outbound server runs as. It reads the dictionary and the tables. Create it in the CDB root:
CREATE USER c##xstrm IDENTIFIED BY "<connect password>"
DEFAULT TABLESPACE xstream_tbs
QUOTA UNLIMITED ON xstream_tbs
CONTAINER=ALL;
GRANT CREATE SESSION, SET CONTAINER, CREATE TABLE TO c##xstrm CONTAINER=ALL;
GRANT SELECT ON V_$DATABASE TO c##xstrm CONTAINER=ALL;
GRANT FLASHBACK ANY TABLE, SELECT_CATALOG_ROLE, EXECUTE_CATALOG_ROLE TO c##xstrm CONTAINER=ALL;
Then switch to the PDB that holds the tables. Add supplemental logging of all columns to each table, and let the connect user read it. The outbound server needs the logged columns to send complete rows, with before and after images:
ALTER SESSION SET CONTAINER = ORCL;
ALTER TABLE sales.customers ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
GRANT SELECT ON sales.customers TO c##xstrm;
ALTER TABLE sales.orders ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
GRANT SELECT ON sales.orders TO c##xstrm;
ALTER SESSION SET CONTAINER = CDB$ROOT;
Repeat the two statements for every table you add later. On a single-tenant database, the user is xstream and you do not switch containers. The reference page also grants INSERT, UPDATE and DELETE on the tables to the connect user, which only a target needs.
SQL
Sign in as the admin user, to the CDB root service, and create the outbound server with the tables you want to capture. Gluesync reads the server named GSXOUT unless you set another name in the connection:
sqlplus c##xstrmadmin/"<admin password>"@//<host>:1521/ORCLCDB
DECLARE
tables DBMS_UTILITY.UNCL_ARRAY;
schemas DBMS_UTILITY.UNCL_ARRAY;
BEGIN
tables(1) := 'SALES.CUSTOMERS';
tables(2) := 'SALES.ORDERS';
schemas(1) := NULL; -- NULL: do not capture whole schemas
DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
server_name => 'GSXOUT',
table_names => tables,
schema_names => schemas,
source_database => 'ORCL');
END;
/
BEGIN
DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
server_name => 'GSXOUT',
connect_user => 'c##xstrm');
END;
/
source_database names the PDB that holds the tables. On a single-tenant database, leave it out and sign in as xstreamadmin with connect_user ⇒ 'xstream'. You can also pass capture_name; without it Oracle names the capture process CAP$_GSXOUT_1.
An outbound server serves one Gluesync instance. Create another one, with another name, for every other Core Hub that reads the same database.
SQL
Run these checks as the admin user. They are the same checks that explain most failures later:
SELECT SERVER_NAME, CAPTURE_NAME, SOURCE_DATABASE, CONNECT_USER FROM ALL_XSTREAM_OUTBOUND;
SELECT CAPTURE_NAME, STATE FROM V$XSTREAM_CAPTURE;
SELECT APPLY_NAME, STATUS FROM DBA_APPLY WHERE PURPOSE = 'XStream Out';
SELECT TABLE_OWNER, TABLE_NAME, CON_ID FROM CDB_CAPTURE_PREPARED_TABLES;
Then change a row in the PDB and watch the counters grow:
SELECT TOTAL_MESSAGES_CAPTURED, TOTAL_MESSAGES_ENQUEUED FROM V$XSTREAM_CAPTURE;
-- as the table owner, in the PDB:
UPDATE customers SET balance = 99 WHERE id = 1;
COMMIT;
SELECT TOTAL_MESSAGES_CAPTURED, TOTAL_MESSAGES_ENQUEUED FROM V$XSTREAM_CAPTURE;
The agent reads DBA_CAPTURE_PREPARED_TABLES to confirm that a table is registered. A table that is missing there is refused when you add it as an entity, see the troubleshooting table.
|
UI
-
In the Control Plane, select Create your first pipeline, or the + button of the pipeline list. The Agents setup wizard opens;
-
In 1. Setup source, choose the Oracle Database tile. It groups four Oracle connectors. Choose Oracle Xstream agent and select Next;
-
Fill the connection:
-
Host name / Address, Port (
1521by default); -
Database name: the service name of the database that the connection uses;
-
Username and Password: the connect user of step 4;
-
Max connections count: the size of the connection pool;
-
-
Open the advanced settings and set Connection type to
SERVICE_NAMEif you enter a service name. The default isSID. Keep Xstream server name atGSXOUT, or enter the name you used in step 5. Wallet, instance name and TLS settings are explained in Oracle XStream Gluesync Configuration; -
Select Test and continue, then choose your target in 2. Setup target(s), name the pipeline and select Finish.
|
The agent attaches to the outbound server through the Oracle client (OCI), with the user and the service of the connection, and it reads the tables and the dictionary with the same user. In the test database, the CDB root service |
UI
-
In the pipeline, select Entities. In the Object browser tick the tables of your schema, confirm the Field mapping and the target table, and select Confirm and continue;
-
On the row of the entity select the play button. Keep Full copy in Start from: the entity copies all the rows and then follows the changes. See Start a snapshot and CDC;
-
When the entity is Active, change rows in the source tables and check the target.
Some points that depend on the database and the agent:
-
The agent keeps a low watermark and sends it to the outbound server, so that Oracle can release the archived logs that Gluesync has processed. Xstream Low Watermark Retention in Hours is
24by default. It matters for tables that have no changes: their old position is ignored after that time, so an idle table does not hold the logs back; -
LOB columns (
BLOB,CLOB,BFILE,LONG) andROWIDcolumns are read with a query on the key of the row, so keep a primary key on the tables; -
To limit the load, capture only the tables you need and remove unneeded columns from the capture, as in the troubleshooting table. The speed of the copy and of the writes depends on the target, see Tune replication throughput and Bulk load.
SQL
Add the supplemental logging and the grant first (step 4), then add the table to the existing server as the admin user. Leave the schema list empty, or the whole schema is added:
DECLARE
tables DBMS_UTILITY.UNCL_ARRAY;
schemas DBMS_UTILITY.UNCL_ARRAY;
BEGIN
tables(1) := 'SALES.PAYMENTS';
schemas(1) := '';
DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
server_name => 'GSXOUT',
table_names => tables,
schema_names => schemas,
add => TRUE);
END;
/
Setting add to FALSE removes the table from the capture. See Oracle XStream Troubleshooting for the other administration commands.
Troubleshooting
| Symptom or message | Cause | Fix |
|---|---|---|
|
The user that runs the procedure is not an XStream administrator. In the test, the connect user failed, while the admin user and |
Sign in as the admin user of step 3, or grant it again with |
|
An outbound server with that name exists already. |
Use another |
|
The table is not in |
Add the table with |
|
The first outbound server that Oracle lists is not the one named in Xstream server name. |
Enter the real name in the advanced settings of the connection, or drop the servers that are not used. |
|
The outbound server exists, but it is disabled or aborted. |
Read the error with |
|
The agent cannot attach: the server does not exist, is stopped, belongs to another connect user, or is attached by another client. |
Check |
The capture state is |
The capture produces more changes than the outbound server consumes, usually because a whole schema or many unneeded columns are captured. |
Capture only the tables you need, and drop unneeded columns with |
After a database restart nothing reaches the target |
The capture was not started again. |
Start it with |
|
The connection names a CDB or PDB on a database that is not multitenant. |
Remove the container names from the configuration, see Oracle XStream Troubleshooting. |