Search docs.molo17.com
    Configuration guide Connectors

    Set up an Oracle XStream source with CDC

    The Oracle XStream agent reads changes through the XStream Out API, the interface that Oracle GoldenGate uses. Oracle itself captures the changes from the redo log and hands them to Gluesync through an outbound server. This guide shows how to prepare the database, create the XStream users and the outbound server, check that Oracle captures your tables, and connect Gluesync. It also helps you choose between XStream and LogMiner.

    In this guide

    You will decide whether XStream or LogMiner fits your database, turn on the GoldenGate replication parameter and ARCHIVELOG mode, add supplemental logging to the tables, create the XStream admin user and the connect user, create the outbound server, and read the Oracle views that tell you whether capture works. You will then create the source connection in the Control Plane. The last part lists the Oracle errors that you are most likely to meet.

    Quick information

    Goal

    Have Oracle capture the changes of your tables into an XStream outbound server that Gluesync reads

    Audience

    Oracle DBAs and Gluesync users who run Oracle Enterprise Edition

    Level

    Advanced

    Time required

    About 45 minutes, plus the restart of the database

    Area

    Oracle database → Control Plane → Pipelines → Oracle Database → Oracle Xstream agent

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You can connect to Oracle as SYSDBA and you can restart the instance. Turning on ARCHIVELOG mode needs a restart;

    2. Your database is Oracle Database Enterprise Edition and your licence covers the XStream APIs. The Oracle XStream reference lists Oracle GoldenGate (XStream APIs) licensing as a requirement. Check the terms with Oracle before you start;

    3. You have the tables to replicate. The examples use the tables CUSTOMERS and ORDERS of the schema SALES, in a pluggable database (PDB) called ORCL of the container database (CDB) ORCLCDB. Every table needs a primary key;

    4. Core Hub can reach the listener of the database;

    5. You have a target connection ready, or you will create it in the pipeline wizard.

    The database steps of this guide were run on Oracle Database 19c Enterprise Edition (CDB with a PDB). The Gluesync agent could not be started in the test setup: the published Core Hub image of Gluesync 2.3 that was used did not contain the XStream agent package, and the Control Plane answered Agent oracle-xstream jar not found when the agent was chosen. The Control Plane steps below follow the reference pages and the agent configuration. Run them in a Core Hub that includes the XStream agent.

    Choose XStream or LogMiner

    Both agents capture from the redo log and both support a CDB with PDBs. The difference is the way Oracle delivers the changes.

    XStream LogMiner

    How changes are read

    Oracle captures the changes in the database and a server-side outbound server hands them to Gluesync

    Gluesync starts LogMiner sessions and reads the mined redo records

    What you create in Oracle

    An admin user, a connect user and an outbound server, with enable_goldengate_replication set to TRUE

    A LogMiner user with grants, supplemental logging and ARCHIVELOG mode

    Editions and licence

    Enterprise Edition with the Oracle GoldenGate (XStream APIs) licence

    The LogMiner agent lists Oracle XE, SE and EE

    Retention of redo

    Oracle keeps the archived logs until the outbound server has processed them

    You keep the archived logs yourself until Gluesync has read them

    Choose XStream when you already license GoldenGate and you want Oracle to keep track of what Gluesync has processed. Choose LogMiner when you do not hold that licence, when you run another edition, or when you do not want an outbound server in the database. For LogMiner, follow Set up Oracle LogMiner CDC end to end.

    Guide steps

    Turn on the GoldenGate replication parameter and ARCHIVELOG mode

    SQL

    Connect to the CDB root as SYSDBA. Set the size and the place of the fast recovery area to values that fit your system, turn on the parameter, and restart the database in MOUNT mode to enable ARCHIVELOG mode:

    alter system set db_recovery_file_dest_size = 20G;
    alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile;
    alter system set enable_goldengate_replication = true;
    shutdown immediate
    startup mount
    alter database archivelog;
    alter database open;
    alter pluggable database all open;

    The directory of db_recovery_file_dest must exist before the restart. On a single-tenant (non-container) database you run the same statements and you skip the last one.

    Check the result:

    archive log list
    select log_mode from v$database;
    show parameter enable_goldengate
    show pdbs
    Expected result

    archive log list shows Archive Mode with Automatic archival set to Enabled, log_mode is ARCHIVELOG, enable_goldengate_replication is TRUE, and your PDB has the open mode READ WRITE.

    Archived logs fill the recovery area. Size it for the redo that your system writes between two reads of Gluesync, and watch it with select * from v$recovery_area_usage;. A full recovery area stops the database from archiving.

    Create the tablespaces

    SQL

    The two XStream users need a tablespace each. Create them in the CDB root. On a multitenant database, create them in the PDB as well, because the users are common users with CONTAINER=ALL. Adapt the file paths to your system:

    CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_adm_tbs.dbf'
      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
    CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/xstream_tbs.dbf'
      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
    
    ALTER SESSION SET CONTAINER = ORCL;
    CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCL/xstream_adm_tbs.dbf'
      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
    CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/ORCLCDB/ORCL/xstream_tbs.dbf'
      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;
    ALTER SESSION SET CONTAINER = CDB$ROOT;
    Expected result

    Each statement answers Tablespace created.

    Create the XStream admin user

    SQL

    The admin user creates and manages the outbound server. Create it as a common user in the CDB root, then grant the XStream administrator privileges. Replace the password:

    CREATE USER c##xstrmadmin IDENTIFIED BY "<admin password>"
      DEFAULT TABLESPACE xstream_adm_tbs
      QUOTA UNLIMITED ON xstream_adm_tbs
      CONTAINER=ALL;
    
    GRANT CREATE SESSION, SET CONTAINER TO c##xstrmadmin CONTAINER=ALL;
    
    BEGIN
      DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE(
        grantee                 => 'c##xstrmadmin',
        privilege_type          => 'CAPTURE',
        grant_select_privileges => TRUE,
        container               => 'ALL');
    END;
    /

    On a single-tenant database, name the user xstreamadmin, drop the c## prefix and the CONTAINER clauses, and call GRANT_ADMIN_PRIVILEGE without the container argument, as in Oracle XStream Single-Tenant Setup.

    Expected result

    User created., Grant succeeded. and PL/SQL procedure successfully completed.

    Create the connect user and prepare the tables

    SQL

    The connect user is the one that Gluesync signs in with, and the one the outbound server runs as. It reads the dictionary and the tables. Create it in the CDB root:

    CREATE USER c##xstrm IDENTIFIED BY "<connect password>"
      DEFAULT TABLESPACE xstream_tbs
      QUOTA UNLIMITED ON xstream_tbs
      CONTAINER=ALL;
    
    GRANT CREATE SESSION, SET CONTAINER, CREATE TABLE TO c##xstrm CONTAINER=ALL;
    GRANT SELECT ON V_$DATABASE TO c##xstrm CONTAINER=ALL;
    GRANT FLASHBACK ANY TABLE, SELECT_CATALOG_ROLE, EXECUTE_CATALOG_ROLE TO c##xstrm CONTAINER=ALL;

    Then switch to the PDB that holds the tables. Add supplemental logging of all columns to each table, and let the connect user read it. The outbound server needs the logged columns to send complete rows, with before and after images:

    ALTER SESSION SET CONTAINER = ORCL;
    
    ALTER TABLE sales.customers ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
    GRANT SELECT ON sales.customers TO c##xstrm;
    
    ALTER TABLE sales.orders ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
    GRANT SELECT ON sales.orders TO c##xstrm;
    
    ALTER SESSION SET CONTAINER = CDB$ROOT;

    Repeat the two statements for every table you add later. On a single-tenant database, the user is xstream and you do not switch containers. The reference page also grants INSERT, UPDATE and DELETE on the tables to the connect user, which only a target needs.

    Expected result

    Table altered. and Grant succeeded. for each table. Check the log groups in the PDB with select table_name, log_group_type from dba_log_groups where owner = 'SALES';: each table shows a row ALL COLUMN LOGGING, next to the key log groups that Oracle creates by itself.

    Create the outbound server

    SQL

    Sign in as the admin user, to the CDB root service, and create the outbound server with the tables you want to capture. Gluesync reads the server named GSXOUT unless you set another name in the connection:

    sqlplus c##xstrmadmin/"<admin password>"@//<host>:1521/ORCLCDB
    DECLARE
      tables  DBMS_UTILITY.UNCL_ARRAY;
      schemas DBMS_UTILITY.UNCL_ARRAY;
    BEGIN
      tables(1)  := 'SALES.CUSTOMERS';
      tables(2)  := 'SALES.ORDERS';
      schemas(1) := NULL;  -- NULL: do not capture whole schemas
    
      DBMS_XSTREAM_ADM.CREATE_OUTBOUND(
        server_name     => 'GSXOUT',
        table_names     => tables,
        schema_names    => schemas,
        source_database => 'ORCL');
    END;
    /
    
    BEGIN
      DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
        server_name  => 'GSXOUT',
        connect_user => 'c##xstrm');
    END;
    /

    source_database names the PDB that holds the tables. On a single-tenant database, leave it out and sign in as xstreamadmin with connect_user ⇒ 'xstream'. You can also pass capture_name; without it Oracle names the capture process CAP$_GSXOUT_1.

    An outbound server serves one Gluesync instance. Create another one, with another name, for every other Core Hub that reads the same database.

    Expected result

    Both blocks end with PL/SQL procedure successfully completed. The capture needs a short time to find the dictionary: if the state in the next step is WAITING FOR DICTIONARY REDO, run alter system archive log current; as SYSDBA and read it again.

    Check that Oracle captures your tables

    SQL

    Run these checks as the admin user. They are the same checks that explain most failures later:

    SELECT SERVER_NAME, CAPTURE_NAME, SOURCE_DATABASE, CONNECT_USER FROM ALL_XSTREAM_OUTBOUND;
    SELECT CAPTURE_NAME, STATE FROM V$XSTREAM_CAPTURE;
    SELECT APPLY_NAME, STATUS FROM DBA_APPLY WHERE PURPOSE = 'XStream Out';
    SELECT TABLE_OWNER, TABLE_NAME, CON_ID FROM CDB_CAPTURE_PREPARED_TABLES;

    Then change a row in the PDB and watch the counters grow:

    SELECT TOTAL_MESSAGES_CAPTURED, TOTAL_MESSAGES_ENQUEUED FROM V$XSTREAM_CAPTURE;
    -- as the table owner, in the PDB:
    UPDATE customers SET balance = 99 WHERE id = 1;
    COMMIT;
    SELECT TOTAL_MESSAGES_CAPTURED, TOTAL_MESSAGES_ENQUEUED FROM V$XSTREAM_CAPTURE;
    Expected result

    The outbound server GSXOUT is listed with SOURCE_DATABASE ORCL and the connect user. The state of the capture is WAITING FOR TRANSACTION when it is idle. DBA_APPLY shows GSXOUT with the status ENABLED. CDB_CAPTURE_PREPARED_TABLES lists SALES.CUSTOMERS and SALES.ORDERS with CON_ID 3 (the number of your PDB). After the update, the enqueued counter grows by a few messages: the begin, the change and the commit.

    The agent reads DBA_CAPTURE_PREPARED_TABLES to confirm that a table is registered. A table that is missing there is refused when you add it as an entity, see the troubleshooting table.
    Create the Oracle XStream source connection

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list. The Agents setup wizard opens;

    2. In 1. Setup source, choose the Oracle Database tile. It groups four Oracle connectors. Choose Oracle Xstream agent and select Next;

    3. Fill the connection:

      • Host name / Address, Port (1521 by default);

      • Database name: the service name of the database that the connection uses;

      • Username and Password: the connect user of step 4;

      • Max connections count: the size of the connection pool;

    4. Open the advanced settings and set Connection type to SERVICE_NAME if you enter a service name. The default is SID. Keep Xstream server name at GSXOUT, or enter the name you used in step 5. Wallet, instance name and TLS settings are explained in Oracle XStream Gluesync Configuration;

    5. Select Test and continue, then choose your target in 2. Setup target(s), name the pipeline and select Finish.

    Expected result

    The source passes the connection test and the pipeline appears in the pipeline list.

    The agent attaches to the outbound server through the Oracle client (OCI), with the user and the service of the connection, and it reads the tables and the dictionary with the same user. In the test database, the CDB root service ORCLCDB showed GSXOUT in DBA_APPLY but no tables of the PDB in ALL_TABLES. The PDB service ORCL showed the tables and the prepared tables but no row in DBA_APPLY. The agent configuration has no separate fields for a CDB and a PDB. Run your first pipeline on a test system, and open a support request if the entities of your PDB do not appear.

    Add the entities, run the snapshot and follow the changes

    UI

    1. In the pipeline, select Entities. In the Object browser tick the tables of your schema, confirm the Field mapping and the target table, and select Confirm and continue;

    2. On the row of the entity select the play button. Keep Full copy in Start from: the entity copies all the rows and then follows the changes. See Start a snapshot and CDC;

    3. When the entity is Active, change rows in the source tables and check the target.

    Expected result

    The row count of the target equals the row count of the source after the copy, and inserts, updates and deletes reach the target within seconds.

    Some points that depend on the database and the agent:

    • The agent keeps a low watermark and sends it to the outbound server, so that Oracle can release the archived logs that Gluesync has processed. Xstream Low Watermark Retention in Hours is 24 by default. It matters for tables that have no changes: their old position is ignored after that time, so an idle table does not hold the logs back;

    • LOB columns (BLOB, CLOB, BFILE, LONG) and ROWID columns are read with a query on the key of the row, so keep a primary key on the tables;

    • To limit the load, capture only the tables you need and remove unneeded columns from the capture, as in the troubleshooting table. The speed of the copy and of the writes depends on the target, see Tune replication throughput and Bulk load.

    Add a table to the outbound server later

    SQL

    Add the supplemental logging and the grant first (step 4), then add the table to the existing server as the admin user. Leave the schema list empty, or the whole schema is added:

    DECLARE
      tables  DBMS_UTILITY.UNCL_ARRAY;
      schemas DBMS_UTILITY.UNCL_ARRAY;
    BEGIN
      tables(1)  := 'SALES.PAYMENTS';
      schemas(1) := '';
      DBMS_XSTREAM_ADM.ALTER_OUTBOUND(
        server_name  => 'GSXOUT',
        table_names  => tables,
        schema_names => schemas,
        add          => TRUE);
    END;
    /

    Setting add to FALSE removes the table from the capture. See Oracle XStream Troubleshooting for the other administration commands.

    Troubleshooting

    Symptom or message Cause Fix

    ORA-01031: insufficient privileges with ORA-06512: at "SYS.DBMS_XSTREAM_UTL_IVK" when you run CREATE_OUTBOUND

    The user that runs the procedure is not an XStream administrator. In the test, the connect user failed, while the admin user and SYSDBA succeeded.

    Sign in as the admin user of step 3, or grant it again with DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE.

    ORA-26668: Streams process GSXOUT exists

    An outbound server with that name exists already.

    Use another server_name, or remove the old server with DBMS_XSTREAM_ADM.DROP_OUTBOUND(server_name ⇒ 'GSXOUT');. Dropping it removes the capture position that it holds.

    Missing Xstream table registration. Please add table <schema>.<table> to xstream outbound server. Check Gluesync documentation for more information.

    The table is not in DBA_CAPTURE_PREPARED_TABLES for the user and the service of the connection.

    Add the table with ALTER_OUTBOUND as in step 9. In a PDB, read CDB_CAPTURE_PREPARED_TABLES as SYSDBA and compare the CON_ID.

    error: invalid xstream apply name provided: <name>

    The first outbound server that Oracle lists is not the one named in Xstream server name.

    Enter the real name in the advanced settings of the connection, or drop the servers that are not used.

    error: xstream status not enabled: <status>

    The outbound server exists, but it is disabled or aborted.

    Read the error with SELECT APPLY_NAME, STATUS, ERROR_NUMBER, ERROR_MESSAGE FROM DBA_APPLY WHERE PURPOSE = 'XStream Out'; and SELECT * FROM DBA_OUTSTANDING_ALERTS;. Start it with DBMS_XSTREAM_ADM.START_OUTBOUND(server_name ⇒ 'GSXOUT'); as the admin user.

    Couldn’t attach to XStream outbound server GSXOUT in the Core Hub log, repeated every second

    The agent cannot attach: the server does not exist, is stopped, belongs to another connect user, or is attached by another client.

    Check ALL_XSTREAM_OUTBOUND, the connect user and the server name. One outbound server serves one Gluesync instance.

    The capture state is PAUSED FOR FLOW CONTROL

    The capture produces more changes than the outbound server consumes, usually because a whole schema or many unneeded columns are captured.

    Capture only the tables you need, and drop unneeded columns with ADD_SUBSET_OUTBOUND_RULES, see Oracle XStream Troubleshooting.

    After a database restart nothing reaches the target

    The capture was not started again.

    Start it with DBMS_XSTREAM_ADM.START_OUTBOUND, or use the startup trigger of the reference page.

    ORA-02248: invalid option for ALTER SESSION

    The connection names a CDB or PDB on a database that is not multitenant.

    Remove the container names from the configuration, see Oracle XStream Troubleshooting.