Oracle OpenLogReplicator Setup Guide
Prerequisites
Before setting up the Oracle OpenLogReplicator agent, ensure you have:
-
Oracle Database (version 11.2g or higher)
-
A database user with permissions to:
-
Read access to the Oracle tables subject to the replication;
-
The user should also be granted with rights to read some system tables (see below);
-
-
The database must be working in ARCHIVELOG mode and having enabled MINIMAL SUPPLEMENTAL LOGGING
-
The agent requires physical access to the database redo log files. Access can be either via SSHFS or NFS. For testing purposes it is possible to use volumes mounting (typical usage of docker containers).
To create a valid user for Gluesync on your Oracle database you can run/adapt the following query: |
CREATE USER GLUESYNC IDENTIFIED BY password;
ALTER USER GLUESYNC quota unlimited on USERS;
GRANT CREATE SESSION TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.CCOL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.CDEF$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.COL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.DEFERRED_STG$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.ECOL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOB$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOBCOMPPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOBFRAG$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.OBJ$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TAB$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABCOMPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABSUBPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TS$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.USER$ TO GLUESYNC;
GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$DATABASE TO GLUESYNC;
GRANT SELECT ON SYS.V_$DATABASE_INCARNATION TO GLUESYNC;
GRANT SELECT ON SYS.V_$LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$LOGFILE TO GLUESYNC;
GRANT SELECT ON SYS.V_$PARAMETER TO GLUESYNC;
GRANT SELECT ON SYS.V_$STANDBY_LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$TRANSPORTABLE_PLATFORM TO GLUESYNC;
Database requirements
Enable ARCHIVELOG and MINIMAL SUPPLEMENTAL LOGGING
SELECT SUPPLEMENTAL_LOG_DATA_MIN, LOG_MODE FROM V$DATABASE;
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER SYSTEM ARCHIVE LOG CURRENT;
SELECT SUPPLEMENTAL_LOG_DATA_MIN, LOG_MODE FROM V$DATABASE;
Force Logging
In order for the OpenLogReplicator to capture all transactions from the database, it is recommended to have also FORCE LOGGING turned on.
SELECT FORCE_LOGGING FROM V$DATABASE;
ALTER DATABASE FORCE LOGGING;
ALTER SYSTEM ARCHIVE LOG CURRENT;
SELECT FORCE_LOGGING FROM V$DATABASE;
FRA configuration
It is suggested that Fast Recovery Area (FRA) is configured.
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE = 50G; -- put your desired value
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST = '/opt/db/fra'; -- put your desired value
Table configuration requirements
The default requirements of having enabled MINIMAL SUPPLEMENTAL LOGGING are enough for OpenLogReplicator to work. In such a case, the output contains only modified columns. However, typically this is not enough, and it is required that the output contains also values of columns which haven’t been modified: primary key, unique key or all columns. For the Gluesync implementation it is required to enable the logging for all columns, even if this approach will increase the size of the redo log files, this will avoid changes to not be captured by the agent; you will be able to select which columns you want to capture later on while configuring your replication pipeline. Supplemental logging for all columns can be enabled for all tables in the database with the following script:
ALTER SESSION SET CONTAINER = your_pdb_name; -- use this in case you're working on a PDB
-- Replace with your actual tables
ALTER TABLE SCHEMA1.MYFIRSTTABLE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
GRANT SELECT, INSERT, UPDATE, DELETE ON SCHEMA1.MYFIRSTTABLE TO GLUESYNC;
ALTER TABLE SCHEMA1.MYSECONDTABLE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
GRANT SELECT, INSERT, UPDATE, DELETE ON SCHEMA1.MYSECONDTABLE TO GLUESYNC;
--- proceed similar to the previous example with the rest of the tables
Only for testing reasons you could run the following script to add supplemental logging to all tables within your database: |
--- ONLY FOR TESTING PURPOSES
SELECT SUPPLEMENTAL_LOG_DATA_ALL FROM V$DATABASE;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
SELECT SUPPLEMENTAL_LOG_DATA_ALL FROM V$DATABASE;
--- ONLY FOR TESTING PURPOSES
Complete working example
The following script is an example of a complete configuration for a working use case:
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
ALTER DATABASE ARCHIVELOG;
ALTER DATABASE OPEN;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER SYSTEM ARCHIVE LOG CURRENT;
ALTER SYSTEM SET DB_CREATE_FILE_DEST = '/opt/oracle/oradata' SCOPE = BOTH;
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE = '50G' SCOPE = BOTH;
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST = '/opt/oracle/fra' SCOPE = BOTH;
ALTER SESSION SET CONTAINER = XEPDB1;
CREATE TABLESPACE TBLS1 DATAFILE SIZE 100M AUTOEXTEND ON NEXT 100M;
ALTER TABLESPACE TBLS1 FORCE LOGGING;
CREATE USER GLUESYNC IDENTIFIED BY <password> DEFAULT TABLESPACE TBLS1 TEMPORARY TABLESPACE TEMP;
ALTER USER GLUESYNC QUOTA UNLIMITED ON TBLS1;
GRANT CONNECT TO GLUESYNC;
GRANT RESOURCE TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.CCOL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.CDEF$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.COL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.DEFERRED_STG$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.ECOL$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOB$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOBCOMPPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.LOBFRAG$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.OBJ$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TAB$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABCOMPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TABSUBPART$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.TS$ TO GLUESYNC;
GRANT SELECT, FLASHBACK ON SYS.USER$ TO GLUESYNC;
GRANT SELECT ON SYS.V_$ARCHIVED_LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$DATABASE TO GLUESYNC;
GRANT SELECT ON SYS.V_$DATABASE_INCARNATION TO GLUESYNC;
GRANT SELECT ON SYS.V_$LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$LOGFILE TO GLUESYNC;
GRANT SELECT ON SYS.V_$PARAMETER TO GLUESYNC;
GRANT SELECT ON SYS.V_$STANDBY_LOG TO GLUESYNC;
GRANT SELECT ON SYS.V_$TRANSPORTABLE_PLATFORM TO GLUESYNC;
GRANT SELECT, FLASHBACK ON XDB.XDB$TTSET TO GLUESYNC;
DECLARE
CURSOR C1 IS SELECT TOKSUF FROM XDB.XDB$TTSET;
CMD VARCHAR2(2000);
BEGIN
FOR C IN C1 LOOP
CMD := 'GRANT SELECT, FLASHBACK ON XDB.X$NM' || C.TOKSUF || ' TO GLUESYNC';
EXECUTE IMMEDIATE CMD;
CMD := 'GRANT SELECT, FLASHBACK ON XDB.X$QN' || C.TOKSUF || ' TO GLUESYNC';
EXECUTE IMMEDIATE CMD;
CMD := 'GRANT SELECT, FLASHBACK ON XDB.X$PT' || C.TOKSUF || ' TO GLUESYNC';
EXECUTE IMMEDIATE CMD;
END LOOP;
END;
/
GRANT CREATE SESSION TO GLUESYNC;
GRANT
CREATE TABLE
TO GLUESYNC;
GRANT
SELECT
ANY TABLE TO GLUESYNC;
GRANT CREATE ANY SEQUENCE TO GLUESYNC;
-- Connect as SYSDBA or SYSTEM and run:
GRANT CREATE PROCEDURE TO GLUESYNC;
GRANT EXECUTE ANY PROCEDURE TO GLUESYNC;
ALTER USER GLUESYNC QUOTA UNLIMITED ON USERS;
alter session
set
current_schema = GLUESYNC;
-- Add the following line for each table that has to be synced using gluesync
ALTER TABLE GLUESYNC.<TABLENAME> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
ALTER SESSION SET CONTAINER = CDB$ROOT;
ALTER SYSTEM ARCHIVE LOG CURRENT;
Additional information regarding the configuration of the OpenLogReplicator can be found at this page: |
Setup via Web UI
-
Hostname / IP Address: DNS record or IP Address of your server;
-
Port: Optional, defaults to
1521
; -
Database name: Name of your database;
-
Username: Username with read access to the source tables;
-
Password: Password belonging to the given username.
Custom properties
-
Timeout seconds: (optional, defaults to
60
seconds) Number of seconds to set as timeout for operations involving communication with the server; -
CDB name: (optional, defaults to
NULL
) Name of the container database of your scope, required in case of multi-tenant; -
PDB name: (optional, defaults to
NULL
) Name of the pluggable database that contains the data you need to replicate, required in case of multi-tenant; -
Connection type: (optional, defaults to
SID
) Connection type used to connect to Oracle. Available types are:SID
orSERVICE_NAME
; -
Oracle server name: (defaults to
XEPDB1
) The name of the Oracle outbound server used by Gluesync to detect table changes; -
Source change retention: (optional, defaults to
5
) Number of retention days preserved in the Gluesync GDCCDC
table inside your database; -
Use partition: (optional, defaults to
false
) Enables partitioning atCDC
table, this ensures that no locks are being made while purging old records from that table as well as preserving the Oracle database well in shape. Partitioning is available only on Oracle Enterprise editions, check compatibility with your Oracle database instance before turning on this functionality; -
Thin net connect timeout: (optional, defaults to
60
) The connect timeout controls how much time is allowed to connect the socket to the database; -
Thin read timeout: (optional, defaults to
60
) Read timeout while reading from the socket; -
Down hosts timeout: (optional, defaults to
60
) To specify the amount of time in seconds that information about the down state of server hosts is kept in the driver’s cache; -
Thin outbound connect timeout: (optional, defaults to
60
) The outbound connect timeout controls the time allowed to connect the socket, let the server accept the connection to the desired service, negotiate the NS protocol as well as complete the ASO negotiation. -
OLR port: (optional, defaults to
50000
) The port exposed by the OpenLogReplicator. -
OLR DB Name: (optional, defaults to
ORA1
) Database name used by the OpenLogReplicator. -
OLR Host: (optional, defaults to
127.0.0.1
) Host used by the OpenLogReplicator in conjunction with the OLR port (The Gluesync client will connect to this port). -
Mount Type: (optional, defaults to
NFS
) Defines the method used to exposed the Oracle redo log folders, available options areNFS
,SSHFS
andVOLUME
. -
Mount Host: (optional, used only when Mount Type is either
NFS
orSSHFS
) Defines the host that expose the service used to mount the Oracle redo log files. -
Oradata Path: (optional, used only when Mount Type is either
NFS
orSSHFS
) Defines the oracle oradata directory location to be mounted on the Gluesync agent container. -
Fra Path: (optional, used only when Mount Type is either
NFS
orSSHFS
) Defines the Oracle Flash Recovery Area directory to be mounted on the Gluesync agent container. -
Mount Options: (optional, used only when Mount Type is either
NFS
orSSHFS
) Specifies the mounting options used to mount the oradata and fra directories. -
Mount KeyPath: (optional, used only when Mount Type is
SSHFS
) Specifies the SSH identityFile location. -
Mount User: (optional, used only when Mount Type is
SSHFS
) Specifies the user used tp mount the oradata and fra directories.
Setup via Rest APIs
Here following an example of calling the CoreHub’s Rest API via curl to setup the connection for this Agent.
Connect the agent
curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
"customHostCredentials": {
"defaultConnectionType": "SERVICE_NAME",
"mountType": "VOLUME",
"mountHost": "127.0.0.1",
"olrHost": "127.0.0.1",
"olrPort": 50000,
"olrDbName": "ORA1"
},
"hostCredentials": {
"connectionName": "",
"host": "host-address",
"port": 1521,
"databaseName": "db-name",
"username": "",
"password": "",
"disableAuth": false,
"enableTls": false,
"trustServerCertificate": false,
"additionalHosts": [],
"maxConnectionsCount": 100
}
}'
Working with Before & After images
Before & after images are a feature that allows Gluesync to track the changes that have occurred in your database and comparing them with their previous values. This enables Gluesync to apply only the changes that have occurred, saving bandwidth and processing time.
This feature is automatically enabled and available for use in this agent.