IBM i (AS/400) journal data capture architecture

Gluesync provides a robust, high-performance Change Data Capture (CDC) engine for IBM i (AS/400) systems. It extracts real-time database modifications directly from system journals by orchestrating native service program calls and optimizing data throughput.

To adapt to different workloads and resource constraints, Gluesync offers two distinct runtime models for reading journal entries:

  1. Standard Journal Reader Mode (default — shared, grouped by journal)

  2. Dedicated Journal Reader Mode (isolated per entity)

Architectural core requirements and prerequisites

Regardless of the selected reading mode, the following configuration must be applied to the IBM i database layer:

  • Journal images mode: The journal must be configured with IMAGES(BOTH). Gluesync relies on both *Before-Images (for example, UB — Update Before, BR — Rollback Before) and After-Images (for example, UP — Update After, UR — Rollback After) to compute accurate delta states, detect changes in primary keys, and correctly stream transactions to target destinations.

Standard journal reader mode (default)

The Standard Mode is designed for enterprise resource optimization. Instead of polling the system per individual table, Gluesync multiplexes data retrieval by grouping entities that share the same system journal.

Infrastructure and pipeline flow

                 +---------------------------+
                 |    IBM i System Journal   |
                 +-------------+-------------+
                               |
                   [Shared Grouped API Call]
                               |
                               v
               +---------------+---------------+
               |  Gluesync Shared Journal Loop |
               +---------------+---------------+
                               |
                       [Internal Fan-Out]
                               |
                               v
               +---------------+---------------+
               |       ArenaCache (local)      |
               |  Disk-persistent, per entity  |
               +---+-----------------------+---+
                   |                       |
          [Entity A Reader]       [Entity B Reader]
                   |                       |
                   v                       v
            +--------------+        +--------------+
            | Target DB A  |        | Target DB B  |
            +--------------+        +--------------+

  1. Dynamic initialization: A shared background process spins up for a given journal as soon as at least one associated entity is put into Play status. The process automatically shuts down when all related entities are paused.

  2. The shared API query: Gluesync targets the system journal procedures, bundling up to 300 target table filters in a single request structure.

    • The 300-file threshold: If a single journal is shared by more than 300 entities, Gluesync automatically omits the specific API file filter completely to prevent parameter overflow, fetching all journal transactions and offloading the evaluation filtering logic to Gluesync’s core runtime engine.

  3. Local disk caching via ArenaCache: Extracted raw records cannot be sent directly to target instances because some entities might be paused or throttled. Gluesync decouples the pipeline by streaming these records into a local, disk-persistent cache powered by ArenaCache (from 2.2.11.2; Chronicle Queue in earlier 2.2 builds). Entries are stored pre-sorted, with one cache file per entity. See Source-side cache for the platform-wide behavior.

  4. Asynchronous target delivery: In parallel, each active entity runs an independent process that fetches its designated data blocks from the local disk cache and securely pushes transactions to its configured target system.

  5. Cache retention: To avoid infinite disk growth, the local cache automatically purges entries based on the "Source change retention in hours" parameter (default: 24 hours).

Checkpoint isolation and persistence

In Standard Mode, responsibility for state tracking is strictly decoupled into two separate checkpoint layers:

  • Journal checkpoint: Maintained strictly to track the reading position inside the IBM i journal environment. It consists of a JSON object containing the journal coordinates (journalLibrary, journalName), receiver tracking (receiverLibrary, receiverName), the 20-digit sequenceNumber, and an internal execution timestamp.

    • This checkpoint is persisted directly in a dedicated filesystem subdirectory structure: journal-checkpoints/JOURNAL_LIBRARY/JOURNAL_NAME.

    • Lifespan note: The file is never automatically pruned or altered during normal operations. To reset it, explicit operator intervention is required via the "Pipeline checkpoint reset" action in the Danger Zone section of the Gluesync User Interface.

  • Target transaction checkpoint: Each individual entity records its last successfully committed transaction identifier directly on the target instance database cache. This identifier governs the point from which the entity resumes reading from the local disk cache upon resume, having zero impact on the global shared journal-reading coordinates.

Dedicated journal reader mode

The Dedicated Mode allocates an isolated, high-throughput pipeline to a specific entity. It completely bypasses the internal disk cache mechanism, executing direct, real-time transfers from the source journal system to the destination target.

Infrastructure and pipeline flow

                 +---------------------------+
                 |    IBM i System Journal   |
                 +-------------+-------------+
                               |
                   [Direct Isolated API Call]
                               |
                               v
               +---------------+---------------+
               |   Gluesync Dedicated Reader   |
               +---------------+---------------+
                               |
                   [Direct Serialized Stream]
                               |
                               v
                        +--------------+
                        |  Target DB   |
                        +--------------+

Pipeline flow and execution mechanics

  • Direct serialized stream: Each table handles its own connection loop. Changes fetched from the journal are immediately evaluated, serialized, and written to the database target.

  • Resuming from pauses: If an entity is paused or hits an unmanaged network exception, it preserves transactional continuity by reading the last acknowledged transaction identifier written on the target database destination. When the pipeline swings back to Play status, it utilizes those exact coordinates to re-invoke the journal retrieval API.

Automated receiver pursuit logic

If a database entity experiences long periods of inactivity (for example, no target changes for several hours), the Dedicated Reader will continuously inspect the journal metadata, shifting forward to follow newly detached or attached journal receivers to ensure the pipeline stays synchronized with the current system state.

  • Current operational behavior: Because Dedicated Mode does not persist local journal checkpoints on disk, if a quiet pipeline goes into error or restarts after a prolonged pause, it must re-scan the receiver chain starting back from the exact block that contained the last target-confirmed mutation.

  • Future feature roadmap: An upcoming release will introduce isolated journal reading checkpoint persistence for the Dedicated Mode, eliminating the need to re-scan empty historical receivers during restarts.

Understanding polling intervals, fetch limits, and IBM i CPU impact

When configuring Gluesync, users encounter three different tuning parameters across the Agent and Entity configurations. Understanding their correlation is vital for optimizing IBM i resource usage and avoiding high CPU utilization.

The parameters at a glance

  1. Journal reader polling interval in milliseconds (agent-level)

  2. Polling interval (ms) (entity-level)

  3. Max number of items for interaction when performing fetch (entity-level)

Operational mechanics and decoupling (Standard Mode)

In the default Standard Mode, the performance of the entity has no direct bearing on the IBM i server load. The runtime isolates the ingestion workload into two decoupled layers:

  • The ingestion layer (agent to cache): The agent interacts with the IBM i operating system via system API procedures. It queries the system journal globally for all entities.

    • How the agent polling works: The Journal reader polling interval (for example, 10,000 ms) dictates how long the agent process sleeps only when the journal is completely drained (that is, the system returns no new records).

    • The continuation mechanism: If the IBM i response buffer indicates that more journal rows are available, Gluesync ignores the polling interval completely and immediately spawns the next API call loop to flush out the backlog. The pipeline acts as a continuous real-time stream until it catches up with the head of the journal receiver.

  • The delivery layer (cache to target): Once data lands inside Gluesync’s local ArenaCache, individual entities wake up based on their own polling interval (for example, 5,000 ms) and extract up to their configured Max number of items (for example, 100,000 rows).

    • Zero IBM i impact: These entity loops read directly from Gluesync’s local machine disk cache. Pumping 100,000 rows or changing the entity polling frequency executes completely off-box and causes zero CPU utilization on the IBM i server.

IBM i CPU optimization guidelines

If your IBM i system is suffering from high CPU usage related to journal tasks, apply the following adjustments:

  • Increase agent polling: Increase the Journal reader polling interval in milliseconds at the agent level (for example, from 10,000 ms to 20,000 ms or 30,000 ms). This reduces the frequency of repetitive diagnostic API calls to the AS/400 when database activity is low.

  • Enforce Standard Mode: Ensure that entities are not running in Dedicated Journal Reader Mode. Running multiple entities in Dedicated Mode bypasses the shared cache architecture and forces every single table to spin up an isolated API connection thread on the IBM i, multiplying the CPU context-switching overhead.

Journal API parameterization and extraction logic

Both reading modes share the underlying low-level ingestion framework when communicating with the journal retrieval program structures, employing highly precise parameters:

Native service program calls

Gluesync does not rely on the DSPJRN CL command or an RCVJRNE exit program for its standard ingestion. Instead, it invokes the IBM i journal APIs directly through the IBM Toolbox for Java (JT400). This approach is journal-position based: it reads entries by receiver and sequence number and never locks or scans the underlying physical files.

  • Primary entry retrieval: The QjoRetrieveJournalEntries procedure, exported by the QSYS/QJOURNAL service program (/QSYS.LIB/QJOURNAL.SRVPGM), is called via a ServiceProgramCall aligned on 16 bytes, using the RJNE0200 output format. It is invoked with the six parameters mandated by the IBM API specification, all passed by reference:

    # Parameter Description

    1

    Receiver variable

    Output buffer that receives the retrieved journal entries.

    2

    Length of receiver variable

    The size, in bytes, of the receiver buffer (see Receiver buffer and per-call data volume).

    3

    Qualified journal name

    20 characters: the 10-character journal name followed by its 10-character library.

    4

    Format name

    RJNE0200.

    5

    Journal entries to retrieve

    Selection block carrying the starting receiver and sequence number (or timestamp), the journal code and entry-type filters, and the list of files to capture.

    6

    Error code

    The standard IBM i error-code structure.

  • Supporting metadata APIs: To resolve and follow journal receivers, Gluesync additionally calls — on the same QSYS/QJOURNAL service program — QjoRetrieveJournalInformation (format RJRN0200) to discover the currently attached receiver, and QjoRtvJrnReceiverInformation (format RRCV0100) to read receiver metadata and walk the receiver chain. The file and field layout of each captured table (buffer offsets, field sizes, CCSID, date and time formats) is read once via the QDBRTVFD program (formats FILD0100 and FILD0200).

The legacy retrieval path (and the per-entity SQL-based path used internally) instead issues a DSPJRN-style query bounded by NBRENT(<Max number of items>). The direct API path described above is the default for Standard Mode.

Receiver buffer and per-call data volume

  • Receiver buffer size: Each QjoRetrieveJournalEntries call writes into a single contiguous receiver buffer whose size is governed by the agent-level Journal reader API buffer size in KB parameter (default 16384 KB). Independently of the configured value, every individual call is hard-capped at 10 MB, the maximum receiver size Gluesync requests from the IBM i API in one round trip.

  • Entries per call: The number of journal entries returned by a single call is therefore bounded by how many fit in that buffer, not by a fixed row count. When the buffer fills before the journal is drained, the API flags that more entries are available and Gluesync immediately issues a continuation call, with no polling delay, until it catches up with the head of the receiver (see the continuation mechanism described earlier).

  • Capture latency: In steady state, with the default 10,000 ms polling interval, committed changes are captured within roughly ten seconds of being written to the journal. Lowering the interval tightens latency at the cost of more frequent API calls when the journal is idle.

API filter selection

  • Journal codes: Restricted to R (record operations) and F (file operations).

  • Journal entry types: Strictly filters out the following transaction markers:

    • PT / PX: Row inserts (physical table records)

    • UP / UR: Row updates (after-images / rollback after-images)

    • UB / BR: Row updates (before-images / rollback before-images)

    • DL / DR: Row deletes

    • CR: Table truncations

Ingestion range boundaries and starting coordinates

  • First iteration or reset play: When a pipeline starts anew, or resumes after a "reset checkpoint" command, the API sets the receiver range to the current receiver and registers the starting coordinate with the current system time of the Gluesync client host.

  • Incremental iterations: Subsequent polling sequences explicitly inject the receiver name and receiver library into the parameters, configuring the starting coordinate with the exact sequence number from the previous iteration.

Advanced polling and stream control

Gluesync manages network and CPU overhead via an advanced, conditional streaming algorithm controlled by the "Journal reader polling interval in milliseconds" setting:

  • Idle polling: The configuration delay is strictly applied only when the API returns zero new entries, signalling that the journal has been completely drained.

  • Error backoff: If an API call triggers an unexpected error, Gluesync initiates an automated error handler loop, logging system diagnostic details and backing off before attempting to query the same position again.

Precision ingestion optimization features

  • Large Object (LOB) batch recovery: When a journal entry flags the presence of incomplete data rows (typically caused by Large Object columns like LOBs/CLOBs), instead of issuing singular database hits per row, Gluesync gathers affected Relative Record Numbers (RRNs) and executes high-speed batch executions (up to 1,000 records per SQL statement) utilizing optimized clauses to stitch back missing large object payloads.

  • Built-in recursion protection: To prevent loop feedback loops in bi-directional sync arrangements, enabling recursion protection instructs Gluesync to immediately parse the user metadata block of the incoming journal entry and silently discard any operations executed by the agent’s own connection username.