Search docs.molo17.com
    Troubleshooting guide Connectors

    Troubleshoot IBM i journals and receivers

    Changes of an IBM i table stop reaching the target, an entity will not start, or a notification says that a journal checkpoint is outdated. This guide shows how to find out which part of the journal chain is at fault, what the Control Plane shows for each failure, which CL commands the IBM i administrator runs, and how to recover without guessing which data is missing.

    In this guide

    You will learn how to see where Gluesync is in the journal, fix a table that is not journaled or is journaled with the wrong images, recover when a journal receiver was deleted before Gluesync read it, choose how to restart CDC (there is no timestamp option), check that the receiver number wrap from 9999 to 0000 is harmless, decide whether an entity needs the dedicated journal reader, and see what the Gluesync work library holds and which settings make it grow.

    Quick information

    Goal

    Find the cause of an IBM i CDC problem from the journal side, fix it and resynchronize only the data that is really missing

    Audience

    Gluesync users and IBM i administrators who run Db2 for IBM i as a source

    Level

    Intermediate

    Time required

    About 45 minutes to read, less to apply one fix

    Area

    Control Plane → Pipelines, Settings → Danger zone, Query Studio; IBM i CL commands

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a pipeline with a Db2 for IBM i source and at least one entity on a journaled physical file (the examples use the pipeline IBM i journal demo, the library GSDOCSJRN, the journal JRNDEMO and the table ORDERS; replace them with your own names);

    2. You can run CL commands on the IBM i, from a 5250 session or with CALL QSYS2.QCMDEXC('<command>') from any SQL client, or you can ask the IBM i administrator to run them;

    3. You can pause and start entities and open Settings in the Control Plane.

    Several commands in this guide delete or end journaling (DLTJRNRCV, ENDJRNPF). The examples were run only against a test library created for this guide. Run them on production objects only after the administrator has checked that no other product or recovery job still needs those receivers.

    How Gluesync follows a journal

    Gluesync reads the journal with the IBM i journal APIs. It does not lock or keep any receiver, and it never deletes one. These facts explain every failure in this guide:

    • The read position of a journal is a checkpoint: library and name of the journal, library and name of the receiver, and the 20-digit sequence number. Core Hub stores it, not the IBM i. One checkpoint serves every entity whose table is in the same journal;

    • To move from one receiver to the next, Gluesync follows the link that the IBM i keeps between receivers (the NEXT_JOURNAL_RECEIVER of the receiver it is reading), not the receiver name;

    • The Last trans. column of the entity table shows the last transaction as <library>-<receiver>-<sequence>. The value is cut in the table, but the same three parts are in the _RECEIVER_LIBRARY, _RECEIVER_NAME and _SEQUENCE_NUMBER technical fields, if you enabled them (see Technical fields);

    • A changed row needs the table to be journaled with before and after images (IMAGES(*BOTH)). The images belong to the journaled file, not to the journal.

    Guide steps

    Optional: see where Gluesync is in the journal

    UI + API + SQL

    Skip this step if you already know the journal position. It gives you the two facts every later step compares: where Gluesync stands, and which receivers exist.

    1. Read the journal position of the pipeline from the Core Hub REST API. <pipelineId> is the identifier shown in the pipeline URL (for example 9fc6ac3b) and <token> is a valid API token (see Core Hub APIs):

      curl -k -H "Authorization: Bearer <token>" \
        https://<core-hub-host>/pipelines/<pipelineId>/read-checkpoints

      The answer has one item for each journal. The fields object holds receiverLibrary, receiverName and sequenceNumber;

    2. In the Control Plane, open Query Studio;

    3. In SQL-capable Agent, select your pipeline and then its (SOURCE) agent;

    4. Run the query that lists the receivers of your journal library, oldest first:

      SELECT JOURNAL_RECEIVER_NAME AS RECEIVER, STATUS,
             FIRST_SEQUENCE_NUMBER AS FIRST_SEQ,
             LAST_SEQUENCE_NUMBER AS LAST_SEQ,
             NEXT_JOURNAL_RECEIVER AS NEXT_RECEIVER
      FROM QSYS2.JOURNAL_RECEIVER_INFO
      WHERE JOURNAL_RECEIVER_LIBRARY = 'GSDOCSJRN'
      ORDER BY CAST(FIRST_SEQUENCE_NUMBER AS BIGINT)
    Query Studio on the IBM i journal demo source agent with the receiver query marked 1
    Figure 1. Run the query (2) in the editor (1) on the IBM i source agent and read the receiver chain in the results (3).
    Expected result

    The API answer shows the receiver and sequence of the last position Gluesync read, for example JRCVBB0002 and 00000000000000000015. The query lists one row for each receiver that still exists. Every receiver except the one with STATUS ATTACHED has the name of the next receiver in NEXT_RECEIVER, and the attached receiver has NULL. The receiver named in the API answer must be in this list.

    Query Studio opens the source in read-only mode. To list the files of a journal and their images, run SELECT OBJECT_NAME, JOURNAL_NAME, IMAGES FROM QSYS2.JOURNALED_OBJECTS WHERE JOURNAL_LIBRARY = 'GSDOCSJRN'. See Query data with Query Studio.

    Fix a table that is not journaled, or journaled with the wrong images

    UI + CL

    1. Start the entity (Start sync, then Start entity). If the table is not journaled for CDC, the start fails and a red notification appears;

    2. Read the notification. Its title is Missing Journal Table Definition and its text starts with Missing journal configuration for entity <name> and table <LIBRARY.TABLE>. Sync cannot be started. Only snapshot can be executed. The entity stays in the Hold group;

    3. Ask the IBM i administrator for the journaling of the file:

      SELECT OBJECT_LIBRARY, OBJECT_NAME, JOURNAL_LIBRARY, JOURNAL_NAME, IMAGES
      FROM QSYS2.JOURNALED_OBJECTS
      WHERE OBJECT_LIBRARY = 'GSDOCSJRN' AND OBJECT_NAME = 'NOJRN'
    4. If the query returns no row, the file is not journaled. Start journaling with both images:

      STRJRNPF FILE(GSDOCSJRN/NOJRN) JRN(GSDOCSJRN/JRNDEMO) IMAGES(*BOTH) OMTJRNE(*OPNCLO)
    5. If IMAGES is *AFTER, end journaling and start it again with both images:

      ENDJRNPF FILE(GSDOCSJRN/AFTERONLY) JRN(GSDOCSJRN/JRNDEMO)
      STRJRNPF FILE(GSDOCSJRN/AFTERONLY) JRN(GSDOCSJRN/JRNDEMO) IMAGES(*BOTH) OMTJRNE(*OPNCLO)
    6. Start the entity again with Full copy, so that rows changed while the file was not journaled reach the target (see [resync-entities]).

    Sync changes dialog for the nojrn entity with the red Missing Journal Table Definition notification marked 1
    Figure 2. The start fails with Missing Journal Table Definition (1); the entity stays on hold.
    Expected result

    After STRJRNPF, the query returns the file with IMAGES equal to *BOTH and the entity starts. A row inserted on the IBM i reaches the target within the polling interval. The same notification and the same fix apply to a file journaled with *AFTER images only.

    A table created with SQL CREATE TABLE in a library made with CRTLIB is not journaled. The IBM i answers with the warning SQL7905 ("table created but not journaled"). Create the journal for the library, or run STRJRNPF on every file, before you create entities.

    Ending journaling on a file whose entity is running does not raise any error. The entity stays Active, but changes are not captured. After STRJRNPF, new changes flow again, while inserts, updates and deletes made in between never reach the target. Resynchronize the entity with Full copy and Empty the table first: Keep and merge does not remove the rows that were deleted on the IBM i in the meantime.

    Recover when a journal receiver was deleted before Gluesync read it

    UI + CL

    This happens when a receiver that Gluesync has not finished reading is deleted, for example by a housekeeping job, while the entities are stopped or the reader is behind. Receivers are deleted in the order in which they were attached: DLTJRNRCV of a receiver fails with CPF7023 while older receivers still exist. This is the command that was used for the test (DLTOPT(*IGNINQMSG) answers the "not saved" inquiry):

    DLTJRNRCV JRNRCV(GSDOCSJRN/JRCVBB0003) DLTOPT(*IGNINQMSG)
    1. With the receiver deleted, start the entity with Where it stopped (the default for an entity that has run);

    2. Wait for the first journal read, then open the notification Journal checkpoint outdated.

    Sync changes dialog with the Where it stopped tile marked 1 and the Start entity button marked 2
    Figure 3. Where it stopped (1) is the normal restart; Start entity (2) begins it.
    Journal checkpoint outdated notification: last read receiver GSDOCSJRN/JRCVBB0005 and sequence 53 not found
    Figure 4. The notification names the missing receiver, the receiver Gluesync switched to, and says that a resync is needed.
    Expected result

    The notification reads Last read receiver (receiver <library>/<receiver> and sequence <n>) not found in journal <library>/<journal>. Automatically switched to current receiver (<library>/<receiver> and sequence <n>). Re-Sync needed for all entities. The entities are started anyway, and new changes written after the switch reach the target.

    What the entity did and what is lost:

    • Gluesync moves the checkpoint to the end of the current receiver at that moment. Everything before that point is not read: the changes stored in the deleted receivers, and also the changes already written in the receivers that still exist. In the test, a row inserted after the rotation, in the surviving receiver, was lost together with the rows of the deleted one;

    • The data to resynchronize is every entity whose table is in that journal. Entities on other journals are not affected;

    • Gluesync does not resynchronize by itself.

    To resynchronize every entity of the journal:

    1. Pause the entity (Stop sync, then Pause entity);

    2. Select Start sync and, in Start from, select Full copy;

    3. In Data already in the target, select Keep and merge if no row was deleted on the IBM i in the meantime, or Empty the table first if rows may have been deleted;

    4. When you empty the table, tick I understand that rows in the target will be deleted. Start entity stays disabled until you tick it. If you do not see the box, scroll the dialog;

    5. Select Start entity.

    Sync changes dialog with Full copy marked 1
    Figure 5. Full copy (1) with Empty the table first (2) realigns the table. Confirm the deletion (3), then start (4).
    Expected result

    After the copy, a count and a comparison of the keys on the source and on the target agree, and a row inserted later reaches the target. Without the resync, the rows lost in the gap were missing from the target.

    To prevent the problem, keep every receiver from the one named in the read-checkpoints answer (step 1) onward until Gluesync has moved past it. The answer shows the position of the shared reader. The retention of Gluesync (Source change retention in hours, 24 by default) applies to the local cache of Gluesync, not to the receivers on the IBM i.

    Restart CDC from a chosen point

    UI

    Gluesync 2.3 cannot start CDC from a timestamp or from a sequence number that you type: neither the start dialog nor the start API has such a field. Choose the closest option:

    Option What it does Use it when

    Where it stopped

    Continues from the last checkpoint. Changes written while the entity was paused are delivered; in the test, rows written while the entity was stopped across three receiver changes arrived in order.

    The journal still holds every receiver since the entity stopped.

    Now

    Forgets the checkpoint and follows only new changes. A row written while the entity was stopped never reaches the target; a row written after the start does.

    You do not need the changes made while the entity was stopped, or the target is already aligned.

    Full copy

    Copies every row, then follows live changes from the moment the copy begins.

    You must realign the table, for example after a deleted receiver.

    Settings → Danger zone → Pipeline checkpoint reset

    Pauses every entity of the pipeline, deletes the disk cache and the stored read checkpoint of the journal.

    The position of the journal itself is wrong for the whole pipeline. The Control Plane advises using it only under the supervision of MOLO17 Support.

    To reset the checkpoint of the pipeline:

    1. Select Settings (the gear icon) and then Danger zone;

    2. In Pipeline checkpoint reset, select the pipeline in Select a pipeline;

    3. Select Reset pipeline checkpoint;

    4. Read the dialog Confirm Reset and select Confirm.

    Settings dialog on Danger zone with the pipeline selector of Pipeline checkpoint reset marked 1 and the Reset pipeline checkpoint button marked 2
    Figure 6. Select the pipeline (1), then Reset pipeline checkpoint (2).
    Expected result

    Every entity of the pipeline is paused and the read-checkpoints answer (step 1) is empty. Start each entity with Now if the target is already aligned with the source, or with Full copy otherwise. Changes made between the reset and the start are not delivered.

    Use Pipeline checkpoint reset only when no other option works. The dialog states that every entity must be resynchronized, because changes not yet replicated may be lost. After the reset, start the entities with Now or Full copy, the two options that were checked after a reset.

    Check that the receiver number wrap is harmless

    CL + SQL

    The receiver name does not matter to Gluesync: it follows the chain of links, so a name that goes from 9999 back to 0000 needs no action. To check it on your system:

    1. Look at how the IBM i names the next receiver (CHGJRN …​ JRNRCV(*GEN) generates it). With a short name such as JRCV9999 the number simply gets longer (JRCV10000). With a name that already has ten characters, such as JRCVAA9999, the number wraps to JRCVAA0000;

    2. Run the receiver query of step 1 and check that every receiver except the attached one names its successor, also across the wrap.

    CRTJRNRCV JRNRCV(GSDOCSJRN/JRCVAA9998)
    CHGJRN JRN(GSDOCSJRN/JRNDEMO) JRNRCV(GSDOCSJRN/JRCVAA9998)
    CHGJRN JRN(GSDOCSJRN/JRNDEMO) JRNRCV(*GEN)
    CHGJRN JRN(GSDOCSJRN/JRNDEMO) JRNRCV(*GEN)
    Expected result

    The query shows JRCVAA9998, JRCVAA9999, JRCVAA0000 and JRCVAA0001 in this order, each naming the next one. An entity that was running during the changes captured a row in each receiver, with _RECEIVER_NAME JRCVAA9998, JRCVAA9999 and JRCVAA0000. An entity that was stopped during a wrap and a change with SEQOPT(*RESET) (the sequence number restarts at 1 in the new receiver) delivered every row in order when it was started with Where it stopped.

    Decide whether an entity needs the dedicated journal reader

    UI

    By default, all entities of one journal share one reader: it reads the journal once and keeps the changes in a disk cache for each entity. With Use Dedicated Journal Reader an entity reads the journal on its own, directly to the target, without the shared reader and its cache, and without saving its position in the checkpoint of the journal.

    Enable it for a table that needs isolated, high-throughput reading. Every dedicated entity opens its own journal reads on the IBM i, so enable it only where needed (see the journal architecture page).

    1. Pause the entity;

    2. In the entity row, select Options (the three-dot icon) and then Edit settings;

    3. On the Source tab, switch Use Dedicated Journal Reader to On;

    4. Close the dialog and select Confirm and continue;

    5. Start the entity with Where it stopped.

    Settings for orders dialog on the Source tab with the Source tab marked 1 and the Use Dedicated Journal Reader switch set to On marked 2
    Figure 7. Open the Source tab (1) and switch Use Dedicated Journal Reader (2) to On.
    Expected result

    Rows inserted on the IBM i reach the target, also after a CHGJRN ... JRNRCV(*GEN). To verify that the entity uses the dedicated reader, read the read-checkpoints answer (step 1) before and after a few changes: the receiver and the sequence of the journal do not move, while the Last trans. column of the entity advances.

    See what the Gluesync work library holds

    Config + SQL

    Gluesync keeps service objects in the library named in Gluesync library of the source connection (default GLUESYNC; the test used GSDOCSJWK). It never stores application data there. List its objects:

    SELECT OBJNAME, OBJTYPE, OBJSIZE, OBJCREATED
    FROM TABLE(QSYS2.OBJECT_STATISTICS('GSDOCSJWK', '*ALL')) X
    Setting or mode What appears in the library

    Default settings (journal reader on the IBM i journal APIs)

    Nothing. In the test the library was empty while entities captured changes.

    Legacy mode journal reader (use DSPJRN command) on

    One *FILE named JRN plus eight hex characters, about 60 KB, that holds the output of DSPJRN. Gluesync recreates it at every read, so its creation time moves and its size does not grow.

    Enable DDS Discovery and Enable multi-member physical file support

    DDSKEYS and one SQL alias for each multi-member entity, as listed in Source data from Db2 for IBM i. These objects were not produced in the test.

    Delete objects of the library only while all the entities of the pipeline are paused. What can really grow on the IBM i is the set of journal receivers that your housekeeping keeps, not this library.

    Symptom, cause and fix

    Symptom Cause Fix

    Start fails with Missing Journal Table Definition; the entity stays on hold

    The file is not journaled, or is journaled with IMAGES(*AFTER), or the entity is on a logical file or view

    Run STRJRNPF ... IMAGES(*BOTH) (after ENDJRNPF if the images are wrong), or create the entity on the physical file; start the entity with Full copy

    The entity is Active and green, but changes do not reach the target

    Journaling of the file was ended (ENDJRNPF); no error is raised

    Check QSYS2.JOURNALED_OBJECTS, run STRJRNPF, then Full copy with Empty the table first

    Journal checkpoint outdated with Re-Sync needed for all entities

    The receiver named in the checkpoint no longer exists on the IBM i

    Resynchronize every entity of the journal with Full copy; keep receivers until Gluesync has read them

    After the notification, the target misses rows, also rows that are still in existing receivers

    Gluesync moved to the end of the current receiver

    Full copy of every entity of the journal; rows deleted on the source need Empty the table first

    You need CDC from a given time or sequence number

    The Control Plane offers Where it stopped, Now and Full copy only

    Pick the closest option from the table in "Restart CDC from a chosen point"

    After Pipeline checkpoint reset the entities do not resume as before

    The reset deletes the stored position and the cache of the pipeline

    Start each entity with Now or Full copy

    The receiver name goes from 9999 to 0000

    IBM i naming of the next receiver

    Nothing: Gluesync follows the chain. Check the chain with the receiver query

    Start entity stays disabled in the Full copy dialog

    Empty the table first needs the confirmation box

    Scroll the dialog and tick I understand that rows in the target will be deleted

    CL commands for the IBM i administrator

    Task Command or query

    Journal the file with both images

    STRJRNPF FILE(lib/file) JRN(lib/jrn) IMAGES(*BOTH) OMTJRNE(*OPNCLO)

    End journaling of the file

    ENDJRNPF FILE(lib/file) JRN(lib/jrn)

    List the journaled files and their images

    SELECT OBJECT_NAME, JOURNAL_NAME, IMAGES FROM QSYS2.JOURNALED_OBJECTS WHERE JOURNAL_LIBRARY = 'lib'

    Create a receiver and attach it to the journal

    CRTJRNRCV JRNRCV(lib/rcv) then CHGJRN JRN(lib/jrn) JRNRCV(lib/rcv)

    Attach a new receiver with a generated name

    CHGJRN JRN(lib/jrn) JRNRCV(*GEN)

    Attach a new receiver and restart the sequence number

    CHGJRN JRN(lib/jrn) JRNRCV(*GEN) SEQOPT(*RESET)

    Delete a detached receiver (oldest first)

    DLTJRNRCV JRNRCV(lib/rcv) DLTOPT(*IGNINQMSG)

    List the receivers, their range and successor

    SELECT JOURNAL_RECEIVER_NAME, STATUS, FIRST_SEQUENCE_NUMBER, LAST_SEQUENCE_NUMBER, NEXT_JOURNAL_RECEIVER FROM QSYS2.JOURNAL_RECEIVER_INFO WHERE JOURNAL_RECEIVER_LIBRARY = 'lib'

    Show the attached receiver and the receiver management options

    SELECT JOURNAL_NAME, ATTACHED_JOURNAL_RECEIVER_NAME, MANAGE_RECEIVER_OPTION, DELETE_RECEIVER_OPTION FROM QSYS2.JOURNAL_INFO WHERE JOURNAL_LIBRARY = 'lib'

    Run a CL command from SQL with CALL QSYS2.QCMDEXC('<command>').

    DLTLIB of a library that still holds journal receivers that were never saved waits for an answer to the inquiry message CPA7025, so the SQL session looks frozen. Delete the receivers first, oldest first, with DLTJRNRCV …​ DLTOPT(*IGNINQMSG), and then delete the library.

    Verify the result

    You have successfully diagnosed and fixed the journal problem if:

    • QSYS2.JOURNALED_OBJECTS lists every table of your entities with IMAGES *BOTH;

    • Every receiver that exists, except the attached one, names its successor in NEXT_JOURNAL_RECEIVER, and the receiver of the read-checkpoints answer is among them;

    • A row inserted, updated and deleted on the IBM i reaches the target within the polling interval;

    • After a resync, the row count and the keys of each entity match between the source and the target;

    • No entity of the pipeline shows an unread Journal checkpoint outdated notification.

    Contact MOLO17 Support if the notification keeps coming back or the journal position does not move.