Search docs.molo17.com
    Troubleshooting guide Deploy and run

    Read Gluesync logs and diagnose errors

    Most Gluesync problems explain themselves in the Control Plane: the entity shows an error, the notification carries the stack trace, and Copy error gives you a message that support can read. This advanced guide starts there, and then shows what to do when that is not enough: where each Gluesync component writes its log, how to read a line, how to filter by pipeline, entity and agent, how to find the first real error, how to set the time zone, the level and the retention, and what the most common error messages mean.

    In this guide

    You will learn how to read what the Control Plane already tells you, collect a support bundle, find the log of each component on Linux and Windows, read the format of a line, filter a log by pipeline, entity and agent, find the first real error above a stack trace, fix the time zone of the logs, set the log level, retention and rollover, move the logs to another disk, and look up 20 common error messages.

    Quick information

    Goal

    Diagnose a Gluesync problem from the Control Plane first, then from the logs, and send support what it needs

    Audience

    Administrators and support engineers who run Gluesync with Docker Compose on Linux or Windows Server

    Level

    Advanced

    Time required

    About 45 minutes to read and try, 5 minutes to use as a reference

    Area

    Control Plane → entity page, Notifications hub, Settings → Support; host shell

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. Gluesync runs from the Docker Compose kit, in a folder that contains docker-compose.yml (for example /opt/gluesync on Linux or C:\Program Files\Gluesync on Windows Server). The paths below are relative to it;

    2. You can sign in to the Control Plane with a role that can change settings;

    3. For the steps marked CLI, you have a shell on the host: bash on Linux, PowerShell on Windows Server. The Control Plane steps need only a browser.

    The commands use the kit service names: gluesync-core-hub, gluesync-chronos, gluesync-conductor, reverse-proxy, prometheus and grafana. Run docker compose ps --services in the Gluesync folder to see yours. The examples come from a pipeline named Logs demo with the entity customers (pipeline 2dbcfeaf, source agent d1454a92, target agent b364a2e1, entity 50adb808). Your ids are different: you find them in the Control Plane and in Copy error.

    Guide steps

    Start from what the Control Plane shows

    UI

    Do this before you open any file. The entity page and the Notifications hub already hold the error, its stack trace and the ids you need to search the logs.

    1. Open the pipeline and select the entity whose status is Error;

    2. Read the red banner. Its title tells which side failed: Target write failed, Source read failed, Connection pool couldn’t start or Agent unhealthy. The text below it is the message of the database or the driver;

    3. Select Copy error and paste the text into your ticket or your notes;

    4. Select View notifications to open the Notifications hub filtered on this entity;

    5. Expand the row of the error and select Stack trace.

    Entity page banner Target write failed with the database message
    Figure 1. The banner names the failing side. View notifications (1) and Copy error (2) give you the text and the ids.
    Notifications hub with an expanded Error row Batch Update showing the message and the Stack trace
    Figure 2. The notification keeps the full message and the Java stack trace, without opening any log file.
    Expected result

    You know which side failed, you have the database message, and you have the text that Copy error produced. It has three parts: the title, the message, and a last line with the ids and the time, for example Pipeline: 2dbcfeaf · Entity: 50adb808 · Agent: b364a2e1 · 2026-10-28T20:51:16.426Z. The agent in that line is the one that reported the error (here the target agent), and the time is in UTC.

    The banner title and the message usually point to one row of the error table. Many problems end here: a constraint on the target, a wrong password, a stopped database.

    Error messages can contain row values. In the lab, a rejected row appeared as Failing row contains (9001, Ada Lovelace, ada@example.com, …​), in the banner, in the notification and in the log. Treat Copy error, notifications and logs as data that may be personal, and remove values before you share them outside your organisation.
    Collect a support bundle

    UI CLI

    When support asks for logs, send a compressed archive made by Gluesync, never single .log files or screenshots.

    1. Open Settings (the gear icon in the top bar) and select Support;

    2. Scroll to Log collection for support;

    3. To send the logs to a ticket, enter your Email and the Ticket ID (GSSD- and digits) and select Upload Logs to Support. The Core Hub host needs access to webdav.molo17.com or ftp.molo17.com;

    4. To keep the archive on your side, select Include database only if support asked for it, and select Collect Local;

    5. When Archive created appears, select the file name to download it.

    Settings Support page with Log collection for support
    Figure 3. Upload to a ticket (1), or collect an archive locally (2).
    Expected result

    The page shows Archive created: with a name such as gluesync-logs-1793218610868.tar.xz. Core Hub also leaves the archive in its log folder (logs/core-hub on the host). In the lab it was 47 KB for 620 KB of log.

    What is inside the archive from Collect Local:

    File Content

    gluesync-core-hub.log and the rolled archived/*.log.gz files

    The Core Hub log, as it is on disk, with no change to the lines

    .diagnostics-<number>/system-report.txt

    A Time block (the zone, the UTC offset, local and UTC time, and the TZ variable), the operating system, the Java heap and arguments, free disk space, and the container address. Use the Time block to place the log on a timeline

    .diagnostics-<number>/pipelines-config.json

    Pipelines, agents and entities with host and database names. It holds no passwords

    .diagnostics-<number>/configuration-files-dump.txt

    Auxiliary files found on disk, for example a custom logback.xml. It leaves out the files that hold secrets

    .diagnostics-<number>/docker-report.txt

    The Docker details. Inside the Core Hub container it says that Docker is not available

    When the Conductor is connected, the page also shows Collect full environment logs, statistics and metrics. Tick it when support needs the logs of the other modules. The collection can take several minutes and uses extra disk space while it redacts the logs.

    If the Control Plane is not available, use the command line of the kit. These collect the logs of the Compose services, the installer logs and the configuration with the passwords replaced by *:

    # Linux
sudo gluesync collect-logs --local-only
# Windows Server, elevated PowerShell
gluesync collect-logs --local-only

    Read Exporting / Uploading logs to support for the options, and Gluesync Control for Linux and Gluesync Control for the folder where the bundle is written.

    Find where each component writes its log

    CLI

    Gluesync is several containers. Each one writes to a different place, and not every one writes a file. Paths are relative to the Gluesync folder.

    Component File on the host (Linux and Windows Server) Command that always works

    Core Hub (agents run inside it)

    logs/core-hub/gluesync-core-hub.log. Rolled files, compressed, are in logs/core-hub/archived/gluesync-core-hub.<date>.<n>.log.gz

    docker compose logs --tail 200 gluesync-core-hub

    Chronos (schedules)

    logs/chronos/: play_pause.log (the scheduler), job_<id>.log (one file for each schedule), sdk_client.log and chronos.log. In the lab chronos.log stayed empty and the start-up and HTTP lines went only to the container output

    docker compose logs --tail 200 gluesync-chronos

    Conductor (modules, updates)

    logs/gluesync-conductor.log, and logs/conductor-self-update.log after a self-update. One JSON object per line

    docker compose logs --tail 200 gluesync-conductor

    Traefik (the reverse proxy)

    None. It writes to the container output only, and the kit has no access log. The level is log.level in proxy/traefik.yml

    docker compose logs --tail 200 reverse-proxy

    Prometheus and Grafana

    None in the kit. Prometheus keeps its data inside the container

    docker compose logs --tail 100 prometheus grafana

    Docker (what a container printed)

    Linux: /var/lib/docker/containers/<container-id>/<container-id>-json.log. Windows Server: C:\ProgramData\docker\containers\<container-id>\<container-id>-json.log

    docker inspect --format '{{.LogPath}}' <container> prints the path

    Linux and Windows differ only in the Gluesync folder and in the shell. Inside the Core Hub container the folder is always /opt/gluesync/logs (on Windows containers, C:\opt\gluesync\logs), and the kit maps it to logs/core-hub.

    The docker compose logs options you need most:

    docker compose logs --since 30m gluesync-core-hub          # the last 30 minutes
docker compose logs -t --tail 50 gluesync-core-hub         # add Docker's own UTC time
docker compose logs -f gluesync-core-hub gluesync-chronos  # follow two services
docker compose logs --no-log-prefix gluesync-core-hub > core-hub.txt
    Expected result

    You can open the right log for the component that fails: Core Hub for pipelines and agents, Chronos for schedules, Conductor for updates and module management, and the container output for the proxy.

    docker compose logs and the Core Hub log file are not the same text. In the kit the container output has no time zone offset and shows lines of libraries that the file does not keep, and the file has a thread name and an offset on every line. When you compare the two, use docker compose logs -t, which adds Docker’s own UTC time. The service name is gluesync-core-hub, not core-hub.
    Read one log line

    CLI

    This is a line from the Core Hub log file in the lab. The lab used the pattern of the next step, which adds the pipeline and entity ids (the line is cut at the end):

    2026-10-28T20:51:15.578Z [DefaultDispatcher-worker-1 @coroutine#223] [2dbcfeaf] 50adb808 c.m.g.p.t.[postgresql] PgSqlTargetAgentService ERROR - Error writing transaction on target for entity customers and transaction message -1305519641: java.sql.BatchUpdateException: ...
    Part Meaning

    2026-10-28T20:51:15.578Z

    Time. Z means UTC. A Core Hub with another time zone writes an offset, such as -03:00. See the time zone step

    [DefaultDispatcher-worker-1 @coroutine#223]

    The thread. Lines of one operation share the thread or the coroutine number

    [2dbcfeaf] 50adb808

    The pipeline id and the entity id. [core] and - mean that the line does not belong to one pipeline or entity. These two fields are written only when the pattern asks for them: the logback.xml of the kit does not, and Core Hub’s built-in pattern (used when no LOG_CONFIG_FILE is set) does. See the next step

    c.m.g.p.t.[postgresql] PgSqlTargetAgentService

    The logger. The part in square brackets is the agent: [corehub] is Core Hub itself, [postgresql] the PostgreSQL agent, [oracle-cdc] the Oracle LogMiner agent, [ibm-iseries] the IBM i agent. In this line, .p.t. abbreviates postgresql.target, so this is the target side

    ERROR

    The level: TRACE, DEBUG, INFO, WARN or ERROR

    - Error writing transaction …​

    The message. A long message, and a Java stack trace, continue on the next lines, which do not start with a date

    The container output (docker compose logs) has no offset and prints the logger in a shorter form. With the same pattern, a line looks like this:

    2026-10-28T20:46:26.594Z [DefaultDispatcher-worker-14 @coroutine#73] [2dbcfeaf] - [postgresql] PgSqlSourceAgentService INFO - Loading existing bootstrap file...

    Lines of the first seconds after a start can use an older layout, without the thread and without the offset: Core Hub rewrites the file layout once it has read its settings. Do not read that as a different time zone.
    Filter a log by pipeline, entity and agent

    CLI

    Search for the ids, and for the agent tag. All the commands below read the file. Replace the ids with yours. With the logback.xml of the kit, only some lines name the ids in their text, so start with the plain id and the agent tag. Add the ids to the pattern (see below) to get every line of a pipeline or an entity.

    LOG=logs/core-hub/gluesync-core-hub.log

grep -F '2dbcfeaf' "$LOG"                        # the lines that name the pipeline (kit pattern)
grep -F '[2dbcfeaf]' "$LOG"                      # everything of one pipeline (pattern with ids)
grep -F '[2dbcfeaf] 50adb808' "$LOG"             # one entity of that pipeline
grep -F '[postgresql]' "$LOG"                    # one agent type: [oracle-cdc], [ibm-iseries], ...
grep -E ' (WARN|ERROR) ' "$LOG"                  # only warnings and errors
grep -F 'd1454a92' "$LOG"                        # one agent, by agent id (in the text)
awk '$1 >= "2026-10-28T20:50" && $1 <= "2026-10-28T20:55"' "$LOG"   # a time window (UTC)
zgrep -F '[2dbcfeaf]' logs/core-hub/archived/*.gz   # the rolled files

    On Windows Server, in PowerShell:

    $Log = 'logs\core-hub\gluesync-core-hub.log'
Select-String -Path $Log -SimpleMatch '[2dbcfeaf] 50adb808'
Select-String -Path $Log -Pattern ' (WARN|ERROR) '

    The ids of the pipeline, entity and agent are in the URL of the entity page, in Copy error, and in the notification tags. The agent id also appears in text such as Entity customers (50adb808) of agent d1454a92 for pipeline 2dbcfeaf encountered and exception. The [pipeline] entity fields are filled only for work that belongs to one pipeline or entity: a request from the browser, or Core Hub start-up, shows [core].

    Add the pipeline and entity fields. The logback.xml of the kit does not write them. In the lab, its file wrote 2026-10-28T17:31:55.491 c.m.g.Ktor DEBUG …​ with no ids. Add them to the pattern of both appenders in shared/logback.xml:

    <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} [%t] [%X{pipelineId:-core}] %X{entityId:--} %c{1} %p - %m%X{requestInfo:-}%n</pattern>

    %X{requestInfo:-} adds ` | req=<id> by <user> (<ROLE>) via <method>` to the lines of a REST call, so you can tell who started an action. Core Hub reads the file again within about a minute, with no restart. Restart Core Hub afterwards (see the retention step).

    Expected result

    The commands return only the lines of your pipeline. In the lab, grep -F '[2dbcfeaf]' returned 94 lines and the entity filter 17 lines, from a log of 3,476 lines. The rest was REST requests and monitoring.

    Find the first real error

    CLI

    An error is rarely one line. A failed write printed about 200 lines in the lab, about 90 of them a stack trace: the first line is the cause, and the rest is the stack trace and the retries.

    1. List the errors with their line numbers and read the first one, not the last:

      grep -n ' ERROR ' "$LOG" | head -n 5

    2. Read what follows the first line. A stack trace is made of lines that do not start with a date:

      sed -n '944,960p' "$LOG" | cut -c1-240

    3. Find the root cause of a stack trace. It is the last Caused by of the chain:

      grep -n 'Caused by' "$LOG" | head

    4. Look at the lines just before the first error. Core Hub logs a WARN when it retries, and the first WARN is often older than the first ERROR.

    In the lab the order was:

    20:51:15.070Z ... PgSqlTargetOperations ERROR - Error executing batch for entity customers and table logs_demo.customers from ... Clear batch before throw exception.
20:51:15.578Z ... PgSqlTargetAgentService ERROR - Error writing transaction on target for entity customers ...: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO "logs_demo"."customers" ...
    ... was aborted: ERROR: new row for relation "customers" violates check constraint "credit_cap"
  Detail: Failing row contains (9001, ...)
Caused by: org.postgresql.util.PSQLException: ERROR: new row for relation "customers" violates check constraint "credit_cap"
20:51:16.486Z ... EntitySyncManager WARN  - Coroutine context no more active for entity customers, retrying...
20:51:16.503Z ... EntitySyncManager ERROR - Entity customers (50adb808) of agent d1454a92 for pipeline 2dbcfeaf encountered and exception

    The first line names the side (the target operation), the table and the transaction range. The text after ERROR: is the message of the database: here, a CHECK constraint on the target. The lines that follow are consequences: the retry (Retrying…​) and the generic encountered and exception line. A message about a stack trace deep inside Gluesync classes is almost never the cause; the cause is the message of the database or of the driver.

    Expected result

    You can say in one sentence what failed, on which side, on which entity, and at what time, and you can match the database message with a row of the error table.

    Optional, for advanced users

    Failing row contains (…​) is the database naming the row that it refused. It is the fastest way to find the source row: look it up with its key in the source and compare it with the rules of the target table.
    Choose the log level

    UI

    1. Open Settings → Support and scroll to Log level settings;

    2. Select the level in Log level: TRACE, DEBUG, INFO, WARN or ERROR. It applies at once, with no Save and no restart, and it is kept after a restart.

    Settings Support page with Log level settings marked 1 and Log retention settings with Max history in days
    Figure 4. Set the level (1) and the retention (2) in the Support page.
    Expected result

    The next lines of the Core Hub log are written at the new level. The level controls Core Hub and the agents, which run inside it. It does not change Chronos, the Conductor or Traefik.

    The default in the kit is DEBUG. This is what each level wrote in the lab, for a Core Hub with one idle pipeline and Prometheus scraping every 5 seconds:

    Level Lines per minute Size per minute What you lose by choosing it

    TRACE

    77

    10.6 KB

    Nothing. The most detail, for a short test

    DEBUG

    39

    6.1 KB

    Nothing. It adds every REST call (Ktor DEBUG …​ 200 OK) and the polling of the sources

    INFO

    12

    0.6 KB

    The REST calls and the details of each batch. Start, stop and error lines stay

    WARN, ERROR

    0

    0

    Everything that happens normally, such as Starting sync for entity …​ and Rows stored successfully. Use only when you know that you can find the cause from the notifications

    A busy pipeline writes far more. Use INFO in production, and switch to DEBUG for the time that you reproduce a problem. Support often asks for DEBUG; set it, reproduce the problem, collect the bundle, and set the level back.

    Set the retention and understand the rollover

    UI CLI

    1. In Settings → Support, scroll to Log retention settings;

    2. Enter Max history (Days) and Total size cap (GB), and select Save;

    3. Check the Core Hub log. At the next start Core Hub writes Configuring log retention: maxHistory=4 days, totalSizeCap=4GB.

    How the files roll over, as observed in the lab:

    • The active file is gluesync-core-hub.log. When it reaches the size limit, Core Hub moves it to archived/gluesync-core-hub.<date>.<n>.log.gz, compressed, and starts a new file;

    • The values of Log retention settings are applied at once, and again at every start. They win over the maxHistory and totalSizeCap written in shared/logback.xml;

    • The size limit of a file is fixed by Core Hub. A maxFileSize in logback.xml is not used after the start (see the result below);

    • Archives older than Max history (Days), or beyond Total size cap (GB), are deleted;

    • Clear Logs, in the same page, empties the active file in place and deletes the archives, without a restart. It works even when the disk is full. Do not delete the active file with rm. See Free disk space and manage cache and logs.

    Expected result

    The log folder never holds more than the days and the size that you set.

    Editing logback.xml while Core Hub runs is picked up within about a minute, but then Core Hub’s own settings are not applied again: the retention from the Control Plane, the fixed size limit, the compressed archives and the corrected layout (offset and thread). In the lab, after an edit with no restart, the lines went back to the old layout and a 577 KB file was rolled at the 100 KB written in the XML into an uncompressed archived/gluesync-core-hub.2026-10-28.0.log. After a restart, the same XML wrote a 156 KB active file with no rollover, because Core Hub applies its own limit of 100 MB. Restart gluesync-core-hub after every change to logback.xml.
    Move the logs to another disk

    Config

    Use a second volume, and tell Core Hub the new path in shared/logback.xml. The Control Plane has no setting for the path.

    1. In docker-compose.yml, add a volume to gluesync-core-hub (on Windows, a folder on the other drive):

          volumes:
      - ./logs/core-hub:/opt/gluesync/logs
      - /mnt/bigdisk/gluesync-logs:/opt/gluesync/logs-custom

    2. In shared/logback.xml, change the two paths of the FILE appender:

      <file>/opt/gluesync/logs-custom/${JAR_NAME:-gluesync}.log</file>
...
<fileNamePattern>/opt/gluesync/logs-custom/archived/${JAR_NAME:-gluesync}.%d{yyyy-MM-dd}.%i.log</fileNamePattern>

    3. Recreate the service:

      docker compose up -d gluesync-core-hub
    Expected result

    gluesync-core-hub.log is written in the new folder and the old one stops growing. Collect Local and Clear Logs follow the path of the FILE appender, so they work on the new folder. In the lab the first start after this change took several minutes, because the container changes the owner of every file under /opt/gluesync; wait for the Control Plane to answer.

    Docker’s own copy of the container output is separate. Cap it with max-size and max-file as described in Log rotation. In one support case a Windows Core Hub left a 100 GB *-json.log for this reason.

    Make the log time match your time zone

    Config

    The logs of every container are in UTC unless you set TZ. In the lab all five containers reported UTC with no setting, and the same time with TZ=America/Montevideo (UTC-3):

    Component Setting What the log shows

    Core Hub

    TZ=America/Montevideo

    File: 2026-10-28T17:06:38.686-03:00 …​. Container output: 2026-10-28T17:06:38.686 …​. Without TZ, the file ends with Z

    Chronos

    TZ=America/Montevideo. TIMEZONE is deprecated

    2026-10-28 17:06:18,984 - httpx - INFO - …​

    Conductor

    TZ=America/Montevideo

    "time":"10/28/2026, 17:05:36" (month first, no zone)

    Traefik

    TZ=America/Montevideo

    2026-10-28T17:03:14-03:00 INF …​

    Prometheus and Grafana

    TZ=America/Montevideo

    time=2026-10-28T17:03:35.702-03:00 …​ and t=2026-10-28T17:08:15.462916881-03:00 …​

    1. Add TZ to each service of docker-compose.yml, with a value in the form Region/City:

          environment:
      - SSL_ENABLED=true
      - TZ=America/Montevideo

    2. Recreate the services: docker compose up -d;

    3. Check with docker compose exec gluesync-core-hub date.

    Expected result

    New lines carry your local time, and date prints your zone. The Core Hub file keeps the offset on every line, so a bundle sent to another zone stays readable. system-report.txt in a support bundle names the zone and the TZ value.

    Write the value without quotes in the list form. - TZ="America/Montevideo" passes the quotes as part of the value. In the lab, Core Hub did not complain and silently went back to UTC (TZ=["America/Montevideo"] and a Z on every line). Chronos failed to start and restarted in a loop with Error initializing scheduler: 'No time zone found with key "America/Montevideo"' followed by SSL not available - falling back to HTTP mode. In the map form (TZ: America/Montevideo), quote marks are read by YAML and are fine.

    What else does not follow TZ, and is worth knowing when you compare times:

    • The Control Plane shows times in the time zone of your browser (the Timestamp column of the Notifications hub), and Copy error ends with a UTC time. The lab showed 9:51 PM in the table for the log line 20:51:15.578Z, from a browser one hour ahead of UTC;

    • The text of a notification is written with the time of the Core Hub zone. In the lab, CDC task for customers started in pipeline Logs demo at 17:12 10-28-2026 with TZ=America/Montevideo, and …​ at 20:00 10-28-2026 without it. The stored time stamp of the notification is always UTC;

    • Schedules in Chronos run in the time zone set for the schedule in the Control Plane. TZ changes the log lines, not the schedules.

    IBM i. The time of a journal entry is the wall-clock time of the IBM i, not UTC. In the lab, an IBM i in UTC+1 (QUTCOFFSET +0100, QTIMZON QP0100CET4) wrote a journal entry at 2026-10-28 21:22:56.077584 when the UTC time of the host was 20:22:56. The Gluesync log line about the same change shows the Core Hub zone, so the two differ by the offset of the IBM i. Gluesync corrects the lag figure by the clock difference between the IBM i and Core Hub, but not the times that it stores in checkpoints and in the technical timestamp column. The agent logs As400 system timezone: …​ at start.

    Oracle. LogMiner returns a time without a zone, in the time zone of the database server. The agent compares it with the clock of the Core Hub host and corrects it. When the two clocks differ, it logs Database clock is <offset> from this host. Redo timestamps carry no time zone, so they are corrected by that amount before any age is computed from them. A visible gap between a commit in Oracle and the log line means that the clock of the database server or of the host is wrong, not that the zone is wrong. Fix the clock with NTP.

    Look up a common error

    The table lists messages from the Control Plane and from the logs, with what each one means and the first thing to do. Each row names a guide for the details. The messages are as they appear in Gluesync 2.3; the names of tables, hosts and ids are examples.

    Message Meaning What to do

    Check connection failed. Cannot reach host "<host>" at port <port>. Please check that the database is running and reachable (network, firewall, TLS settings) and try again.

    The agent cannot open a connection to its database: it is stopped, the name or port is wrong, or a firewall or TLS setting blocks it

    Test the host and port from the Core Hub host, then start the database or fix the address in Edit source or Edit target and select Test and continue. See Diagnose a stalled pipeline or entity

    Failed to initialize pool: FATAL: password authentication failed for user "<user>" (title Invalid password Error)

    The database refuses the user name or password, often after a password rotation

    Enter the new password in Edit source or Edit target and select Test and continue. See Diagnose a stalled pipeline or entity

    Failed to initialize pool: The connection attempt failed.

    The host does not answer: wrong host name or port, or the database is down. The edit is saved even when the test fails

    Correct the host and port and test again. An entity without changes can stay Active while it does not replicate, so check with Count both

    Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=0)

    The pool of the target could not get a connection in 30 seconds: the target database is stopped or unreachable

    Start the target. Changes wait in the source cache and are written when it is back. See Free disk space and manage cache and logs for what a long outage costs

    Warm-up failed, check the notification for details of the error. A new attempt will be made in <n> seconds from now

    A source or target agent could not start its connection after a restart or after maintenance. Core Hub retries after 10, 20 and 30 seconds, then every 5 minutes

    Read the Check connection failed notification that came with it, fix the database, and wait for the next attempt or toggle maintenance mode. See Pipeline maintenance mode

    Source read failed: ERROR: replication slot "gs_<entity>_slot" does not exist

    The PostgreSQL replication slot of the entity was dropped, so Gluesync cannot continue from where it stopped

    Start the entity with a full copy, which creates the slot again. See Start a snapshot and CDC replication from the Control Plane

    Target write failed with violates check constraint, violates not-null constraint or duplicate key value

    The target database refused a row. The row is in the message (Failing row contains (…​)). The entity retries and stays in error until the cause is removed

    Fix the constraint or the row on the target, and the entity writes the row at the next retry. See Find and fix row differences and Insert conflict strategy

    value too long for type character varying(<n>), String or binary data would be truncated

    The target column is shorter than the source value

    Widen the target column. Recreating the replication helps only until the next long value. See Map IBM i data types

    Error invoking mapping function: java.lang.NullPointerException …​

    A field function (UDF) failed on a row, often on a null or an empty value

    Test the function on the failing value, and handle null. See Write and test a UDF for common conversions

    [Thread 1] ORA-00942: table or view does not exist (title Oracle LogMiner error), ORA-01031: insufficient privileges

    The LogMiner user lacks a grant on a catalog view or on DBMS_LOGMNR. The pipeline stays Active and only the notification shows it

    Grant the privileges listed in Set up Oracle LogMiner end to end

    Oracle LogMiner: redo gap (ORA-01291)

    The redo that holds some changes is no longer available (archive logs were removed). Gluesync moved to the current position and skipped the changes

    Run a new snapshot of the affected entities, and keep archive logs until Gluesync has read them. See Set up Oracle LogMiner end to end

    ORA-12514, ORA-12528, ORA-01089 after a database restart

    The Oracle listener or the database is shutting down or starting

    Wait. In the lab the entities stayed Active and replicated again within seconds of the restart

    ORA-12505: Cannot connect to database. SID <name> is not registered with the listener, ORA-12263

    The connection type or the service name is wrong, or the wallet folder is

    Choose Service name or SID to match the database. See Set up Oracle LogMiner end to end

    IO Error: Socket read timed out (Oracle)

    A query ran longer than the read timeout. In one case the indexes of the Gluesync log table were UNUSABLE after a partition drop

    Rebuild unusable indexes (ALTER INDEX …​ REBUILD ONLINE), and raise the read timeout of the agent if the table is large

    Journal checkpoint outdated: Last read receiver (…​) not found in journal …​ Re-Sync needed for all entities.

    IBM i: the journal receiver that Gluesync read last was deleted before it was read

    Run a new snapshot, and keep receivers until they are read. See Troubleshoot IBM i journals and receivers

    Unable to activate session (error -83713) (Informix)

    The saved position is inside a logical log that was removed, for example after the logical logs were replaced

    Do not remove logical logs before Gluesync has read past them. To restart from the current position, work with MOLO17 Support: the pipeline checkpoint reset is in Settings → Danger zone

    Pipeline with id <id> is in maintenance. Please wait until the maintenance mode is disabled. (HTTP 406), Entity sync start failed for <entity> due to maintenance mode. Stop all remaining jobs

    A start was requested while the pipeline is in maintenance mode. A schedule that fires during maintenance gets the same answer

    Exit maintenance, then start again. See Pipeline maintenance mode

    Received <request> but the database of the source agent '<name>' (<id>) is not initialized yet, Impossible to …​ before the database of the …​ is initialized

    The agent has no connection to its database yet, so it cannot change entities. Older versions wrote Impossible to upsert entities before database initialization

    Open Edit source or Edit target, select Test and continue, and check Check connection failed in the notifications

    Invalid license key. Please check your license file. (log, followed by SignatureVerificationException), License loaded, status: INVALID, License expired or invalid. Please try again later or contact an admin. (HTTP 410 when you start an entity), The license expired on <date>. Please contact sales@molo17.com.

    The license file is expired, was changed, or is damaged. In the lab a damaged file made Core Hub start, but every start of an entity was refused

    Install a valid license file and restart Core Hub. See Licensing and license management

    database or disk is full (Error code: 13), Request failed with status code 500 when saving a setting

    The disk of the Core Hub data folder is full

    Free space in the order of Free disk space and manage cache and logs

    Chronos: Error in connection: [Errno -2] Name or service not known, Reconnection attempt 10 (cycle 1) failed, Connection to wss://gluesync-core-hub:1717/ext-module timed out. Conductor: connect ECONNREFUSED <ip>:1717, Failed to discover CoreHub

    Chronos or the Conductor cannot reach Core Hub, which is stopped, still starting, or on another address

    Check docker compose ps and wait for Core Hub to answer. The modules retry by themselves (Chronos logs Waiting 16.0 seconds before next reconnection attempt…​) and reconnect when Core Hub is back

    Error initializing scheduler: 'No time zone found with key "<zone>"' (Chronos)

    The time zone variable holds quotes or a name that does not exist

    Use TZ=Region/City without quotes. See the time zone step

    unable to find the IP address for the container "<name>": the server is ignored (Traefik)

    The reverse proxy tried to route to a container that was not running yet

    Wait for Core Hub to start, then check docker compose ps. The Control Plane does not open until the route exists
    Verify the result

    You have successfully learned to read your logs if:

    • You can open an entity error, copy it, and read its stack trace in the Notifications hub;

    • You can name the log file or the command for Core Hub, Chronos, the Conductor and Traefik on your host;

    • A search for the pipeline and entity ids returns only their lines, and you can say what the first error of an incident is;

    • The log time matches your zone, or you know the offset;

    • The log level is INFO, and the retention and the folder of the logs are the ones that you chose;

    • You have a support bundle made by Collect Local or gluesync collect-logs.

    Troubleshooting

    Problem Possible cause What to do

    The log file has no lines for hours, or logs/core-hub is empty

    The log level is WARN or ERROR, or the log path in logback.xml points to another folder

    Check Log level settings and the <file> path of the FILE appender in shared/logback.xml

    After deleting the log with rm, no new file appears

    Core Hub keeps the deleted file open

    Restart gluesync-core-hub, and use Clear Logs next time

    The lines of the file have no offset and no thread name

    logback.xml was edited while Core Hub ran, or the line is from the first seconds after a start

    Restart gluesync-core-hub

    docker compose logs core-hub says no such service

    The service of the kit is called gluesync-core-hub

    Run docker compose ps --services

    The log shows Z but you set TZ

    The value has quotes, or the container was not recreated

    Remove the quotes, run docker compose up -d, and check with date

    Chronos restarts in a loop after you set a time zone

    TIMEZONE="Region/City" with quotes

    Use TZ=Region/City without quotes

    Notifications show a time one or two hours different from the log

    The Notifications hub uses the time of your browser, the log uses UTC or TZ

    Compare with the UTC time at the end of Copy error

    The Core Hub archive is much larger than the log

    The logs of many days are on disk

    Lower Max history (Days) and Total size cap (GB), or use Clear Logs before you reproduce a problem

    Related guides

    Contact MOLO17 Support if you need assistance with logs or with an error that is not in the table.