Search docs.molo17.com
    Operational guide Deploy and run

    Diagnose high CPU usage

    High or growing CPU usage can come from the Core Hub container, from an agent, from the source database or from the target database. The fix is different for each. This guide shows how to measure which one it is, the known causes, which setting to change and how to check that the CPU went down.

    In this guide

    You will learn how to find which container or system uses the CPU, how to read the Host CPU, JVM and garbage-collection metrics, which settings raise the CPU of Core Hub and of an IBM i source (entity polling interval, dedicated journal reader, connection pool, number of entities, snapshot concurrency, heap), what to change, and how to verify the result.

    Quick information

    Goal

    Find the system that uses the CPU and bring the load back to normal without losing changes

    Audience

    Administrators who run Gluesync with Docker Compose

    Level

    Intermediate

    Time required

    About 30 minutes

    Area

    Control Plane → pipeline → entity Fine tuning, Edit source; Grafana Core Hub metrics; host shell

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You can sign in to the Gluesync Control Plane with a role that can change settings;

    2. You have shell access to the host that runs Gluesync;

    3. You can see the CPU of the source and target databases with the tools you already use (cloud console, top, your monitoring);

    4. You know when the CPU became high, and what changed just before: a new entity, a new pipeline, an edited polling interval, an update.

    Change one setting at a time and measure again after each change. Write down the CPU before you start. Without a number from before, you cannot tell whether a change helped.

    Guide steps

    Find which system uses the CPU

    CLI

    1. On the Gluesync host, show the CPU and memory of every container once:

      docker stats --no-stream

    2. Read the CPU % column. A container at 100% uses one full core. Core Hub can use more than one core, so 300% means three cores;

    3. Note which container is high: Core Hub, Chronos, Conductor, Prometheus, Grafana or Traefik;

    4. Compare with the CPU of the source database and of the target database, from their own tools.

    Example output of a healthy Core Hub in a support log, on a host with 31 GiB of memory:

    NAME                           CPU %     MEM USAGE / LIMIT
gluesync-gluesync-core-hub-1   27.05%    1.084GiB / 31.06GiB
gluesync-grafana-1             0.60%     173.3MiB / 31.06GiB
gluesync-gluesync-chronos-1    0.23%     92.51MiB / 31.06GiB
gluesync-reverse-proxy-1       0.26%     27.96MiB / 31.06GiB
    Expected result

    You know which side is high. Use the table below to choose the next step.

    What is high What it usually means Go to

    The Core Hub container

    Many entities, a busy snapshot, heavy garbage collection, or a heap that is too small

    Lower the CPU that Core Hub uses

    The CPU of an IBM i

    Dedicated journal readers with a short polling interval, or a short agent polling interval

    Lower the CPU that Gluesync causes on an IBM i

    The CPU of the target database

    The target is the limit, not Gluesync. Gluesync waits for it

    Tune snapshot and CDC throughput safely

    The number of connections or the load on the source

    The connection pool is larger than the source can serve

    Limit the connections to a source or target

    Nothing is high, but the replication is slow

    The limit is a setting or the network, not the CPU

    Tune snapshot and CDC throughput safely
    Read the CPU and JVM metrics of Core Hub

    UI CLI

    docker stats shows a moment. The metrics show how the CPU moves over time.

    1. In Grafana, open the Core Hub metrics dashboard (see Monitor pipelines in Grafana);

    2. Read the Host CPU panel. It shows the system-wide CPU load of the Core Hub host;

    3. Scroll to Host & JVM. Read JVM heap and GC pauses and live threads;

    4. To query the values directly, use these metrics from the Metrics Reference in Prometheus:

      Metric Meaning

      gluesync_system_cpu_load

      Overall system CPU load as a ratio from 0 to 1

      gluesync_processor_cpu_load

      Per-processor CPU load as a ratio from 0 to 1

      jvm_gc_overhead

      Approximate fraction of CPU time spent in garbage collection, from 0 to 1

      jvm_memory_used_bytes

      Memory used by the JVM, by pool. Use the heap area

      jvm_threads_live_threads

      Number of live threads

    5. Run the query against the Prometheus of the kit:

      curl -sk "https://<host>/prometheus/api/v1/query" --data-urlencode "query=gluesync_system_cpu_load"
curl -sk "https://<host>/prometheus/api/v1/query" --data-urlencode "query=jvm_gc_overhead"
    Expected result

    gluesync_system_cpu_load is a number between 0 and 1. A high jvm_gc_overhead means that the CPU goes to garbage collection. Go to Check the memory and garbage collection. A saw-tooth JVM heap in the Eden space with a stable Old Gen is normal. An Old Gen that rises after each collection is not.

    The Core Hub metrics endpoints answer without authentication, so that Prometheus can scrape them. See Prometheus integration. CPU load and RAM are declared as counters in the metrics, but they are sampled values. Query them directly and do not use rate() on them.
    Lower the CPU that Gluesync causes on an IBM i

    UI

    On an IBM i source, two settings decide how many journal calls Gluesync sends to the system.

    Standard mode. One shared reader per journal reads the journal and writes the changes to the Gluesync cache. It sleeps for the agent setting Journal reader polling interval in milliseconds only when the journal has no new entries. Entities read from the cache, so the entity Polling interval (ms) does not load the IBM i.

    Dedicated journal reader. An entity with Use dedicated journal reader enabled skips the cache. It runs its own reader that calls the journal API at the entity polling interval. Every dedicated entity adds its own calls to the IBM i.

    In a reported case, 50 entities with a dedicated journal reader had their polling interval lowered from 10,000 ms to 1,000 ms. The CPU of the IBM i went from 60% to 99% and stayed there. The CPU went back to normal within seconds after the interval was set to 10,000 ms again. The change had been applied to every entity of the group at once.

    To lower the load:

    1. Open the pipeline and select the gear icon above the entity table;

    2. For each entity, open Fine tuning and check Use dedicated journal reader. Switch it off on every entity that does not need it, so that these entities share one reader per journal. Keep it only for entities with a very high change volume or that need near-real-time latency;

    3. For each entity that stays dedicated, set Polling interval (ms) to the latency you need, not 1000 for all of them;

    4. If you changed a group of entities together, open a few of them and check the value, because the change applies to all of them;

    5. If the load stays high, raise Journal reader polling interval in milliseconds on the source agent (Edit source). A higher value means fewer calls when the journal is quiet and a later capture of changes;

    6. Select Save or Test and continue.

    Expected result

    The CPU of the IBM i goes down within seconds to minutes after the change, as in the reported case. The changes reach the target within the new polling interval.

    For the full explanation, read IBM i journal data capture architecture and Source advanced settings.

    Limit the connections to a source or target

    UI

    A connection pool that is larger than the system can serve loads that system. In a reported case, an ERP source slowed down because Gluesync held many connections to it. The recommendation was to lower the maximum number of connections of the pool in the source agent settings. In the same case, Gluesync also queried a system view that touched every journal receiver of the IBM i before it filtered. A later release replaced that view with a call that reads only the journals of the registered entities. Run a current release of Gluesync.

    1. Open the connection of the source (or target) with Edit source (or Edit target);

    2. Lower Max connections count. The default is 25 and the minimum for Min connections count is 3;

    3. Select Test and continue;

    4. Check the number of connections from the database side.

    Expected result

    The database shows fewer sessions from Gluesync. The pipeline stays Active.

    Fewer connections can slow down a snapshot with many writers. Keep Max connections count at least as high as Snapshot writing concurrency. See Tune snapshot and CDC throughput safely.
    Lower the CPU that Core Hub uses

    UI

    Check these causes in this order.

    1. Polling readers and many entities. On agents that poll the source, the number of queries per second is the number of entities divided by the polling interval. Raise Polling interval (ms) on the entities that do not need a low latency, and stop the entities you do not use. On PostgreSQL streaming, the interval did not change the CPU in a test (about 1% CPU at any value);

    2. Snapshots in parallel. Each snapshot adds readers and writers. With Core Hub at the limit of its CPU, more writers do not help. Run fewer snapshots at the same time, or lower Snapshot writing concurrency (default 2, allowed 1 to 10) and Parallel reads (default 1). A snapshot with bulk load raises the CPU of Core Hub and lowers the CPU of the target;

    3. Log level. DEBUG is the default level in the kit and it writes more log lines than INFO: about 14 KB per minute against about 6.6 KB per minute at idle. Throughput did not change between DEBUG and INFO in a test. Set Settings → Support → Log level settings → INFO when you do not need the detail. Use DEBUG for the time you need it, for example when MOLO17 Support asks for a snapshot run at DEBUG;

    4. Heap and garbage collection. See Check the memory and garbage collection;

    5. Host size. See Size the host for the number of pipelines.

    Expected result

    docker stats --no-stream and the Host CPU panel show a lower value after the same workload.

    Check the memory and garbage collection

    CLI Config

    A heap that is too small makes the JVM collect garbage all the time, and the CPU goes to garbage collection instead of replication.

    1. Read JVM heap and GC pauses and live threads in Grafana, and jvm_gc_overhead (Read the CPU and JVM metrics of Core Hub);

    2. If the heap is close to its limit, lower the memory demand first: a smaller Max snapshot page size (KB) (default 8192) and fewer entities in snapshot at the same time;

    3. Check the memory of the Core Hub container. The kit sets the heap to MAX_RAM_PERCENTAGE (default 75.0) of the memory the container may use. Without a limit on the core-hub service, that is 75% of the memory of the host. Give Core Hub at least 2 GB;

    4. To change the value, edit the core-hub service in docker-compose.yml and recreate the container:

      docker compose up -d gluesync-core-hub
    Expected result

    jvm_gc_overhead drops, the JVM heap panel stays well below its limit, and no Out Of Memory Error toast appears. Read Tune snapshot and CDC throughput safely for the measured effect of the container memory.

    Size the host for the number of pipelines

    Config

    If the CPU is high on every container and every setting is reasonable, the host may be too small for the number of pipelines.

    • The reference host for Docker Compose or Podman is 8 vCPU and 16 GB of memory. The real need depends on the data volume, the number of concurrent connections, the transformations and the monitoring;

    • One single Linux host with 24 vCPUs ran about 90 pipelines for a customer with a similar use case. Use it as an order of magnitude, then refine it on your own workload;

    • For larger or highly available setups, MOLO17 recommends Kubernetes with three nodes;

    • Put the source cache on a local SSD. A slow volume shows up as CDC latency.

    See Install Gluesync on Linux and size the host and System requirements.

    Expected result

    After you add CPU, docker stats --no-stream shows Core Hub below the new limit with the same workload.

    Verify that the CPU is back to normal

    CLI UI

    1. Run docker stats --no-stream three times, one minute apart, and compare with your first reading;

    2. Read Host CPU and JVM heap in Grafana over the same period;

    3. On an IBM i, read the CPU from the IBM i tools for at least one polling cycle;

    4. In the Control Plane, check that the entities are Active and that the delay from source to target did not grow.

    Verify the result

    • The container or system that was high is lower than your first reading, with a comparable workload;

    • jvm_gc_overhead is low and the JVM heap panel is stable;

    • Entities stay Active and the changes reach the target within the polling interval;

    • You wrote down every setting you changed and its previous value.

    Troubleshooting

    Problem Possible cause What to do

    The CPU of the IBM i rose to 99% after you lowered an entity polling interval

    The entities use a dedicated journal reader, and each one calls the journal API at the entity interval

    Switch the dedicated reader off where you do not need it, or set the interval back, for example to 10000, and set a value that matches the latency you need (Lower the CPU that Gluesync causes on an IBM i)

    The CPU rose after you changed a setting on one entity

    The change was saved on a whole group of entities

    Open several entities of the group and check the value

    An IBM i is busy when the journals are quiet

    The agent polling interval is short

    Raise Journal reader polling interval in milliseconds on the source agent

    The ERP or source shows too many open connections

    The pool Max connections count is high

    Lower Max connections count on the source connection (Limit the connections to a source or target) and run a current release

    Core Hub is high and jvm_gc_overhead is high

    The heap is too small for the snapshots that run

    Lower Max snapshot page size (KB), run fewer snapshots at once, and give Core Hub more memory (Check the memory and garbage collection)

    Core Hub is high with many entities and nothing else runs

    Many polling entities

    Raise Polling interval (ms) on the entities that do not need a low latency, and stop the entities you do not use

    Core Hub, Chronos and Conductor are all low but the snapshot is slow

    The limit is a setting, the network or the source, not the CPU

    Follow Tune snapshot and CDC throughput safely

    The snapshot stops with Communication link failure, Connection reset after many rows

    The network path closes idle connections while the target writes

    Check the idle timeout of the firewall, load balancer or VPC endpoint between Gluesync and the source, and run the snapshot again

    Related guides

    Contact MOLO17 Support if you need assistance with CPU usage.