Search docs.molo17.com
    Operational guide Control Plane

    Monitor pipelines in Grafana with the Core Hub metrics dashboard

    The Gluesync installation kit ships with Grafana and Prometheus already connected to Core Hub. This guide opens the Core Hub metrics dashboard from the Control Plane and directly in the browser, filters it to one pipeline, generates real traffic so the panels fill with data, and shows how to read the panels that matter day to day.

    In this guide

    You will learn how to open Metrics & Insights, sign in to Grafana, find the Core Hub metrics dashboard, use the Pipeline, Entity and Source agent filters and the time range, read the throughput, latency, JDBC connection and JVM panels, and recognise a panel that is empty because Core Hub does not publish that series.

    Quick information

    Goal

    Watch a running pipeline in Grafana and read its main health panels

    Audience

    Gluesync administrators and operators

    Level

    Basic

    Time required

    About 30 minutes

    Area

    Control Plane → Metrics & Insights → Core Hub metrics

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You installed Gluesync with the default installation kit, which includes Grafana and Prometheus. If you run your own Grafana, see Grafana dashboards;

    2. A pipeline is running in CDC. This guide uses the Shop orders pipeline with the entities customers and orders, started as described in Start a snapshot and CDC replication from the Control Plane;

    3. You can sign in to Grafana with the kit default admin / password;

    4. The pipeline has changes to replicate. If it has none, a test environment where you can run SQL on the source lets you generate traffic in an optional step.

    Grafana has its own login, separate from the Control Plane login. If your browser warns about the certificate when you open the Control Plane or Grafana, the installation uses a self-signed certificate: accept it for this host.

    Guide steps

    Open Metrics & Insights from the Control Plane

    UI

    1. Sign in to the Gluesync Control Plane;

    2. In the left menu, select Metrics & Insights (the gauge icon);

    3. If Grafana shows its login form inside the Control Plane, enter admin in Email or username and password in Password, then select Log in.

    Control Plane with the Metrics & Insights menu item marked 1 and the Grafana login form with the username field
    Figure 1. Select Metrics & Insights (1), then sign in to Grafana with the username (2), the password (3) and Log in (4).
    Expected result

    Grafana opens inside the Control Plane directly on the Core Hub metrics dashboard. The Control Plane left menu stays on the left. Select the menu icon at the top left of the dashboard to open Grafana’s own navigation.

    Core Hub metrics dashboard with its filters and the Overview panels
    Figure 2. The Core Hub metrics dashboard in the Control Plane. Metrics & Insights stays highlighted (1).
    Open Grafana directly and find the dashboard

    UI

    1. In a new browser tab, open https://localhost/grafana. Use the address of your Gluesync host instead of localhost;

    2. If Grafana asks you to sign in, enter admin and password, then select Log in. A session you opened through the Control Plane is reused: the same browser is already signed in;

    3. Open the Grafana menu and select Dashboards;

    4. Select Core Hub metrics.

    Grafana login page at the grafana address of the host with the username field
    Figure 3. The Grafana login page (1 username, 2 password, 3 Log in).
    Grafana Dashboards page listing the Core Hub metrics dashboard marked 1 with the tags cdc
    Figure 4. The only dashboard in the kit is Core Hub metrics (1).
    Expected result

    The Core Hub metrics dashboard opens in Grafana full screen. It shows the same panels as the Metrics & Insights page of the Control Plane.

    Optional: generate test traffic on the source

    SQL

    Skip this step if your pipeline already replicates live changes: the dashboard shows that traffic. A pipeline with no changes draws flat lines, so in a test environment you can create some traffic yourself. Run this loop for a few minutes on a machine that can reach the source database. It inserts 3000 orders and updates 2000 of them every few seconds, which Core Hub delivers to the target through CDC. Replace the connection parameters with yours, and set PGPASSWORD so that psql does not prompt on every pass.

    for i in $(seq 0 59); do
  NEXT=$((100000 + i * 3000))
  psql -q -h <source-host> -U <user> -d shop <<SQL
INSERT INTO shop.orders (id, customer_id, amount, currency, status, order_date)
  SELECT g, 1 + (g % 200), round((random() * 500)::numeric, 2),
         (ARRAY['EUR','USD','GBP'])[1 + (g % 3)], 'new', current_date
  FROM generate_series($NEXT, $NEXT + 2999) g;
UPDATE shop.orders
  SET status = (ARRAY['paid','shipped','delivered','cancelled'])[1 + (id % 4)],
      amount = amount + 1
  WHERE id BETWEEN $NEXT AND $NEXT + 1999;
SQL
  sleep 3
done

    While it runs, check on the target that the rows arrive:

    -- run on the target
SELECT count(*) FROM shop.orders;
    Expected result

    The count keeps growing, one batch of 3000 rows at a time, and it equals the count on the source a few seconds after each batch. Leave the loop running while you read the dashboard. Do not run it against a production database.

    Filter the dashboard and set the time range

    UI

    1. In Pipeline, select Shop orders. Every panel now shows only that pipeline. The list contains the pipelines that delivered at least one row;

    2. In Entity, leave All to compare the entities, or select one entity such as orders to see only its table;

    3. Leave Source agent, Checkpoint field and Connection purpose on All for now;

    4. Select the time picker (Last 15 minutes) to change the window, and the 5s arrow next to Refresh to change how often the panels reload.

    Core Hub metrics dashboard with the Pipeline filter set to Shop orders and the Pipeline
    Figure 5. Pipeline (1), Entity (2), Source agent (3), the time range (4) and the refresh control (5).
    Open time range picker with the absolute time range fields and the quick ranges Last 5 minutes
    Figure 6. The picker offers quick ranges from the last 5 minutes to the last 24 hours, and a custom From and To.
    Expected result

    The dashboard defaults to the last 15 minutes with a 5-second refresh. After you select Shop orders only that pipeline remains in the panels. With Entity set to orders, Entities delivering (2m) drops from 2 to 1 and Rows delivered /s shows only the orders rate.

    Source agent lists the source connectors that publish a checkpoint. For the PostgreSQL pipeline in this guide it offers only All, because no checkpoint series exists (see Checkpoint age and entity lag).
    Read the Overview panels

    UI

    The Overview section answers one question: is data moving and is anything wrong right now? With the traffic loop running, read these panels. The info icon on each panel title shows what it measures.

    Overview section with Rows delivered per second at 1.8K
    Figure 7. Rows delivered (1), entities delivering (2), DB connections (3), host CPU (4), and the reader, source and cache lag stats (5, 6) that stay empty for this connector.
    Panel What it shows What it showed in the example

    Rows delivered /s

    Rows delivered to the targets per second, summed over the selected pipeline and entities

    About 1.8K rows/s while the loop ran, and a flat line before it started

    Bytes delivered /s

    Payload delivered to the targets per second

    About 40 kB/s

    Entities delivering (2m)

    Entities that delivered at least one row in the last two minutes

    2 (customers and orders)

    Entities snapshotting (2m)

    Entities whose initial snapshot is still advancing. 0 means every entity is in CDC

    0

    Active DB connections

    JDBC connections currently borrowed from Core Hub’s pools, for every purpose (pipeline tasks, Query Studio, Query Forge)

    0 or 1

    Host CPU

    System-wide CPU load of the Core Hub host

    2% to 7%
    Expected result

    Rows delivered /s rises when the loop starts and stays high while it runs. Entities delivering (2m) equals the number of entities that receive changes.

    Read throughput and volume

    UI

    1. Scroll to Throughput & volume;

    2. In Rows/min by entity, find which entity carries the volume. The legend lists each entity with its latest value;

    3. In Throughput by entity, compare the payload rate. An entity with moderate rows but a high byte rate has wide rows;

    4. In Cumulative rows replicated, read the total of rows delivered in the window.

    Throughput and volume section with Rows/min by entity at about 111 thousand for orders marked 1
    Figure 8. Rows per minute (1) and payload rate (2) per entity, and the running total of replicated rows (3).
    Expected result

    orders shows about 111 K rows/min, which matches the 1.8K rows/s of the overview, while customers stays near zero. Cumulative rows replicated climbs steadily.

    Snapshot progress stays empty here because no entity is copying. It fills during an initial snapshot (see Start a snapshot and CDC replication from the Control Plane).
    Read latency

    UI

    1. Scroll to Internal latency;

    2. Read Source read time (SMA) and Target write time (SMA): the moving average of the time spent reading from the source and writing to the target. A high write time with a low read time points at the target;

    3. Read Internal end-to-end (SMA): the time Core Hub spends on a change after it is read. It is not the source-to-target freshness;

    4. Read Latency breakdown for the selected entities, which draws read, write, storage and end-to-end side by side. Narrow Entity to one or two tables to read it clearly;

    5. Read Last end-to-end time per entity: the latest measurement for each entity.

    Internal latency section with three moving average panels marked 1
    Figure 9. Read, write and end-to-end time (1), the breakdown (2) and the latest value per entity (3).
    Expected result

    Under load the moving averages move between a few milliseconds and about 100 ms, and the last end-to-end time was 14 ms for orders and 3 ms for customers. A line that stays flat at a low value after the loop stops is normal.

    Read JDBC connections and the JVM

    UI

    1. Scroll to Connections & pools;

    2. In Active connections by purpose, read how many JDBC connections are borrowed for pipeline tasks, Query Studio, Query Forge endpoints and monitoring. A rising query_studio or query_forge_endpoint band means ad-hoc queries compete with replication for your database;

    3. In Active connections by agent, read the same connections per connector. One pipeline task connection appears for the PostgreSQL pipeline;

    4. Scroll to Host & JVM, then read JVM heap: a saw-tooth in the Eden space with a stable Old Gen is normal, while an Old Gen that keeps climbing after each collection is a leak signature;

    5. Read GC pauses and live threads: the average pause per collection and the live thread count.

    Connections and pools section with Active connections by purpose marked 1 and Active connections by agent marked 2 showing one PostgreSQL pipeline task connection
    Figure 10. Connections by purpose (1) and by agent (2).
    Host and JVM section with the JVM heap panel marked 1 and the GC pauses and live threads panel marked 2
    Figure 11. JVM heap (1) and GC pauses with live threads (2).
    Expected result

    The connection panels stay at 0 or 1 for the pipeline while the loop runs, the heap draws a saw-tooth with an Old Gen near 100 MiB, and the live threads stay around 100.

    Pool utilisation and Pool queue and acquisition time are empty in this setup because no HikariCP pool metrics are published. They fill only when the matching pools exist.
    Check checkpoint age and entity lag

    UI + SQL

    Two sections measure how far behind the source a pipeline is: Checkpoints & lag (checkpoint age, shown by Reader freshness lag) and Entity lag — source backlog and cache backlog (Time behind the source, Time waiting in the local CDC cache). Their stats in the Overview are Worst reader lag, Worst source lag and Worst cache lag.

    1. Scroll to Checkpoints & lag;

    2. Read Reader freshness lag: the seconds between now and the CDC position persisted by the source reader. Rising values mean the reader is falling behind;

    3. Scroll to the entity lag section and read Time behind the source and Time waiting in the local CDC cache. High cache lag with low source lag means the target write path is the bottleneck.

    In the example, every one of these panels shows a dash: for the PostgreSQL pipeline Core Hub published no checkpoint and no lag series. To confirm that a series is not exported, and not merely quiet, ask Prometheus:

    1. Open https://localhost/prometheus;

    2. In Query, enter gluesync_read_checkpoint_age_seconds and run the query.

    Checkpoints and lag section with Reader freshness lag
    Figure 12. Panels of the Checkpoints & lag section when the connector publishes no checkpoint series.
    Expected result

    Prometheus returns no data for gluesync_read_checkpoint_age_seconds in this lab, which matches the empty panels. When a connector does publish the series, the panels fill and Source agent lists the reader names. See Metric availability for when each series appears.

    Check that Prometheus scrapes Core Hub

    UI

    Do this when many panels are empty at once, or when nothing moves.

    1. Open https://localhost/prometheus;

    2. Select Status, then Target health (or open https://localhost/prometheus/targets);

    3. Find the gluesync-core-hub job and read its State and Last scrape.

    Prometheus Target health page listing the gluesync-core-hub target and the prometheus target
    Figure 13. Both scrape targets are UP and were scraped a few seconds ago.
    Expected result

    The gluesync-core-hub target is UP and its last scrape is a few seconds old. A target in DOWN with an error message means Prometheus cannot reach Core Hub: fix that first.

    Stop the traffic and clean up

    SQL

    1. Stop the loop (press Ctrl+C) if it is still running;

    2. Delete the test rows on the source. CDC deletes them on the target:

      -- run on the source
DELETE FROM shop.orders WHERE id >= 100000;

    3. Wait until the counts are equal again:

      -- run on the source, then on the target
SELECT count(*) FROM shop.orders;
    Expected result

    Both databases hold 1000 orders again, and the dashboard shows the delete burst followed by a flat line.

    Verify the result

    You have successfully used the dashboard if:

    • Metrics & Insights and https://localhost/grafana both open the Core Hub metrics dashboard;

    • The Pipeline filter limits every panel to Shop orders and Entity limits it to one table;

    • Rows delivered /s and Rows/min by entity rose while the traffic loop ran;

    • The latency, connection and JVM panels show values;

    • You can tell which empty panels are caused by series Core Hub does not publish, and you checked the Prometheus targets.

    Troubleshooting

    Problem Possible cause What to do

    Grafana asks for a login again

    Grafana has its own session, separate from the Control Plane

    Sign in with admin and password, the kit default, or with the credentials your administrator changed them to

    Pipeline does not list your pipeline

    The pipeline has not delivered a row yet in the window

    Start its entities (see Start a snapshot and CDC replication from the Control Plane) and wait for the next refresh

    Rows delivered /s stays at zero

    No changes reach the source, the entity is on hold, or the time range does not include the activity

    Check that the entity is Active, run the traffic loop, and select a range such as Last 15 minutes

    Several panels show a dash

    The series is not published for your connector, or Prometheus is not scraping Core Hub

    Query the metric name in Prometheus, then check Status → Target health

    Source agent offers only All

    No checkpoint series exists, and the list is built from it

    Expected for connectors without a persisted checkpoint; filter by Pipeline and Entity instead

    A deleted entity still appears in the panels, sometimes under its identifier

    Prometheus keeps the old series for the selected window

    Wait until the time range moves past the deletion, or shorten the range

    The gluesync-core-hub target is DOWN

    Prometheus cannot reach Core Hub on the network, or the certificate is rejected

    Read the error shown for the target in Target health, and check that Core Hub is running

    Related guides

    Contact MOLO17 Support if you need assistance with Grafana or Core Hub metrics.