Grafana dashboards

The official Core Hub metrics dashboard turns the Core Hub /metrics scrape into an operational view of replication delivery, CDC source freshness, internal latency, JDBC connections, and host/JVM health.

Configure Prometheus as described in Prometheus integration before importing the dashboard.

Import the official dashboard

  1. Open Core Hub metrics and copy dashboard ID 22907.

  2. In Grafana, select Dashboards  New  Import.

  3. Enter 22907, then select Load.

  4. Select the Prometheus data source that scrapes Core Hub.

  5. Select Import.

The dashboard defaults to the last 15 minutes with a 5-second refresh. A short window is useful during active diagnosis; use a longer range for capacity and trend analysis.

Empty checkpoint, stall, pool, or latency-threshold panels are expected when those metric families are not in the scrape yet. See Metric availability for which series are always present in 2.2.11 and which appear only after a CDC reader, stall monitor, or connection pool registers.

The Entity lag section and its four Overview stats were added in a later dashboard revision. If your copy does not have them, re-import the dashboard. They also require the per-entity lag probe, which is enabled by default and samples every 15 seconds; GS_ENTITY_LAG_PROBE_INTERVAL_SECONDS changes the interval and 0 disables the probe, in which case those panels stay empty.

Understand the filters

Every selector filters by name. Core Hub labels each family with the pipeline, entity, and agent names beside their internal ids, so one pipeline picker drives the whole dashboard.

Filter Metric source Use

Pipeline

label_values({name=~"gluesync_total_count|gluesync_read_checkpoint_age_seconds"}, pipelineName)

The Control Plane pipeline nickname. Filters every section, checkpoints included. The query spans both families so that a pipeline with a checkpoint but no delivered row yet still appears.

Entity

label_values(gluesync_total_count{pipelineName=~"$pipelineName"}, entityName)

The Control Plane table or entity name. Leave All for a top-N overview or select one or two entities for detailed latency analysis.

Source agent

label_values(gluesync_read_checkpoint_age_seconds{pipelineName=~"$pipelineName"}, sourceAgentName)

The source CDC reader, by agent name. An agent that is not running has no name and appears under its id, which is what the metric falls back to.

Checkpoint field

label_values(gluesync_read_checkpoint_position, field)

A connector-specific numeric position component, such as scnNumber, binlogPosition, sequenceNumber, or id.

Connection purpose

label_values(gluesync_db_connections_active_total, purpose)

Filters connection panels by pipeline_task, query_studio, query_forge_endpoint, or unspecified.

The name labels the pickers read arrive with Core Hub 2.2.11.5. On an older Core Hub the checkpoint families carry ids only: Pipeline and Source agent set to All still show every series, because their All value matches an absent label as well, but selecting a single value filters nothing out of the checkpoint section. Upgrade Core Hub before relying on those two pickers.

Adding the name labels changes the identity of the affected series. After the upgrade the pre-upgrade series stay queryable for the Prometheus lookback window and then disappear, so a panel or an alert that joins on labels can behave differently across the upgrade window.

Dashboard KPI guide

Overview

The first row answers whether data is moving and whether Core Hub has an immediate resource or freshness problem.

Panel Metric or calculation Interpretation

Rows delivered /s

sum(rate(gluesync_total_count[$__rate_interval]))

Current delivered row throughput across the selected pipeline names and entities.

Bytes delivered /s

sum(rate(gluesync_total_size_bytes[$__rate_interval]))

Current delivered payload throughput. Compare this with row rate to identify changes in average row size.

Entities delivering (2m)

Count of entities whose gluesync_total_count rate is greater than zero over two minutes

How many selected entities are actively delivering rows.

Entities snapshotting (2m)

Count of entities whose gluesync_total_snapshot_count rate is greater than zero over two minutes

A non-zero value means at least one initial snapshot is still advancing.

Worst reader lag

Maximum gluesync_read_checkpoint_age_seconds

Source CDC freshness for the slowest selected reader. Dashboard colors change at 5 and 15 minutes.

Longest stall (no data)

Maximum non-negative gluesync_entity_dry_seconds

Longest time an entity has gone without a positive-row batch. Dashboard colors change at 10 and 30 minutes.

Active DB connections

Sum of gluesync_db_connections_active_total

All currently borrowed JDBC connections for the selected purposes.

Host CPU

Maximum gluesync_system_cpu_load

System-wide host CPU ratio. Dashboard colors change at 70% and 90%.

Worst source lag

Maximum gluesync_source_lag_milliseconds over entities whose probe is answering

Longest any entity’s oldest unread change has been waiting at the source. Dashboard colors change at 5 and 15 minutes.

Worst cache lag

Maximum gluesync_cache_lag_milliseconds over entities whose probe is answering

Longest any change already read has been waiting in the local CDC cache for the target. Dashboard colors change at 5 and 15 minutes.

Lag probes failing

Sum of gluesync_entity_lag_probe_info for results error, timeout, busy, and stale

Entities whose lag probe is not answering. stale is the one to act on, because it usually means the pipeline itself is stuck.

Lag not measurable

Sum of gluesync_entity_lag_probe_info for results unsupported and no_read_position

Entities publishing no lag figures for a reason that is not a fault.

These dashboard colors are visual defaults, not universal service-level objectives. Tune them to workload volume, expected source idleness, and your operational targets.

Checkpoints and reader lag

Checkpoint panels expose the persisted source-reader position:

  • Reader freshness lag charts gluesync_read_checkpoint_age_seconds. Rising lag means the source reader is falling behind.

  • Checkpoint advance rate uses a five-minute derivative, expressed in connector position units per minute.

  • Checkpoint age by source lists the oldest persisted positions first and includes checkpoint kind and key.

  • Checkpoint positions lists numeric position fields.

  • Checkpoint details displays non-numeric labels such as an Oracle rsId, binlog file name, or AS/400 journal details.

  • Checkpoint position over time shows whether each numeric position is advancing.

  • Read position wall-clock renders gluesync_read_checkpoint_position_timestamp_seconds as local time.

Checkpoint age by source and Checkpoint positions also carry an Entity column, filled in for the connectors whose read position Core Hub keeps per entity and empty for a connector’s own key, which covers a journal or a binlog several entities share. Checkpoint details has no such column: its metric only ever covers a connector’s own key. Ids stay in all three tables as trailing columns.

Connector position units differ. An Oracle SCN can move by millions while an AS/400 sequence moves by thousands, so compare a series only with itself over time. Steps can be normal for batch-oriented readers.

# Source freshness
gluesync_read_checkpoint_age_seconds{
  pipelineName=~"$pipelineName",
  sourceAgentName=~"$sourceAgentName"
}

# Position advance per minute
clamp_min(
  deriv(gluesync_read_checkpoint_position{
    pipelineName=~"$pipelineName",
    sourceAgentName=~"$sourceAgentName",
    field=~"$checkpointField"
  }[5m]) * 60,
  0
)

A flat or missing advance-rate series, together with continuously rising reader lag, is a strong stalled-reader signal.

Entity lag: source backlog and cache backlog

The Entity lag row measures how far behind each entity is at the two hops where a change can pile up:

  • gluesync_source_* covers source database to Gluesync: what the source has generated and Core Hub has not read yet.

  • gluesync_cache_* covers local CDC cache to target: what Core Hub has read and not yet written.

The two are reported separately because they have different causes and different fixes. A reader two minutes behind the source while the cache hop shows twenty hours means the target is the bottleneck, not the source, and a single combined number would hide which.

Lag at each hop is the wait of the oldest change that has not moved past that hop yet, computed at scrape time as now minus that change’s timestamp. Worked example at the source hop: the last change is at 16:30, Core Hub has read up to 16:15, the first unread change is at 16:20, and it is now 17:00, so the lag is 40 minutes, not 15. Unlike checkpoint age it keeps growing while a backlog drains and does not freeze when the source goes quiet.

Entity lag is not gluesync_read_checkpoint_age_seconds. Checkpoint age is how long the read position has stood still; entity lag is how far behind the source or the target the entity actually is. The two families are deliberately kept apart and answer different questions. See Per-entity CDC lag for the full family, label, and flag list.

A figure a connector cannot produce is an absent series, never a zero. Zero is a legitimate backlog value that means caught up, so it cannot also stand for unknown. gluesync_entity_lag_probe_info carries the reason instead, which is what separates "this connector has no probe" from "the probe is failing".

Panels in the row

  • Time behind the source charts gluesync_source_lag_milliseconds. Dashboard colors change at 5 and 15 minutes. A missing line is a connector that cannot date its backlog, not a zero.

  • Time waiting in the local CDC cache charts gluesync_cache_lag_milliseconds. This is the hop Gluesync owns: rising here while source lag stays flat points at the target write path rather than at the reader.

  • Unread change rows at the source charts gluesync_source_pending_rows.

  • Rows waiting in the local CDC cache charts gluesync_cache_pending_rows. Only cache-based connectors have this hop; one that reads the source and emits directly publishes nothing here.

  • Backlog in bytes plots two different quantities on purpose. gluesync_cache_pending_bytes is this entity’s own arena file. gluesync_source_log_distance_bytes is a distance along a log the whole database shares, such as a WAL or a binlog, so most of those bytes are changes this entity will never receive, and a distance of zero is a log position rather than proof that nothing is waiting.

  • Worst offenders right now lists one row per entity with both hops side by side, sorted by source lag, plus the two qualifying flags. An empty cell is a figure that entity’s connector cannot produce; it is painted neutral rather than the dashboard’s usual green because a green dash reads as caught up when it means not measured.

  • Why an entity has no figures lists the current probe outcome per entity, which is the panel to open when a lag panel is blank.

  • Cost of measuring, and freshness charts gluesync_entity_lag_probe_duration_milliseconds together with gluesync_entity_lag_sample_age_seconds, the second multiplied by 1000 so both share a millisecond axis. A probe borrows its connection under the monitoring purpose, which the pool refuses rather than queues, and carries a server-side query timeout, so a rising duration is a source getting slower and not the pipeline being starved.

Read the count flags before trusting a small number

gluesync_source_pending_rows is an exact count where the connector can produce one, but two conditions qualify it, and Worst offenders right now carries both next to the count.

Column Metric Meaning

Row count

gluesync_source_pending_rows_is_lower_bound

undercounts means a hop upstream of the counted one is invisible, so the real backlog is larger. exact means nothing upstream is hidden.

Zero = caught up

gluesync_source_empty_backlog_is_unproven

unproven means a count of zero does not prove an empty backlog: on MSSQL CDC and DB2 LUW CDC the store being counted is filled by something whose position nothing here can read, so a stopped capture job leaves it empty and the count reads zero. In that state the lag itself is withheld rather than published as zero.

gluesync_source_pending_rows_is_upper_bound and the matching gluesync_cache_* flags are exposed as well but are not on the dashboard. The upper-bound flag is set where the count covers more than this entity alone, such as an AS/400 journal whose sequence delta counts every object on the journal.

Why an entity has no figures

gluesync_entity_lag_probe_info exposes one series per possible outcome with 1 on the current one, so a recovered entity does not keep showing as failing for the Prometheus lookback window.

Outcome Panel label Meaning and action

ok

not listed

The probe answered. Figures are current.

stale

stale

No probe has completed for three of that entity’s sampling intervals, and the backlog figures are withheld. A probe can only start on a CDC poll tick, so this usually means the pipeline itself is stuck: a target too slow to drain the emission channel, a source that will not connect, or a read wedged in a driver. This is the one to act on.

timeout

timeout

The probe did not answer within GS_ENTITY_LAG_PROBE_TIMEOUT_SECONDS and was abandoned.

error

error

The probe threw. Check the Core Hub log for the source error.

busy

busy

Every connection was serving the pipeline, so the measurement was skipped. Not a failure, and the figures shown are the last ones measured.

abandoned

probe pool saturated

The probe waited longer than one interval for a thread and was dropped before it ran. This is the probe pool, not the pipeline: raise GS_ENTITY_LAG_PROBE_CONCURRENCY or lengthen the interval. The figures shown are the last ones measured.

no_read_position

no read position

Nothing has been read for this entity yet, so there is no position to measure a backlog from. Expected on a freshly configured entity, where it clears itself on the first read, and a fault if it persists.

unsupported

no probe here

That connector has no probe for this figure. Not a fault.

What a connector can measure differs widely. A trigger-based staging table gives an exact count for the price of one indexed COUNT(*), while a redo, binlog, or journal reader can only be counted by reading it, so several connectors report a time or a byte distance and no row count at all.

Staleness and the probe gate

Figures older than three of that entity’s sampling intervals are not published at all, the cadence being the slower of the probe interval and the entity’s own polling interval. The probe keeps its last answer when it fails, so a sample can outlive the state it describes, and because the lag is computed against now a stale oldest-pending change would go on growing on an entity that may have caught up. Past the bound the backlog series disappear, the outcome becomes stale, and gluesync_entity_lag_sample_age_seconds is what says why.

Every lag panel gates its series on the probe outcome:

gluesync_source_lag_milliseconds{
  pipelineName=~"$pipelineName",
  entityName=~"$entityName"
}
and on (pipelineId, entityId)
(gluesync_entity_lag_probe_info{result=~"ok|abandoned"} == 1)

The gate is needed because these series carry an explicit timestamp, and Prometheus does not mark an explicitly timestamped series stale when Core Hub stops exposing it. Without the gate, a series that stopped being published, for example after an entity was stopped, would sit on the panel as an unmarked, minutes-old number for the whole lookback window. abandoned is included deliberately: it keeps the last measurement on purpose.

Delivery and stalled entities

gluesync_entity_dry_seconds measures seconds since an entity last received a transaction message containing rows. A value of -1 means the entity has not been active since Core Hub started; dashboard stall panels exclude that value.

  • A repeating sawtooth is normal: the timer rises while idle and resets when data arrives.

  • A continuous climb means no rows are arriving.

  • A high value can still be expected for naturally quiet source tables. Interpret it against the configured stall threshold and monitored-object allowlist.

The Last delivered transaction per entity table reads lastTransactionId from gluesync_last_transaction_info. It keeps only the newest series for each entity/target pair because transaction ID is a label and older label values remain queryable during the Prometheus lookback window. Compare this ID with the source when investigating apparently missing rows or a restart position.

Throughput and volume

The dashboard separates rates from cumulative totals:

  • Snapshot progress shows cumulative snapshot rows only for entities whose snapshot counter is currently advancing.

  • Rows/min by entity ranks the 15 busiest entities by the two-minute rate of gluesync_total_count.

  • Throughput by entity ranks payload rate from gluesync_total_size_bytes.

  • Cumulative rows replicated shows the raw total; a reset to zero indicates a Core Hub process restart.

  • Moving average rows per transaction uses gluesync_sma_count.

  • Moving average transaction size uses gluesync_sma_size_bytes.

Apply rate() to gluesync_total_count and gluesync_total_size_bytes, which are cumulative counters. Do not apply rate() to gluesync_sma_* or gluesync_last_*: Core Hub 2.2.11 declares them as counters in the exposition metadata, but their values are instantaneous snapshots and can decrease.

Internal latency

The latency row displays moving averages for source read, target write, internal storage, and Core Hub internal end-to-end execution:

  • gluesync_sma_read_time

  • gluesync_sma_write_time

  • gluesync_sma_storage_time

  • gluesync_sma_end_to_end_time

  • gluesync_last_end_to_end_time

The read, write, and internal end-to-end panels restrict their top-N view to entities with current throughput. This prevents inactive entities from dominating the display. Select one or two entities to read the combined latency breakdown clearly.

Internal end-to-end time starts after the change has entered Core Hub’s cache. It is not source-to-target freshness and is not the Control Plane CDC lag value. Use gluesync_read_checkpoint_age_seconds for source-reader freshness.

Use the components to localize pressure:

  • Rising read time points toward source access or source-side work.

  • Rising write time points toward the target.

  • Rising storage time points toward Core Hub’s internal persistence path.

  • Rising internal end-to-end time with stable component values can indicate queueing between stages.

Connections and pools

The connection row combines Core Hub’s connection registry with HikariCP metrics:

  • gluesync_db_connections_active_total splits live leases by purpose.

  • gluesync_db_connections_active attributes live leases to agent and purpose.

  • hikaricp_connections_active / hikaricp_connections_max reports pool utilization.

  • hikaricp_connections_pending reports waiting borrowers.

  • The rate of acquire-time sum divided by count reports average acquisition time.

Sustained pool utilization near 1 means the pool is at capacity. Pending borrowers and rising acquisition time confirm pool exhaustion. A rising query_studio or query_forge_endpoint band can show ad-hoc work competing with replication.

Host and JVM

The dashboard correlates host pressure with Core Hub runtime behavior:

  • CPU compares system CPU, hottest processor, and processor average.

  • JVM heap shows jvm_memory_used_bytes{area="heap"} by pool.

  • GC pauses and live threads combines average pause per collection with jvm_threads_live_threads.

  • System memory calculates gluesync_ram_total - gluesync_ram_available.

  • Network throughput applies rate() to the per-interface cumulative receive and send metrics.

  • Core Hub uptime displays gluesync_system_uptime; under five minutes is highlighted as a recent restart.

A sawtooth Eden heap is normal. Old-generation usage that continues rising without falling after garbage collection can indicate retained objects. Correlate it with GC pauses, thread count, and CPU before drawing conclusions.

The 2.2.11 endpoint also exposes Ktor request metrics for custom Core Hub API panels:

# Requests per second by normalized route and status
sum by (route, status) (
  rate(ktor_http_server_requests_seconds_count[$__rate_interval])
)

# Average request duration by normalized route
sum by (route) (
  rate(ktor_http_server_requests_seconds_sum[$__rate_interval])
)
/
sum by (route) (
  rate(ktor_http_server_requests_seconds_count[$__rate_interval])
)

Use the normalized route label instead of individual URLs. Aggregate away address, method, status, and throwable when they are not required to keep panel cardinality manageable.

Smart-alert thresholds

The dashboard treats every gluesync_threshold_* series as configuration, not an observed workload measurement.

  • End-to-end against the latency threshold compares active-entity gluesync_sma_end_to_end_time with gluesync_threshold_latency_minutes * 60000.

  • Smart alert configuration lists every exposed gluesync_threshold_* gauge, including the entity-heartbeat watchdog settings shipped in 2.2.11.

  • Stall time vs smart-alert threshold compares non-negative gluesync_entity_dry_seconds with gluesync_threshold_data_flow_stall_duration_minutes * 60.

The latency overlay is blank when gluesync_threshold_latency_minutes is not configured. Stall overlays are blank until gluesync_entity_dry_seconds is emitted.

Current smart alerts use the monitored-object allowlist, Low/Normal/High sensitivity, cooldown, source watchdog, target-idle stall, and optional latency ceiling. Compatibility gauges for legacy EWMA, latency-pattern, and seasonal tuning may still be exposed, but those legacy detectors do not currently fire alerts.

Recommended diagnostic sequence

When delivery appears delayed:

  1. Select the pipeline in Pipeline, which filters every section.

  2. Check Worst reader lag. If it rises, inspect checkpoint position and advance rate.

  3. Check Worst source lag and Worst cache lag to see which hop the backlog sits at, then open Worst offenders right now for the affected entity.

  4. If a lag panel is blank, check Why an entity has no figures before assuming the entity is fine.

  5. Check Longest stall and the entity stall trend.

  6. Compare rows/s with bytes/s and identify the affected entity.

  7. If reader freshness is healthy, inspect source read, target write, storage, and internal end-to-end time.

  8. Inspect active connection purposes, pool utilization, pending borrowers, and acquisition time.

  9. Correlate the time window with CPU, heap, GC, memory, and network behavior.

  10. Use the latest transaction ID table to verify the last position confirmed for the entity/target pair.

Grafana alert examples

Entity exceeds the configured stall duration

gluesync_entity_dry_seconds >= 0
and
gluesync_entity_dry_seconds
  > on() group_left()
  (gluesync_threshold_data_flow_stall_duration_minutes * 60)

Use a pending period to avoid alerts for short, expected idle intervals. Apply the same monitored-object policy as Core Hub if you do not want Grafana to alert on every entity.

CDC reader freshness

gluesync_read_checkpoint_age_seconds > 900

The 15-minute value matches the dashboard’s default red threshold. Replace it with your own freshness objective.

Entity too far behind the source

gluesync_source_lag_milliseconds > 900000
and on (pipelineId, entityId)
gluesync_entity_lag_probe_info{result=~"ok|abandoned"} == 1

Use gluesync_cache_lag_milliseconds for the same alert on the cache to target hop. Keep the probe gate: without it the alert can fire on a figure that stopped being published.

Lag probe not answering

gluesync_entity_lag_probe_info{result="stale"} == 1

stale means no probe has completed for three of that entity’s sampling intervals. Since a probe can only start on a CDC poll tick, it is usually a stuck pipeline rather than a measurement problem. Alert on result=~"error|timeout" separately if you also want to hear about the probe itself.

Connection pool exhaustion

(hikaricp_connections_active / hikaricp_connections_max) > 0.9
and on(pool)
hikaricp_connections_pending > 0

Core Hub scrape missing

up{job="gluesync"} == 0

Adjust the job label to match your Prometheus scrape configuration.

Build custom panels

For custom dashboards:

  • Use pipelineName and entityName for entity metrics. sourceAgentName and targetAgentName sit beside the agent ids.

  • Use pipelineName, sourceAgentName, and checkpointField for checkpoint metrics. They also carry entityId and entityName, filled in for the checkpoints Core Hub derives per entity and empty for a connector’s own key.

  • Use pipelineName, entityName, and sourceAgentName for the entity lag metrics, with pipelineId, entityId, and sourceAgentId beside them. They have no targetAgentId label: both backlogs belong to the source read and to the shared cache, so an entity fanned out to three targets has one figure per hop, not three.

  • Prefer a name label for display and a join on ids. Names fall back to the id when Core Hub cannot resolve them, so pipelineName can read unknown_pipeline and an agent name can read its own id.

  • Gate every entity lag figure on gluesync_entity_lag_probe_info, and treat an absent series as not measured rather than as zero.

  • Read gluesync_source_pending_rows_is_lower_bound, gluesync_source_pending_rows_is_upper_bound, and gluesync_source_empty_backlog_is_unproven before presenting a pending-row count as exact.

  • Plot gluesync_source_oldest_pending_timestamp_seconds, gluesync_source_read_timestamp_seconds, and their gluesync_cache_* counterparts with a date and time unit when you want wall-clock positions instead of a duration.

  • Use rate() only for genuinely cumulative counters. Plot gluesync_last_*, gluesync_sma_*, host state, and gauges directly, regardless of the legacy exposition type.

  • Use topk() to keep high-cardinality entity views readable.

  • Select the newest timestamped series when a changing value, such as a transaction ID or file name, is stored in a label.

  • Keep source freshness and internal execution latency in separate panels and alerts.

See Metrics reference for metric types, labels, units, and semantics.