Metrics Reference
This page provides a comprehensive reference of all metrics exposed by Gluesync through the /metrics Prometheus scrape endpoint.
Use these metrics to build dashboards, configure alerts, and diagnose performance issues in your Gluesync deployment.
| See Prometheus Integration for setup instructions and Grafana Dashboards for a hands-on guide to building dashboards with these metrics. |
Metric availability and exposition types
The endpoint combines four sources:
-
Gluesync replication, host, threshold, checkpoint, and connection metrics
-
Standard JVM metrics
-
Ktor HTTP server metrics
-
HikariCP pool metrics when pools have registered with the meter registry
A typical 2.2.11 scrape always includes per-entity gluesync_last_*, gluesync_sma_*, and gluesync_total_* series; host CPU/RAM/network counters; JVM and Ktor HTTP families; the gluesync_threshold_* configuration gauges that have a value (including the entity-heartbeat watchdog); and gluesync_db_connections_active_total for every purpose, even when the count is 0.
These families appear only when the matching subsystem is active:
-
gluesync_read_checkpoint_*— a supported Core Hub-managed CDC reader has persisted a position -
gluesync_entity_dry_seconds— the stall/freshness monitor has evaluated the entity -
gluesync_entity_lag_*— the per-entity lag probe has sampled the entity at least once -
gluesync_source_*andgluesync_cache_*— that entity’s connector can produce the figure, and the last sample is recent enough to publish -
gluesync_db_connections_active— at least one connection is borrowed (the_totalseries still emit zeros) -
hikaricp_*— a HikariCP pool has registered with the meter registry -
gluesync_threshold_latency_minutesandgluesync_threshold_row_count_drift_percentage— those optional settings are configured
Absence therefore does not always mean that scraping failed. Check up{job="gluesync"} first, then verify that the corresponding feature is active.
|
In 2.2.11, the Prometheus This page’s Exposition type column reports the wire type exactly as emitted. Interpretation notes explain how to query each family safely. |
Replication performance metrics
These metrics are emitted per pipeline entity and provide granular visibility into data replication performance. All time values are in milliseconds unless otherwise noted.
Replication metrics normally include these labels:
| Label | Description |
|---|---|
|
Unique identifier of the pipeline |
|
Human-readable name of the pipeline |
|
Unique identifier of the replicated entity (table / collection) |
|
Human-readable name of the entity |
|
Agent handling the source read |
|
Display name of the source agent: its nickname, or the connector’s commercial name when no nickname is set |
|
Agent handling the target write |
|
Display name of the target agent |
Label sets vary by metric. Use the Prometheus expression browser or inspect the /metrics output before writing a label join. Every family that identifies a pipeline, an entity, or an agent carries the name beside the id, so filtering by name works across families, checkpoints included.
|
|
The name labels arrive in 2.2.11.5. Two consequences for existing queries and dashboards:
Display by name, join on ids. |
Last-batch metrics
Instantaneous measurements from the most recently processed transaction message.
| Metric Name | Exposition type | Description |
|---|---|---|
|
Counter |
Always |
|
Counter |
Rows in the most recently processed transaction message |
|
Counter |
Source read time for the most recent transaction message (ms) |
|
Counter |
Target write time for the most recent transaction message (ms) |
|
Counter |
Internal storage time for the most recent transaction message (ms) |
|
Counter |
Most recent Core Hub internal execution time (ms), measured after the change enters the cache. This is not source-to-target freshness. |
Simple moving average (SMA) metrics
Smoothed instantaneous values computed across recent transaction messages. Despite their counter exposition type, plot them directly and do not apply rate().
| Metric Name | Exposition type | Description |
|---|---|---|
|
Counter |
Moving average rows per transaction message |
|
Counter |
Moving average payload per transaction message (bytes) |
|
Counter |
Moving average of source read time (ms) |
|
Counter |
Moving average of target write time (ms) |
|
Counter |
Moving average of internal storage time (ms) |
|
Counter |
Moving average of Core Hub internal execution time (ms) |
Cumulative totals
| Metric Name | Type | Description |
|---|---|---|
|
Counter |
Total rows replicated from source to target since process start |
|
Counter |
Total bytes replicated since process start |
|
Counter |
Total rows replicated during initial snapshot phases |
Use rate() for per-second throughput and multiply by 60 for a per-minute row rate:
sum(rate(gluesync_total_count{pipelineName=~"$pipelineName",entityName=~"$entityName"}[$__rate_interval]))
Pipeline health and freshness
Data Freshness
This family is conditional. It is omitted from the scrape until the stall monitor has evaluated the entity.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Seconds elapsed since the entity last received a batch with |
gluesync_entity_dry_seconds is the primary KPI for detecting stalled or idle pipelines.
A pipeline that is dry for longer than its configured dataFlowStall.stallDurationMinutes is a candidate for an alert.
|
A sawtooth pattern is expected for intermittently active entities: the value rises while idle and resets when data arrives. A continuously rising value means delivery has stopped. Exclude -1 when looking for the longest current stall:
max(gluesync_entity_dry_seconds{pipelineName=~"$pipelineName",entityName=~"$entityName"} >= 0)
gluesync_entity_dry_seconds
> on() group_left()
(gluesync_threshold_data_flow_stall_duration_minutes * 60)
CDC read checkpoints
Checkpoint metrics report the source reader’s persisted CDC position. They are omitted until a supported Core Hub-managed CDC reader has persisted a position.
Since 2.2.11.5 they carry pipelineName and sourceAgentName beside pipelineId and sourceAgentId, plus entityId and entityName. The entity pair is filled in only for the checkpoints Core Hub derives per entity, whose checkpointKey already is an entity id, and is empty for a connector’s own key, which covers a journal or a binlog that several entities share. Both labels are emitted either way: an empty value is how Prometheus spells "not applicable", and a label present on only half a family cannot be selected on.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Seconds between now and the source timestamp at the persisted read position. Labels include |
|
Gauge |
A numeric component of the connector-specific read position. The |
|
Gauge |
Unix timestamp in seconds represented by the persisted read position. |
|
Gauge |
Always |
The checkpoint kind identifies the connector family. Current examples include LogminerCheckpoint, BinlogCheckpoint, As400JournalCheckpoint, and InformixCdcCheckpoint. Numeric position fields vary accordingly:
-
Oracle LogMiner:
scnNumber,ssn, andresetlogsId -
MySQL and MariaDB:
binlogPositionandoperationSeq -
AS/400:
sequenceNumber -
Informix:
id
Position values are only comparable with the same reader and field over time. Connector position scales differ, so do not compare an Oracle SCN with an AS/400 journal sequence. A flat checkpoint position or zero advance rate, together with a rising age, indicates a stalled reader.
max(gluesync_read_checkpoint_age_seconds{pipelineName=~"$pipelineName",sourceAgentName=~"$sourceAgentName"})
clamp_min(
deriv(gluesync_read_checkpoint_position{pipelineName=~"$pipelineName",sourceAgentName=~"$sourceAgentName",field=~"$checkpointField"}[5m]) * 60,
0
)
Checkpoint age is source CDC freshness. gluesync_sma_end_to_end_time and gluesync_last_end_to_end_time measure Core Hub internal execution after a change has entered the cache; they are not substitutes for checkpoint age.
|
Per-entity CDC lag
These families answer how far behind an entity is, which the checkpoint families above cannot: they report the position of a reader, not the size or the age of what is waiting for it. Lag is measured at the two hops where a change can pile up:
-
gluesync_source_*covers source database to Gluesync: what the source has generated and Core Hub has not read yet. -
gluesync_cache_*covers local CDC cache to target: what Core Hub has read and not yet written.
The hops are reported separately because they have different causes and different fixes. A reader two minutes behind the source while the cache hop shows twenty hours means the target is the bottleneck, not the source.
The figures come from a probe driven off the CDC poll loop, by default every 15 seconds per entity. GS_ENTITY_LAG_PROBE_INTERVAL_SECONDS sets the interval and disables the probe when set to 0, GS_ENTITY_LAG_PROBE_TIMEOUT_SECONDS bounds a single probe, and GS_ENTITY_LAG_PROBE_CONCURRENCY bounds how many run at once. The probe borrows its connection under the monitoring purpose, which the pool refuses rather than queues, so a measurement is skipped instead of competing with the replication it describes.
All families in this section carry pipelineId, pipelineName, entityId, entityName, sourceAgentId, and, since 2.2.11.5, sourceAgentName.
|
There is no |
Source hop
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
How long the oldest change the source has generated and Gluesync has not read yet has been waiting. |
|
Gauge |
Change rows the source has generated and Gluesync has not read yet. Absent when the connector cannot count them without reading them. |
|
Gauge |
|
|
Gauge |
|
|
Gauge |
|
|
Gauge |
Bytes between the reader’s position and the head of the source’s change log, such as a WAL or a binlog. This is the distance the reader still has to travel, not this entity’s backlog: the log is shared by the whole database and most of those bytes are changes this entity does not sync. A distance of |
|
Gauge |
Unix timestamp of the oldest change the source is still holding, which is the next one Gluesync will read. |
|
Gauge |
Unix timestamp of the last change Gluesync has read from the source. |
Cache hop
Only cache-based connectors have this hop. A connector that reads the source and emits directly publishes nothing here.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
How long the oldest change sitting in the local CDC cache and not yet delivered to the target has been waiting. |
|
Gauge |
Change rows read into the local CDC cache and not yet delivered to the target. |
|
Gauge |
|
|
Gauge |
|
|
Gauge |
|
|
Gauge |
Bytes sitting in the local CDC cache between this entity’s read position and the head. Unlike the source log distance, these bytes belong to this entity alone. |
|
Gauge |
Unix timestamp of the oldest change in the cache not yet delivered to the target, which is the next one the target will receive. |
|
Gauge |
Unix timestamp of the last change this entity consumed from the local CDC cache. |
Probe state
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Outcome of the last probe. One series per possible outcome, carried in the |
|
Gauge |
How long the last probe took, so the cost of measuring on the source database stays visible. A rising duration is a source getting slower, not the pipeline being starved. |
|
Gauge |
Seconds since this entity’s figures were measured. Grows past the probe interval when probes stop completing. |
Every outcome is emitted on every scrape rather than only the current one, so a sum by (result) cannot count one entity twice and a recovered entity does not keep reading as failing:
result |
Meaning |
|---|---|
|
The probe answered. Individual figures can still be absent when the connector cannot produce them. |
|
That connector has no probe for this figure. Not a fault. |
|
Nothing has been read for this entity yet, so there is no position to measure a backlog from. Expected on a freshly configured entity, where it clears on the first read, and a fault if it persists. |
|
The probe threw. The Core Hub log carries the source error. |
|
The probe did not answer within |
|
Every connection was serving the pipeline, so the measurement was skipped. Not a failure, and the figures shown are the last ones measured. |
|
The probe waited longer than one interval for a thread and was dropped before it ran. This is the probe pool, not the pipeline: raise |
|
No probe has completed for three of that entity’s sampling intervals, and the backlog figures are withheld. Substituted at scrape time for whatever the last completed probe said. Since a probe can only start on a CDC poll tick, this usually means the pipeline itself is stuck. |
|
A figure a connector cannot produce is an absent series, never a zero. Zero is a legitimate backlog value that means caught up, so it cannot also stand for unknown, the way |
Two further properties change how these series must be queried:
-
Figures older than three of that entity’s sampling intervals are not published at all, the cadence being the slower of the probe interval and the entity’s own polling interval. Because the lag is computed against now, a stale oldest-pending change would go on growing on an entity that may have caught up. Past the bound the backlog series disappear,
resultbecomesstale, andgluesync_entity_lag_sample_age_secondsis what says why. -
Every line carries an explicit timestamp. Prometheus does not mark an explicitly timestamped series stale when Core Hub stops exposing it, so after an entity is stopped or removed its last values stay queryable for the whole lookback window, five minutes by default. They stop moving, they do not vanish. Gate on the probe outcome instead of trusting the presence of a series:
gluesync_source_lag_milliseconds{pipelineName=~"$pipelineName",entityName=~"$entityName"} and on (pipelineId, entityId) (gluesync_entity_lag_probe_info{result=~"ok|abandoned"} == 1)
abandoned belongs in the gate because it keeps the last measurement on purpose, unlike stale.
What a connector can measure differs widely. A trigger-based staging table gives an exact count for the price of one indexed COUNT(*), while a redo, binlog, or journal reader can only be counted by reading it, so several connectors report a time or a byte distance and no row count at all.
# Source database to Gluesync
max(
gluesync_source_lag_milliseconds
and on (pipelineId, entityId)
(gluesync_entity_lag_probe_info{result=~"ok|abandoned"} == 1)
)
# Local CDC cache to target
max(
gluesync_cache_lag_milliseconds
and on (pipelineId, entityId)
(gluesync_entity_lag_probe_info{result=~"ok|abandoned"} == 1)
)
sum(gluesync_entity_lag_probe_info{result=~"error|timeout|busy|stale"}) or vector(0)
The or vector(0) matters: the probe emits every outcome on every scrape, so summing the failing ones returns a number rather than an empty result once any entity is being probed.
Entity lag is not gluesync_read_checkpoint_age_seconds. Checkpoint age is how long the persisted read position has stood still; entity lag is how far behind the source or the target the entity actually is, computed at scrape time from the oldest change that has not moved past that hop. The two families are deliberately kept apart, and only the lag families carry entity labels.
|
Database connection metrics
These gauges expose Core Hub’s live JDBC connection registry.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Live borrowed connections attributed to an agent and purpose. Labels include |
|
Gauge |
Live borrowed connections aggregated by |
Known purposes are pipeline_task, query_studio, query_forge_endpoint, and unspecified. Rising ad-hoc query usage can indicate that Query Studio or Query Forge is competing with replication work for database capacity.
Core Hub also exposes standard HikariCP pool metrics. Useful KPIs include:
-
Pool utilization:
hikaricp_connections_active / hikaricp_connections_max -
Waiting borrowers:
hikaricp_connections_pending -
Average acquisition time:
rate(hikaricp_connections_acquire_seconds_sum[$rate_interval]) / rate(hikaricp_connections_acquire_seconds_count[$rate_interval])
Sustained utilization near 1, especially with pending borrowers and rising acquisition time, indicates pool exhaustion.
Alert threshold configuration metrics
These metrics expose the active alert threshold configuration as Prometheus gauges. They are emitted once per scrape with no entity labels — they represent the global configuration applied to all pipelines.
Exposing thresholds as metrics enables Grafana to:
-
Show configured vs. observed values on the same panel
-
Trigger alerts when a metric approaches its threshold (e.g., latency at 80% of limit)
-
Record a history of threshold changes over time
Global Thresholds
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Static latency alert threshold in minutes. Only emitted when configured (not null). |
|
Gauge |
Row-count drift percentage threshold. Only emitted when configured (not null). |
|
Gauge |
|
|
Gauge |
Minimum quiet period between repeated alerts of the same type (minutes). |
Data Flow Stall Detector
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
|
|
Gauge |
How many consecutive minutes of zero throughput trigger a stall alert. |
|
Gauge |
Warm-up window excluded from stall detection after a pipeline starts (minutes). |
Entity heartbeat watchdog
The heartbeat family exposes the source-inactivity watchdog configuration introduced for 2.2.11.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
|
|
Gauge |
Absolute inactivity ceiling before a hard-floor alert fires (minutes). |
|
Gauge |
Pattern-aware inactivity threshold (minutes). |
|
Gauge |
Minimum average changes per hour required for pattern-aware alerts. |
|
Gauge |
Historical bucket observations required before pattern-aware alerting activates. |
|
Gauge |
EWMA smoothing factor used by the heartbeat watchdog. |
|
Gauge |
Maximum silence before a heartbeat-missed alert fires (minutes). |
These values are configuration gauges. Use gluesync_entity_dry_seconds, when emitted, as the observed inactivity measurement to compare with the configured limits.
Compatibility detector settings
The following gauges remain exposed for configuration compatibility and historical visibility. In current 2.2 smart-alert behavior, sensitivity is selected as Low, Normal, or High; nested EWMA, latency-pattern, and seasonal tuning values are retained on the wire but are not active detector controls.
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Configured throughput-anomaly toggle. The legacy EWMA detector does not currently fire alerts. |
|
Gauge |
Configured legacy EWMA smoothing factor. |
|
Gauge |
Configured legacy EWMA control multiplier. |
|
Gauge |
Configured legacy minimum sample count. |
Compatibility latency-pattern settings
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Configured latency-pattern toggle. The legacy pattern detector does not currently fire alerts. |
|
Gauge |
Configured legacy spike multiplier. |
|
Gauge |
Configured legacy sustained-latency multiplier. |
|
Gauge |
Configured legacy sustained window (minutes). |
Compatibility seasonal settings
| Metric Name | Type | Description |
|---|---|---|
|
Gauge |
Configured seasonal-detection toggle. The legacy seasonal detector does not currently fire alerts. |
|
Gauge |
Configured legacy minimum history in days. |
|
Gauge |
Configured legacy modified Z-score threshold. |
Host and JVM metrics
System-level resource metrics emitted by the Gluesync host machine.
| Metric Name | Type | Description |
|---|---|---|
|
Counter |
Time since machine start. The 2.2.11 HELP text says milliseconds, but the emitted value is seconds. |
|
Counter |
Unix epoch time of last boot. The 2.2.11 HELP text says milliseconds, but the emitted value is epoch seconds. |
|
Counter |
Per-processor CPU load as a ratio from |
|
Counter |
Overall system CPU load as a ratio from |
|
Counter |
Total CPU interrupt count |
|
Counter |
Total RAM (megabytes) |
|
Counter |
Available RAM (megabytes) |
|
Counter |
OS memory page size (megabytes) |
|
Counter |
Total data received per network interface (megabytes). Label: |
|
Counter |
Total data sent per network interface (megabytes). Label: |
|
Counter |
Link speed per network interface (megabits/s). Label: |
CPU load, RAM, network speed, uptime, and boot time are declared as counters in 2.2.11 even though most are sampled state. Query them directly. Use rate() only for genuinely cumulative series such as interrupt count and network bytes.
|
Useful host queries:
# RAM currently in use
gluesync_ram_total - gluesync_ram_available
# Network receive rate per interface
rate(gluesync_network_interface_mb_received[$__rate_interval])
JVM runtime metrics
Standard Micrometer JVM families use their normal Prometheus types and base units.
| Metric family | Type | Description |
|---|---|---|
|
Gauge |
Always |
|
Gauge |
Estimated buffer count by pool |
|
Gauge |
Memory used by each buffer pool. |
|
Gauge |
Total capacity of each buffer pool. |
|
Summary |
Concurrent GC phase duration. Prometheus exposes |
|
Gauge |
Maximum concurrent GC phase duration in the current publishing window. |
|
Gauge |
Long-lived heap size after reclamation. |
|
Gauge |
Maximum long-lived heap size. |
|
Counter |
Bytes allocated in the young heap between collections. |
|
Counter |
Positive increases in old-generation size across collections. |
|
Gauge |
Approximate fraction of CPU time spent in garbage collection, from |
|
Summary |
GC pause duration. Prometheus exposes quantiles plus |
|
Gauge |
Maximum GC pause in the current publishing window. |
|
Gauge |
Memory committed by area and pool |
|
Gauge |
Maximum memory by area and pool. |
|
Gauge |
Long-lived heap fraction used after the latest GC, from |
|
Gauge |
Current memory usage by |
|
Gauge |
Current live daemon threads. |
|
Gauge |
Current live daemon and non-daemon threads. |
|
Gauge |
Peak live thread count since JVM start or peak reset. |
|
Counter |
Total application threads started. |
|
Gauge |
Current threads by |
The official dashboard uses:
-
jvm_memory_used_bytes{area="heap"}for heap usage by pool -
rate(jvm_gc_pause_seconds_sum[$rate_interval]) / rate(jvm_gc_pause_seconds_count[$rate_interval])for average GC pause -
jvm_threads_live_threadsfor live thread count
An Eden heap sawtooth is normal. Old-generation usage that continues rising without falling after garbage collection can indicate retained objects. Interpret GC pause and thread count together with heap behavior and host CPU.
Ktor HTTP server metrics
| Metric family | Type | Description |
|---|---|---|
|
Gauge |
HTTP requests currently being processed. |
|
Summary |
Request duration in seconds. Exposes configured quantiles and the |
|
Gauge |
Maximum request duration in the current publishing window. |
Request-duration labels are address, method, route, status, and throwable. Prefer the normalized route label for endpoint analysis. The address label can split traffic by the hostname used by internal and external callers.
sum by (route, status) (
rate(ktor_http_server_requests_seconds_count[$__rate_interval])
)
sum by (route) (
rate(ktor_http_server_requests_seconds_sum[$__rate_interval])
)
/
sum by (route) (
rate(ktor_http_server_requests_seconds_count[$__rate_interval])
)
route, status, method, address, and throwable create separate time series. Aggregate away labels that are not needed, and avoid alerts over unrestricted combinations.
|
HikariCP pool metrics
HikariCP metrics are conditional and may be absent when no instrumented pool has registered. Common families used by the official dashboard include hikaricp_connections_active, hikaricp_connections_max, hikaricp_connections_pending, and the hikaricp_connections_acquire_seconds summary.