Smart alerts

Smart alerts control when the anomaly detectors fire events. They are tuned under Settings → Smart alerts in the Control Plane, where you can also view live entity health and simulate new values against real data.

Global defaults and pipeline overrides

Settings under Settings → Smart alerts are the global defaults. Use them as a baseline for pipelines that do not need separate tuning.

Smart Alerts can also be configured on an individual pipeline. Pipeline settings can override the global threshold, sensitivity, cooldown, and enabled anomaly types. Use an override when the traffic pattern of one pipeline differs from the rest of the environment. For example, use higher sensitivity for a critical high-volume pipeline, or disable stall and latency anomaly types for a pipeline that normally receives only a few rows each day.

Pipeline-level configuration does not change the global defaults or other pipelines. Smart Alerts are available in the Control Plane, but anomaly detection still depends on representative traffic history. Review and tune the results for each pipeline.

Enable Smart Alerts and override defaults

  1. Open the pipeline in the Control Plane.

  2. Open its Smart Alerts settings.

  3. Enable Smart Alerts.

  4. Select the anomaly types to monitor.

  5. Override the sensitivity, threshold, or cooldown values that need pipeline-specific tuning.

  6. Save the pipeline.

The resulting configuration for a high-volume pipeline that needs earlier notification looks like this:

Pipeline: Orders CDC
Smart Alerts: Enabled
Anomaly types: Throughput, Latency, Data flow stall
Sensitivity: Higher than global
Cooldown: 5 minutes (global: 15 minutes)

Values that you do not override continue to inherit the global settings.

Reset to inherit

Use Reset to inherit to remove pipeline-specific values and follow the current global defaults again.

  1. Open the pipeline’s Smart Alerts settings.

  2. Click Reset to inherit for the override.

  3. Save the pipeline.

The cooldown override then follows the global value again:

Before reset:
  Pipeline cooldown: 5 minutes

After Reset to inherit:
  Pipeline cooldown: Inherited (15-minute global default)

Resetting an override does not disable Smart Alerts and does not modify the global setting.

Select monitored entities

When the pipeline contains entities, use the monitored-entities selector as an allowlist.

  1. Open the pipeline’s Smart Alerts settings.

  2. Under Monitored entities, select the entities that Smart Alerts should evaluate.

  3. Clear entities that should not generate pipeline anomaly notifications.

  4. Save the pipeline.

For a pipeline whose reference tables change only occasionally, monitor the transactional entities and leave the rest unselected:

Pipeline: Finance CDC
Monitored entities:
  - invoices
  - payments

Not monitored:
  - daily_exchange_rates

Use entity selection to exclude low-frequency objects whose expected idle periods would otherwise look like stalls or latency anomalies.

Static thresholds

Setting Default Description

Latency threshold (minutes)

10

Fire REPLICATION_LATENCY_THRESHOLD_EXCEEDED when end-to-end latency exceeds this value. Severity is WARNING at 1×, CRITICAL at 3×.

Row count drift (%)

5

Fire ROW_COUNT_DRIFT_DETECTED when source and target row counts diverge by more than this percentage.

Alert cooldown

Setting Default Description

Alert cooldown (minutes)

15

Minimum quiet period between repeated alerts of the same type for the same entity. Prevents alert storms during sustained anomalies.

Data flow stall detector

Fires DATA_FLOW_STALL_DETECTED when an entity stops emitting batches.

Setting Default Description

Enabled

Yes

Toggle this detector on or off.

Stall duration (minutes)

5

How long an entity must be idle before an alert fires.

Warmup period (minutes)

10

Grace period after a pipeline starts during which stall detection is suppressed.

Throughput anomaly detector

Fires THROUGHPUT_ANOMALY_DETECTED using EWMA control charts.

Setting Default Description

Enabled

Yes

Toggle this detector on or off.

Smoothing factor (α)

0.3

EWMA smoothing coefficient in the range (0, 1). Higher values give more weight to recent batches. Lower values mean slower baseline adaptation.

Control multiplier (σ)

3.0

Number of standard deviations above the EWMA mean that define the upper control limit. Exceeding this limit triggers an alert.

Minimum data points

20

Number of batches required before the detector activates. Prevents false positives during ramp-up.

Latency pattern detector

Fires LATENCY_ANOMALY_DETECTED for both sudden spikes and sustained elevated latency.

Setting Default Description

Enabled

Yes

Toggle this detector on or off.

Spike multiplier

3.0

Immediate alert when current latency is greater than spike_multiplier × rolling_baseline.

Sustained multiplier

2.0

Sustained alert when latency remains above sustained_multiplier × baseline for the full window.

Sustained window (minutes)

10

Rolling window over which sustained latency is evaluated.

Seasonal throughput detector

Fires THROUGHPUT_ANOMALY_DETECTED based on same-time-of-day historical patterns.

Setting Default Description

Enabled

No

Toggle this detector on or off. Disabled by default — requires sufficient history.

Minimum days history

2

Days of history required before seasonal analysis can activate.

Modified Z-score threshold

3.5

Sensitivity cut-off for the modified Z-score algorithm. Higher values = less sensitive to seasonal deviations.

Pattern status

The Pattern status panel shows the current health of every monitored entity. Each card displays:

  • Pipeline and entity name (or ID if the name is not available)

  • Current latency in milliseconds

  • Current throughput

  • Health status (HEALTHY, CRITICAL, or other states)

This gives you an at-a-glance view of which entities are under observation and whether any are currently flagged by the detectors.

WARNING and ERROR in the UI

A Smart Alerts threshold hit changes the affected pipeline or entity to WARNING. WARNING identifies a non-blocking anomaly that needs attention; replication can continue while you investigate it. ERROR identifies a failure that blocks or breaks the operation.

WARNING: Replication continues, but a Smart Alerts threshold was exceeded.
         Review latency, throughput, stall, and pipeline-specific settings.

ERROR:   Replication has failed or cannot continue.
         Investigate the operation or agent failure before relying on the flow.

Treat WARNING as a prompt to inspect the anomaly and its tuning. Do not interpret it as proof of data loss or as a production-stopping failure.

Simulation

Click Simulate to run the configured thresholds against live data without actually dispatching alerts. The UI displays a list of WebhookDispatchRequestDto objects that would have been generated, including:

  • Event type and severity

  • Affected pipeline and entity

  • Human-readable message explaining why the threshold fired

Use simulation to validate new threshold values before committing them to production.

Viewing active thresholds in Prometheus / Grafana

All threshold settings are automatically published as Prometheus gauge metrics (see Alert threshold configuration metrics).

This means you can:

  • See the exact threshold values in Grafana without opening the settings UI

  • Plot observed metrics alongside their thresholds on the same panel

  • Alert from Grafana when a metric approaches its threshold (for example, 80% of the stall duration)

  • Track historical threshold changes via the Prometheus TSDB

See Recommended Grafana panels for example PromQL queries that overlay thresholds with live metrics.

Related documentation