Error Handling and Pipeline States

Gluesync separates pipeline health from entity replication state. A pipeline can look healthy while one entity is paused, and a single entity can fail without putting the whole pipeline into maintenance. This page explains how to read those signals and what to do when something breaks.

Terminology

Before troubleshooting, keep this distinction in mind:

  • Pipeline status is the overall health signal shown in the Control Plane.

  • Entity status tracks the replication state of a single entity.

  • Notifications describe what failed, when it failed, and which agent or entity is affected.

Gluesync does not expose a dedicated backend pipeline state called "running" or "stopped". In practice, operators often use those words informally:

  • Running usually means the pipeline is not in maintenance mode and the agents are healthy.

  • Paused usually means the pipeline is in maintenance mode or an individual entity has been stopped.

  • Stopped is typically an entity-level condition, not a pipeline-level one.

Pipeline states overview

The pipeline badge in the UI is derived from the pipeline health model:

UI state Meaning What it does not mean

Healthy

All configured agents are healthy, connected, and there are no failed pipeline migrations. If the pipeline is still being configured, the badge can be hidden because the pipeline is marked as draft.

It does not guarantee that every entity is actively syncing right now.

Unhealthy

At least one agent is unhealthy, disconnected, missing configuration, or a pipeline migration has failed.

It does not always mean the whole pipeline is down. Often only one agent or one migration path needs attention.

Maintenance

The pipeline is intentionally paused. Core Hub stops the pipeline activity, disconnects agents from database work, and suppresses the normal noise of outage-related errors.

It is not an error state. It is a deliberate operator action.

Separate from the health badge, the UI also treats draft pipelines differently:

  • A draft pipeline is still being configured.

  • Draft pipelines do not behave like fully active pipelines yet.

  • Once configuration is completed, the normal pipeline state badge becomes meaningful.

Entity states and replication status

Entity-level state is where replication activity actually lives.

The UI exposes:

  • Active - the entity is currently syncing or snapshotting.

  • Hold - the entity is paused.

  • Error - the entity hit a replication problem that needs attention.

The entity row also shows replication activity indicators such as:

  • CDC (Idle) - CDC is enabled but currently not busy.

  • CDC (Busy) - CDC is processing events.

  • Snapshot in progress - a snapshot is running.

  • Error - the replication flow is blocked.

Important distinction:

  • A pipeline can be Healthy while one entity is Hold.

  • A pipeline can be Healthy while one entity is temporarily retrying.

  • A pipeline becomes Unhealthy when the agent-level model or migration state is broken, not just because a single entity was paused.

What Gluesync retries automatically

Gluesync already retries several failure classes for you. Do not manually restart everything immediately; first check whether the system is already recovering.

Failure type Automatic behavior Operator action

Agent heartbeat or connection drop

The UI and Core Hub keep watching status updates. If an agent becomes unreachable for long enough, the pipeline is marked unhealthy. When the agent reconnects and reports a fresh status, the pipeline recovers automatically.

Wait for the reconnect if the root cause is transient. If it stays down, check the agent and database logs.

WebSocket UI connection loss

The UI websocket client reconnects every second until the server is reachable again.

Usually none. Refresh only if the UI stays stale after the backend is healthy.

Source CDC coroutine cancellation or temporary context loss

Gluesync retries the entity sync/snapshot flow and can restart the entity task automatically.

Usually none. If it keeps happening, investigate the source agent and its database connection.

Snapshot start failure

Snapshot start uses a bounded retry loop. Some failures are retried automatically several times before Gluesync gives up.

If retries are exhausted, fix the root cause and start the snapshot again manually.

Source read errors caused by a missing or broken entity context

Gluesync queues the entity for restart and replays the flow after a backoff delay.

Usually none unless the same error keeps returning.

Target write handling and transaction dispatch errors

Gluesync records the failure, writes a notification, and restarts the affected entity through the retry channel when possible.

Check the target-side error first. Restart manually only after the cause is fixed or if retries do not recover the entity.

What surfaces as an error

Not every failure is retried forever. Some errors are intentionally surfaced because they usually need human action.

Common examples:

  • Database not initialized - the agent cannot even start the sync or snapshot path.

  • Schema or mapping problems - a field is missing, renamed, or incompatible on the target.

  • Target constraint violations - unique key, nullability, foreign key, or similar enforcement failures.

  • Migration failures - the pipeline migration record is stored and the UI exposes a retryable failed migration entry.

When Gluesync cannot recover safely, it writes a notification with a human-readable title and message, and the affected entity may remain in an error state until you intervene.

How errors appear in the UI

You should check three places:

  1. Pipeline badge

    • Healthy, unhealthy, or maintenance tells you whether the pipeline control plane is satisfied.

  2. Entity row

    • Active, hold, or error tells you which entity is actually blocked.

    • The entity detail view also shows replication type and current activity.

  3. Notifications Hub

    • Error notifications are the primary troubleshooting trail.

    • Warnings are used for non-fatal problems such as an entity no longer existing when a message arrives.

    • Information entries track operational actions such as snapshot start/end, sync start/stop, and maintenance transitions.

Logs are still the source of truth when the notification text is not enough.

Common error scenarios

Target database unreachable

This usually shows up as a target write failure or a target connection error.

What happens:

  • The affected entity can move into an error state.

  • Gluesync creates an error notification for the entity or transaction.

  • The system may retry the entity flow automatically after a backoff.

What to check:

  • Target database availability.

  • Network reachability from the target agent.

  • Credentials, TLS, and connection pool limits.

  • Whether the target agent itself is still reporting healthy status.

How to resolve:

  • Restore the target database or network path.

  • Fix credentials or TLS configuration if needed.

  • Restart the entity replication once the target is reachable again.

Schema mismatch on the target

This happens when a source change cannot be applied because the target schema no longer matches the mapping.

Typical causes:

  • Missing target field.

  • Type incompatibility.

  • Unexpected nullability or length constraint.

  • A source field renamed without updating the pipeline mapping.

What happens:

  • The target write path fails.

  • Gluesync raises an entity-level error notification.

  • Automatic retry is only useful if the mismatch is transient. In most cases it is not.

How to resolve:

  • Compare the source entity definition and the target table definition.

  • Update the field mapping or target schema.

  • Re-run the entity sync or snapshot if required.

Constraint violation on the target

Common violations include unique keys, null constraints, and foreign key violations.

What happens:

  • The entity write path fails.

  • The error is recorded in Notifications Hub.

  • Gluesync may retry the operation, but it will not magically fix the bad row.

How to resolve:

  • Inspect the failing record in the logs.

  • Decide whether to adjust the data, the mapping, or the target constraint.

  • Reprocess the entity after the data issue is fixed.

A duplicate key is the one violation Gluesync treats as a conflict to resolve rather than a plain error: the entity’s On duplicate key setting decides whether the target row is overwritten, kept and reported, or left to stop the entity. Every other violation surfaces as an error whatever that setting says.

Duplicate rows skipped

An entity set to Skip on duplicate keys does not fail when the target refuses a row it already holds. It keeps the target row, writes the rest of the transaction, and raises a warning titled Duplicate rows skipped on <entity> whose message lists the keys of the skipped rows.

This is a warning, not an error: the entity stays in sync and the pipeline is unaffected. It does mean the source and the target now disagree on those rows, so the keys in the message are the ones to reconcile by hand. See On duplicate key.

CDC source lag or disconnection

Source lag is not always an error. A temporary delay in the source feed can simply mean there are no new changes yet.

What happens when the source is actually disconnected:

  • The entity read flow reports an error.

  • Gluesync writes a notification with the source-side failure.

  • The entity may restart automatically after the backoff delay.

How to resolve:

  • Check source database connectivity and source agent health.

  • Verify CDC is still enabled on the source system.

  • If the source was intentionally restarted, wait for the agent to reconnect and then resume the entity if needed.

Diagnosing a pipeline error

When a pipeline turns unhealthy or an entity enters error state, use this order:

  1. Check the pipeline badge.

  2. Open the affected entity row and note whether it is Active, Hold, or Error.

  3. Open the Notifications Hub and filter by pipeline, entity, or agent.

  4. Read the matching agent logs and Core Hub logs.

Useful clues:

  • A stale agent status for more than roughly one minute usually means the agent stopped reporting updates.

  • A notification with a retry message usually means Gluesync is still trying to recover.

  • A notification with a specific schema or constraint error usually means you need to fix data or mapping before retrying.

Resuming after an error

The recovery path depends on the failure:

  • Transient connection issue - fix the connectivity problem and wait for Gluesync to reconnect or retry.

  • Entity-level replication problem - fix the source or target issue, then restart the entity from the pipeline or entity detail view.

  • Failed migration - use the migration retry action for the failed migration entry once the root cause is fixed.

  • Maintenance mode - exit maintenance mode first, entities will be restarted automatically back to their previous state.

Pipeline error vs entity error

This is the most important troubleshooting distinction.

Pipeline-level error

  • Affects the overall pipeline health badge.

  • Usually involves agent health, maintenance mode, or failed pipeline migrations.

  • Often requires checking the pipeline configuration or agent connectivity.

Entity-level error

  • Affects one source/target entity pair.

  • Usually involves source read problems, target write problems, or a bad row.

  • May be automatically retried by Gluesync.

  • Often only needs a manual restart after the root cause is fixed.

If you are unsure where to start, check the Notifications Hub first and then decide whether the problem is pipeline-wide or isolated to a single entity.

Related topics