Pipelines
A pipeline connects one source agent to one target agent. Inside it, each entity maps a source table or collection to its destination and defines how that data moves.
|
How-to guides
For step-by-step walkthroughs tested on a running Gluesync, see the how-to guides Start a snapshot and CDC replication and Filter rows with a snapshot WHERE clause and target filters. |
Use this page as the map for pipeline work. If you want to build one now, go directly to Build your first pipeline.
Pipeline model
- Source connector
-
Reads changes or snapshot rows from the source system.
- Target connector
-
Writes those records to a destination. A pipeline has exactly one target agent: Core Hub refuses a second one. To send the same source to several targets, create one pipeline per target.
- Entity
-
The replication unit: usually one table or collection and its target mapping.
- Snapshot
-
Copies the current contents of an entity.
- Change data capture (CDC)
-
Continuously forwards inserts, updates, and deletes after replication starts.
The Data pipelines screen is the operational home for this model. Select a pipeline to see its entities, health, throughput, and available actions.
Build and configure
| Task | Guide |
|---|---|
Create a pipeline |
|
Add or replace a source or target |
|
Choose tables and collections |
|
Create many entities from a YAML or ZIP file |
|
Start CDC, run a snapshot, or pause replication |
|
Create groups and ordered chains |
Shape and deliver data
| Goal | Guide |
|---|---|
Filter source rows |
|
Review or change field mappings |
|
Convert, mask, or derive values |
|
Reuse custom logic across entities or compose one column from several inputs |
|
Run custom transformation code |
|
Control target write behavior |
|
Create missing target tables |
Operate a pipeline
-
Read Pipeline states and error recovery before responding to an unhealthy pipeline.
-
Use Snapshot tasks for large initial loads and interrupted snapshots.
-
Put a pipeline in maintenance mode before configuration changes that require agents to disconnect.
-
Configure monitoring and alerts for production.