Search docs.molo17.com

    Pipelines

    A pipeline connects one source agent to one target agent. Inside it, each entity maps a source table or collection to its destination and defines how that data moves.

    How-to guides

    For step-by-step walkthroughs tested on a running Gluesync, see the how-to guides Start a snapshot and CDC replication and Filter rows with a snapshot WHERE clause and target filters.

    Use this page as the map for pipeline work. If you want to build one now, go directly to Build your first pipeline.

    Pipeline model

    Source connector

    Reads changes or snapshot rows from the source system.

    Target connector

    Writes those records to a destination. A pipeline has exactly one target agent: Core Hub refuses a second one. To send the same source to several targets, create one pipeline per target.

    Entity

    The replication unit: usually one table or collection and its target mapping.

    Snapshot

    Copies the current contents of an entity.

    Change data capture (CDC)

    Continuously forwards inserts, updates, and deletes after replication starts.

    The Data pipelines screen is the operational home for this model. Select a pipeline to see its entities, health, throughput, and available actions.

    Build and configure

    Task Guide

    Create a pipeline

    Build your first pipeline

    Add or replace a source or target

    Configure connectors

    Choose tables and collections

    Select data in Objects Browser

    Create many entities from a YAML or ZIP file

    Import entities from a file

    Start CDC, run a snapshot, or pause replication

    Run replication

    Create groups and ordered chains

    Organize entities

    Shape and deliver data

    Goal Guide

    Filter source rows

    Filter data

    Review or change field mappings

    Edit fields and mappings

    Convert, mask, or derive values

    Apply field functions

    Reuse custom logic across entities or compose one column from several inputs

    Use Custom Field Functions

    Run custom transformation code

    Use UDFs

    Control target write behavior

    Choose a write strategy

    Create missing target tables

    Create target tables

    Operate a pipeline