Search docs.molo17.com

    Connector concepts

    A connector, called an agent in the Control Plane, is the part of Gluesync that talks to one database or service. A source agent reads data and changes from a system. A target agent writes them to another one. Since Gluesync 2.2 agents run inside Core Hub, so there are no separate agent images or containers to deploy.

    This page explains the ideas you meet when you set up your first pipeline. For the full component and data-path reference, see Gluesync architecture.

    How a record moves through a pipeline Four stages inside Core Hub, in a loop: top left, top right, bottom right, bottom left. One, capture: the source agent reads a snapshot of the table, then every change through the source's own mechanism, such as the transaction log, change tables, triggers, change streams or the journal. Two, checkpoint: Core Hub saves the position of each entity, so a restart resumes where it stopped. Three, map: the Fields Editor, filters, field functions and user-defined functions shape each record. Four, write: the target agent applies the records in batches or with the target's bulk loader, matched on the key. How a record moves through a pipeline Inside Core Hub 1 Capture The source agent reads a snapshot, then every change after it log · change tables · triggers · streams 2 Checkpoint Core Hub saves the position of every entity a restart resumes from there 3 Map Fields, filters, field functions and UDFs shape each record types converted for the target 4 Write The target agent applies batches or the target's bulk loader rows matched on the key How a record moves through a pipeline Four stages inside Core Hub, top to bottom. One, capture: the source agent reads a snapshot of the table, then every change through the source's own mechanism, such as the transaction log, change tables, triggers, change streams or the journal. Two, checkpoint: Core Hub saves the position of each entity, so a restart resumes where it stopped. Three, map: the Fields Editor, filters, field functions and user-defined functions shape each record. Four, write: the target agent applies the records in batches or with the target's bulk loader, matched on the key. How a record moves through a pipeline Inside Core Hub 1 Capture The source agent reads a snapshot, then every change after it log · change tables · triggers · streams 2 Checkpoint Core Hub saves the position of every entity a restart resumes from there 3 Map Fields, filters, field functions and UDFs shape each record types converted for the target 4 Write The target agent applies batches or the target's bulk loader rows matched on the key

    Source and target agents

    Each agent connects to one technology. Many technologies have both a source agent and a target agent. Some are one-sided: Snowflake, Databricks, Supabase, Apache Kafka and Microsoft Fabric Open Mirroring are targets only.

    • A source agent reads the current contents of tables or collections (the snapshot) and then the changes made after that.

    • A target agent applies those records to the destination.

    • A technology can have more than one agent when it offers more than one way to capture changes. Oracle, SQL Server and Db2 for LUW are examples.

    Agents use the connection method the vendor provides. Relational agents use the vendor JDBC driver, for example the Snowflake JDBC driver or the IBM JTOpen driver for IBM i. MongoDB, Cassandra, Couchbase and others use the official Java or Kotlin SDK. Some targets use an HTTP API instead: Databricks uses the SQL Statement Execution API, Elasticsearch and Apache CouchDB use their HTTP JSON API.

    The Supported integrations page lists every agent with its role, supported versions and connection technology.

    How a source captures changes

    An entity, the unit that moves one table or collection, starts in one of two ways, and you can combine them:

    • Snapshot: a one-time copy of the current data. See Snapshot tasks.

    • Change data capture (CDC): a continuous stream of inserts, updates and deletes that follows the snapshot.

    The usual first run is a snapshot followed by CDC, which you choose in the start panel with the option to run an initial snapshot ahead of CDC.

    The way an agent captures changes depends on what the source database offers. Each agent uses the mechanism of its own platform:

    Transaction or redo logs

    The agent reads the log the database already writes. The PostgreSQL agent reads the write-ahead log through a logical replication slot. The Oracle LogMiner agent reads the online and archived redo logs. MySQL and MariaDB read the binary log.

    Change tables and Change Tracking

    The database records changes in its own structures. The SQL Server Change Tracking agent uses the Change Tracking feature, and the SQL Server CDC agent uses the SQL Agent change data capture feature.

    Triggers and shadow tables

    The agent creates triggers that write changed rows to an audit table, and reads that table. The Oracle Triggers agent keeps a CDC table in a schema you configure. The SAP HANA agent uses shadow tables.

    Change streams and change feeds

    The system offers an API for changes. The MongoDB agent uses Change Streams, DynamoDB uses DynamoDB Streams, and Azure Cosmos DB uses its change feed.

    Journals

    The IBM i agent reads the Db2 for IBM i journal through the journal APIs.

    Source agents for some technologies, such as MySQL, MariaDB, Informix, Oracle and IBM i, keep captured changes in a local cache before sending them to Core Hub, so a short outage between the agent and Core Hub does not force a resynchronization. See Source-side cache for which agents do this and how long changes are kept.

    Setting up a source usually needs permissions or database objects that the agent cannot create on its own. Follow the guide of your source agent before you start an entity.

    How a target writes

    A target agent never writes one row at a time. It groups rows and writes them with one of two strategies, described in Write strategies:

    • Batch mode is the default. The agent sends SQL batch operations in chunks. The chunk size is the maxWriteBatchSize setting of the target agent.

    • Bulk mode uses the native loading mechanism of the target. You turn it on per entity, separately for the snapshot and for CDC. If a target agent does not support bulk mode, Gluesync keeps using batch mode.

    The bulk mechanism depends on the target. PostgreSQL uses COPY through a staging table, Snowflake uses COPY INTO, Amazon Redshift loads CSV files from S3, and Google BigQuery loads Parquet files. In bulk mode Gluesync creates a staging table on the target, fills it with the incoming rows and then applies it to the final table. The steps are described in Bulk load.

    When bulk operations are used during CDC, an update is applied as a delete of the existing row followed by an insert of the new one, not as an UPDATE statement. The target page of each agent says whether this applies and what it means for triggers, foreign keys and change logs on the target.

    Some targets write to something other than a table. Apache Kafka and Google Pub/Sub publish messages, and Microsoft Fabric Open Mirroring writes Parquet change files into a landing zone. These agents publish messages or files instead of updating tables, so the guide of each agent describes its own write behavior.

    Keys and duplicates

    Replication needs a unique identifier for each row. Gluesync uses the primary key of the source table. If a table has none, you can define a logical key in the Fields Editor. See Logical keys. For NoSQL targets you can compose the document key from several source fields. See Custom document keys.

    When an insert collides with a row that is already on the target, the entity’s On duplicate key setting decides whether the row is overwritten, skipped or treated as an error.

    Target tables

    If the target table does not exist when you configure an entity, Gluesync generates a CREATE TABLE statement from the source definition, adapts the data types to the target, and shows it to you to review before it runs. This needs DDL permission on the target. See Target table creation.

    Pipelines, entities and checkpoints

    A pipeline ties one source agent to exactly one target agent. An entity is the replication unit inside it: usually one table or collection with its mapping to the target. See Pipelines.

    Core Hub stores the configuration of pipelines, agents and entities. It also stores the position of each entity, its checkpoint. For the source agents that use a local cache, Core Hub persists the CDC position, and the cache holds the changes that have not been delivered yet.

    A checkpoint is what makes a restart safe:

    • When you stop an entity, restart an agent or update the platform, the entity resumes from its saved position. Checkpoints are preserved during platform updates.

    • An interrupted snapshot resumes from where it stopped. You can discard the saved state by choosing Full copy in Start from when you select Start sync, or Start over under Copy progress when you select Run one-time snapshot. Do not choose them after routine maintenance.

    • Some target agents are written so that a batch applied twice leaves the same result. The Databricks agent is documented this way, because it applies every change as a MERGE matched on the key.

    To suspend a whole pipeline while a source or target database is down or under maintenance, use maintenance mode. It pauses every entity and suppresses connection error notifications. Entities that were running restart when you turn it off. If the exit takes a long time, see the maintenance mode FAQ.

    For pipeline states and how to recover from errors, see Pipeline states and error recovery.

    Mapping and transformation

    Core Hub maps and transforms each record between the source and the target agent, so agents stay focused on reading and writing. You control the result in these places:

    The data type mapping page explains how types are converted between technologies.

    Security

    • Connections to databases can use TLS. Agents trust the certificate authority you provide in a trust store. See Securing database and service communications. Some agents enable TLS by default, such as Supabase, and the IBM i agent prefers TLS ports when they are available.

    • Passwords in connection settings are encrypted with AES256 using the Core Hub storage key. Replace the placeholder key of the trial kit before going to production. See Move to production.

    • Users sign in to Core Hub with local accounts or OIDC, and roles control what each user can change. See User management and RBAC.

    • Core Hub can use HTTPS and secure WebSockets. See HTTPS and WSS encryption.

    Give each agent a dedicated database account with only the permissions its guide lists.