A newer version of this documentation is available.
View Latest (v2.3)

Platform Configuration Backup and Restore (Disaster Recovery)

Looking to backup your database data? If you want to extract a full copy of your database records, you need to run a Snapshot Task. This page instead covers how to save the Gluesync infrastructure configuration (Pipelines, Agents, UDFs) for Disaster Recovery.

This guide describes how to use Gluesync Automator to back up and restore pipeline configurations from a running Gluesync Core Hub.

Automator provides a simple web interface on top of the Bootstrapper’s export and import capabilities, making it easy to:

  • Back up individual pipelines either as YAML metadata files or as full ZIP backups

  • Back up all pipelines in bulk as a ZIP archive (including pipeline configs, agents metadata, schedules and UDF source files)

  • Restore pipelines from existing backups

Automator backups are environment agnostic. A snapshot taken from a staging, production, or disaster recovery site can be restored into any compatible Core Hub cluster, enabling migrations, hot spares, and tenant-specific rollbacks.

When to run backups

Regular backups help in three recurring scenarios:

  • Environment cloning – export from staging or QA and restore into a fresh Core Hub to accelerate new environment bring-up.

  • Disaster recovery – keep recent backups off-site so you can rebuild the control plane in a clean cluster after outages or data loss.

  • Change control – version pipeline YAML files together with infrastructure-as-code so you can roll forward/backward confidently.

For business-critical deployments, schedule backups before large configuration changes and right after every successful deployment. Combine them with periodic restore dry-runs in a sandbox so you know disaster recovery works end to end.

Automator works at the configuration level. It does not capture replication runtime state or connection-level secrets, yet. See Limitations of Automator backups for details.

Prerequisites

Before you start, ensure you have:

  • A running Gluesync Core Hub instance

  • Network connectivity from your workstation to Core Hub

  • Gluesync Automator downloaded for your platform

You can download Automator from the official distribution site:

For details on installation and UI elements, see Gluesync Automator.

Step 1 – Get Automator

  1. Download the latest Automator executable for your operating system.

  2. Start the executable:

    • On Windows and Linux: double-click the file or run it from a terminal.

    • On macOS: extract the .zip file, drag & drop the .app bundle to the Applications folder, then open it from there.

    • On Linux: run the executable from a terminal (chmod +x ./gluesync-automator then ./gluesync-automator).

  3. When Automator starts, it opens a browser window bound to your machine on port 8080 (local-only).

The top bar shows the current Automator version and connection status to Core Hub.

Step 2 – Authenticate against Core Hub

  1. In the Authentication card:

  2. Configure TLS options as needed:

    • Enable TLS for HTTPS connections.

    • Skip certificate verification for self-signed certificates in development.

  3. Click Sign in.

After successful authentication:

  • The authentication fields become read-only, and only the Sign out button and status remain visible.

  • The Import and Export cards are enabled and visible.

  • Automator discovers pipelines from Core Hub and populates the drop-down lists.

Step 3 – Back up pipelines

Automator supports two main backup modes:

  • Selective backup – export a single pipeline either as a YAML-only metadata file or as a full ZIP backup (pipeline config + agents metadata + schedules + UDF source files).

  • Bulk backup – export all pipelines at once to a ZIP archive.

Selective backup of a pipeline

Use this when you need to back up or move a specific pipeline configuration.

  1. Go to the Export (backup pipeline configuration) card.

  2. From the Pipeline drop-down, choose the pipeline you want to back up.

  3. Choose one of the following options:

    • Click Export metadata to download a table-list-style YAML file representing the selected pipeline.

    • Click Full backup to download a ZIP archive that contains:

      • A table-list-style YAML file named backup_<pipelineName>_<pipelineId>.yaml.

      • A pipeline-scoped agents-config.yaml describing agents and their configuration.

      • The UDF source files referenced by that pipeline, grouped by owning agent in folders such as udf-<agentId>/<UDF_NAME>.java.

  4. Store the resulting YAML or ZIP in version control or a secure backup location. YAML files can also be used as input for the Bootstrapper CLI if needed.

You can store this YAML file in version control or a secure backup location. It can also be used as input for the Bootstrapper CLI if needed.

Bulk backup of all pipelines

Use this when you want a snapshot of all configured pipelines on a Core Hub.

  1. Go to the Export (backup pipeline configuration) card.

  2. Click Export all metadata.

  3. Automator contacts Core Hub and exports each pipeline configuration.

  4. When ready, your browser downloads a ZIP archive that contains:

    • One table-list-style YAML file per pipeline (named backup_<name>_<pipelineId>.yaml).

    • A consolidated agents-config.yaml file that lists all exported agents and describes which agents belong to which pipeline.

    • The UDF source files referenced by any exported pipeline, grouped by owning agent in folders such as udf-<agentId>/<UDF_NAME>.java.

Store this ZIP archive in your backup system together with any related infrastructure configuration and UDF deployment artifacts.

Step 4 – Restore from a backup

Restoring from a backup means importing a previously exported YAML file into a target Core Hub.

You can restore into:

  • A freshly provisioned Core Hub (for example, when migrating to a managed Kubernetes cluster or a new region).

  • A warm standby environment that you keep updated for business continuity.

  • A lab environment where you validate pipelines before promoting them back into production.

In case of Kubernetes, before restoring, ensure that the target Core Hub has compatible agent types and versions; for Docker compose deployments this is automatically managed.
Some configuration fields may only be valid in specific Gluesync releases, so make sure to check your current version before restoring.

Restore a single pipeline

  1. Make sure you are authenticated against the target Core Hub (see Step 2).

  2. In the Import (create entities from YAML) card:

    • Under YAML configuration, click Choose file and select the previously exported YAML file.

    • Optionally adjust:

    • Pipeline (if you need to target a different pipeline identifier on the destination Core Hub).

    • Chunk size, scheduling, and Create missing tables options.

    • Leave Customize source/target schemas (override YAML) unchecked, unless you explicitly need to override schema names.

  3. Click Start entity creation.

  4. Monitor the Execution log card to follow the restore progress.

When the run completes successfully, the destination Core Hub will contain entities configured according to the imported YAML.

Restore multiple pipelines from a ZIP

Automator provides a one-click Import All operation from the ZIP archive created by Export all metadata.

  1. Make sure you are authenticated against the target Core Hub (see Step 2).

  2. Go to the Import pipelines, agents and entities card.

  3. Under Import a full backup, choose the ZIP file previously downloaded by Export all metadata or a compatible full backup.

  4. Click Import All.

When you run Import All, Automator will:

  • For each exported pipeline:

    • Create a new pipeline in the target Core Hub.

    • Bind the required agents using the consolidated agents configuration (agents-config.yaml).

    • Apply the corresponding table-list YAML to recreate entities.

  • No replication sync is started automatically; only configuration is recreated.

If a pipeline with the same name already exists on the target Core Hub, the imported pipeline will be created with a suffix such as name (restored) or name (restored 2) while still having a unique pipeline ID.

If you prefer granular control instead of restoring everything at once, you can still:

  1. Extract the ZIP archive created by Export all metadata.

  2. For each YAML file you want to restore: Repeat the steps from Restore a single pipeline.

Limitations of Automator backups

Automator focuses on pipeline configuration and does not create a complete Core Hub snapshot. The following items are not backed up:

  • Replication checkpoints

    • Current replication positions, offsets, or LSNs are not exported.

    • After restore, pipelines may start from their configured initial position, not from the last runtime checkpoint.

  • Agent settings and connection settings

    • Database credentials and secrets must be recreated or managed separately (for example via infrastructure-as-code and secret managers).

Because of these limitations, Automator-based backups are best used together with:

  • Infrastructure-as-code definitions for Core Hub deployment.

  • External backup strategies for databases or message queues.

In addition, the agents configuration exported by Export all metadata intentionally masks passwords in the hostCredentials section. When exporting, any password field is replaced with the literal value * and an inline comment:

password: *******  # original password omitted

Before importing a configuration (either config-only or full backup), you must edit the exported YAML to replace these masked passwords with real values. If any * passwords are still present when you try to import, Automator will reject the import and show an error explaining that the passwords must be filled in first.

When to use Automator for backup and restore

Automator is a good fit when you need to:

  • Capture and version control pipeline configurations.

  • Move pipelines between environments (for example from staging to production).

  • Keep a human-readable snapshot of table-list configurations used by the Bootstrapper.

For disaster recovery scenarios that require full runtime state or connection metadata, combine Automator exports with:

  • Regular database backups or snapshots.

  • Configuration management for agents and connections.

  • UDF deployment pipelines.