Platform Configuration Backup and Restore (Disaster Recovery)

Looking to backup your database data? If you want to extract a full copy of your database records, you need to run a Snapshot Task. This page instead covers how to save the Gluesync infrastructure configuration (Pipelines, Agents, UDFs) for Disaster Recovery.

This guide describes how to use Gluesync Automator to back up and restore pipeline configurations from a running Gluesync Core Hub.

Automator provides a simple web interface on top of the Bootstrapper’s export and import capabilities, making it easy to:

  • Back up individual pipelines either as YAML metadata files or as full ZIP backups

  • Back up all pipelines in bulk as a ZIP archive (including pipeline configs, agents metadata, schedules and UDF source files)

  • Restore pipelines from existing backups

Automator backups are environment agnostic. A snapshot taken from a staging, production, or disaster recovery site can be restored into any compatible Core Hub cluster, enabling migrations, hot spares, and tenant-specific rollbacks.

When to run backups

Regular backups help in three recurring scenarios:

  • Environment cloning – export from staging or QA and restore into a fresh Core Hub to accelerate new environment bring-up.

  • Disaster recovery – keep recent backups off-site so you can rebuild the control plane in a clean cluster after outages or data loss.

  • Change control – version pipeline YAML files together with infrastructure-as-code so you can roll forward/backward confidently.

For business-critical deployments, schedule backups before large configuration changes and right after every successful deployment. Combine them with periodic restore dry-runs in a sandbox so you know disaster recovery works end to end.

Automator works at the configuration level. By default it does not capture connection-level secrets, but you can opt-in to include encrypted credentials. See Including secrets in backups for details.

Prerequisites

Before you start, ensure you have:

  • A running Gluesync Core Hub instance

  • Network connectivity from your workstation to Core Hub

  • Gluesync Automator downloaded for your platform

You can download Automator from the official distribution site:

For details on installation and UI elements, see Gluesync Automator.

Step 1 – Get Automator

  1. Download the latest Automator executable for your operating system.

  2. Start the executable:

    • On Windows and Linux: double-click the file or run it from a terminal.

    • On macOS: extract the .zip file, drag & drop the .app bundle to the Applications folder, then open it from there.

    • On Linux: run the executable from a terminal (chmod +x ./gluesync-automator then ./gluesync-automator).

  3. When Automator starts, it opens a browser window bound to your machine on port 8080 (local-only).

The top bar shows the current Automator version and connection status to Core Hub.

Step 2 – Authenticate against Core Hub

  1. In the Authentication card:

  2. Configure TLS options as needed:

    • Enable TLS for HTTPS connections.

    • Skip certificate verification for self-signed certificates in development.

  3. Click Sign in.

After successful authentication:

  • The authentication fields become read-only, and only the Sign out button and status remain visible.

  • The Import and Export cards are enabled and visible.

  • Automator discovers pipelines from Core Hub and populates the drop-down lists.

Step 3 – Back up pipelines

Automator supports two main backup modes:

  • Selective backup – export a single pipeline either as a YAML-only metadata file or as a full ZIP backup (pipeline config + agents metadata + schedules + UDF source files).

  • Bulk backup – export all pipelines at once to a ZIP archive.

Selective backup of a pipeline

Use this when you need to back up or move a specific pipeline configuration.

  1. Go to the Export (backup pipeline configuration) card.

  2. Decide whether to include secrets:

    • Leave Include secrets unchecked (default) to mask passwords with *.

    • Check Include secrets to preserve encrypted passwords in the backup (requires the same storage key on restore).

  3. From the Pipeline drop-down, choose the pipeline you want to back up.

  4. Choose one of the following options:

    • Click Export metadata to download a table-list-style YAML file representing the selected pipeline.

    • Click Full backup to download a ZIP archive that contains:

      • A table-list-style YAML file named backup_<pipelineName>_<pipelineId>.yaml.

      • A pipeline-scoped agents-config.yaml describing agents and their configuration.

      • The UDF source files referenced by that pipeline, grouped by owning agent in folders such as udf-<agentId>/<UDF_NAME>.java.

  5. Store the resulting YAML or ZIP in version control or a secure backup location. YAML files can also be used as input for the Bootstrapper CLI if needed.

You can store this YAML file in version control or a secure backup location. It can also be used as input for the Bootstrapper CLI if needed.

Bulk backup of all pipelines

Use this when you want a snapshot of all configured pipelines on a Core Hub.

  1. Go to the Export (backup pipeline configuration) card.

  2. Decide whether to include secrets:

    • Leave Include secrets unchecked (default) to mask passwords with *.

    • Check Include secrets to preserve encrypted passwords in the backup (requires the same storage key on restore).

  3. Click Export all metadata.

  4. Automator contacts Core Hub and exports each pipeline configuration.

  5. When ready, your browser downloads a ZIP archive that contains:

    • One table-list-style YAML file per pipeline (named backup_<name>_<pipelineId>.yaml).

    • A consolidated agents-config.yaml file that lists all exported agents and describes which agents belong to which pipeline.

    • The UDF source files referenced by any exported pipeline, grouped by owning agent in folders such as udf-<agentId>/<UDF_NAME>.java.

Store this ZIP archive in your backup system together with any related infrastructure configuration and UDF deployment artifacts.

Including secrets in backups

By default, Automator masks agent passwords when exporting to protect sensitive credentials. Before any import, masked passwords must be manually replaced with real values.

If you want a self-contained backup that includes agent credentials without manual editing, enable the Include secrets checkbox in the Export options fieldset before clicking any export button:

  • The checkbox appears above the pipeline selector in the Export card.

  • A hint explains: "Secrets are encrypted with AES256 using your database storage key which is saved within your bootstrap-core-hub.json under the key gluesync-storage-key."

When Include secrets is enabled:

  • Passwords in hostCredentials are preserved in their encrypted form (AES256) rather than replaced with *.

  • The backup can be restored into any Core Hub that uses the same storage key without editing credentials first.

  • This is ideal for:

    • Disaster recovery where you want a one-click restore.

    • Cloning pipelines between environments that share the same Core Hub storage key.

    • Creating fully self-contained backups that survive the restore process intact.

Backups that include secrets are only usable on Core Hub instances configured with the same AES256 storage key. Restoring into a Core Hub with a different key will fail to decrypt the credentials. In that case, you should export without secrets and manually re-enter the passwords after import.

Step 4 – Restore from a backup

Restoring from a backup means importing a previously exported YAML file into a target Core Hub.

You can restore into:

  • A freshly provisioned Core Hub (for example, when migrating to a managed Kubernetes cluster or a new region).

  • A warm standby environment that you keep updated for business continuity.

  • A lab environment where you validate pipelines before promoting them back into production.

Some configuration fields may only be valid in specific Gluesync releases, so make sure to check your current version before restoring.

Restore a single pipeline

  1. Make sure you are authenticated against the target Core Hub (see Step 2).

  2. In the Import (create entities from YAML) card:

    • Under YAML configuration, click Choose file and select the previously exported YAML file.

    • Optionally adjust:

    • Pipeline (if you need to target a different pipeline identifier on the destination Core Hub).

    • Chunk size, scheduling, and Create missing tables options.

    • Leave Customize source/target schemas (override YAML) unchecked, unless you explicitly need to override schema names.

  3. Click Start entity creation.

  4. Monitor the Execution log card to follow the restore progress.

When the run completes successfully, the destination Core Hub will contain entities configured according to the imported YAML.

Restore multiple pipelines from a ZIP

Automator provides a one-click Import All operation from the ZIP archive created by Export all metadata.

  1. Make sure you are authenticated against the target Core Hub (see Step 2).

  2. Go to the Import pipelines, agents and entities card.

  3. Under Import a full backup, choose the ZIP file previously downloaded by Export all metadata or a compatible full backup.

  4. Click Import All.

When you run Import All, Automator will:

  • For each exported pipeline:

    • Create a new pipeline in the target Core Hub.

    • Bind the required agents using the consolidated agents configuration (agents-config.yaml).

    • Apply the corresponding table-list YAML to recreate entities.

  • No replication sync is started automatically; only configuration is recreated.

If a pipeline with the same name already exists on the target Core Hub, the imported pipeline will be created with a suffix such as name (restored) or name (restored 2) while still having a unique pipeline ID.

If you prefer granular control instead of restoring everything at once, you can still:

  1. Extract the ZIP archive created by Export all metadata.

  2. For each YAML file you want to restore: Repeat the steps from Restore a single pipeline.

Limitations of Automator backups

Automator focuses on pipeline configuration and does not create a complete Core Hub snapshot. The following items are not backed up:

  • Replication checkpoints

    • Current replication positions, offsets, or LSNs are not exported.

    • After restore, pipelines may start from their configured initial position, not from the last runtime checkpoint.

  • Agent settings and connection settings (default mode)

    • By default, database credentials and secrets are masked during export. You must either:

      • Re-enter passwords manually after restore, or

      • Use the Include secrets option to preserve encrypted credentials (only works when restoring into a Core Hub with the same storage key).

Because of these limitations, Automator-based backups are best used together with:

  • Infrastructure-as-code definitions for Core Hub deployment.

  • External backup strategies for databases or message queues.

By default, the agents configuration exported by Export all metadata intentionally masks passwords in the hostCredentials section. When exporting without the Include secrets option, any password field is replaced with the literal value *:

password: *******  # original password omitted

Before importing a configuration with masked passwords, you must edit the exported YAML to replace these placeholders with real values. If any * passwords are still present when you try to import, Automator will reject the import and show an error explaining that the passwords must be filled in first.

If you enable Include secrets during export, passwords are preserved in their encrypted form (AES256) using the Core Hub’s storage key. Restoring such a backup into a Core Hub with the same storage key will automatically decrypt and apply the credentials without any manual editing. See Including secrets in backups for details.

When to use Automator for backup and restore

Automator is a good fit when you need to:

  • Capture and version control pipeline configurations.

  • Move pipelines between environments (for example from staging to production).

  • Keep a human-readable snapshot of table-list configurations used by the Bootstrapper.

For disaster recovery scenarios that require full runtime state or connection metadata, combine Automator exports with:

  • Regular database backups or snapshots.

  • Configuration management for agents and connections.

  • UDF deployment pipelines.