Search docs.molo17.com
    Operational guide Deploy and run

    Back up, restore and move Core Hub

    This guide shows how to protect a Gluesync installation on Linux and how to bring it back. You decide what to back up, make a cold backup of the installation folder and schedule it, restore it on the same server after a loss, restore it from a backup of the whole virtual machine (VM), and move it to a new server with a new name or address. You prove each restore with SQL: the pipelines resume and the target has no gaps.

    In this guide

    You will learn which files make up Gluesync and which you can skip, why the backup is taken with Gluesync stopped, how to back up and schedule the backup, how to restore on the same host and check that change data capture (CDC) continues without gaps, what to check after a VM-level restore, how to move to new hardware and update the database addresses, and whether the license moves with the data.

    Quick information

    Goal

    Be able to rebuild a Gluesync installation, on the same server or a new one, and prove that no change was lost

    Audience

    Gluesync administrators with sudo access to the Linux server

    Level

    Intermediate

    Time required

    About 1 hour for the first backup and restore. A move takes longer, because the new server has to download the images

    Area

    Linux server shell; Control Plane → pipeline → Edit target, Settings → License

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. Gluesync was installed with the Linux installer, so that the installation folder is /opt/gluesync and gluesync is on the PATH. See Install Gluesync on Linux and size the host;

    2. You can use sudo on the server and you have a second place to keep the backups, outside this server;

    3. You know who looks after the source and target databases. They are not part of the backup, and the source must still hold the changes made since the backup (see Check that the pipelines resumed and nothing is missing);

    4. For a move, you have a second Linux server with the same installer and your kit ID, and the databases can be reached from it.

    A backup of the installation folder holds the configuration and the position of Gluesync. It does not hold the data that Gluesync replicates, the databases, or the Docker images. It also holds secrets (the key that encrypts stored passwords, certificate passwords and the license), so keep it where only administrators can read it.

    This guide was tested on Ubuntu 24.04 and Rocky Linux 9.8 with Gluesync 2.3, on a pipeline from PostgreSQL to PostgreSQL that received inserts, updates and deletes every second while Gluesync was stopped and restored. Other sources, such as MySQL or IBM i, keep extra state in the agent folders (see Decide what to back up) and were not part of the test. Windows servers were not tested.

    Guide steps

    Decide what to back up

    CLI

    Everything that Gluesync needs lives in one folder, /opt/gluesync. It is small: 12 MB in the lab, with one pipeline and an hour of metrics. The table says what is in it.

    Path in /opt/gluesync What it holds Back up?

    data/core-hub/database/

    The databases of Core Hub, with their -wal and -shm files. gluesync.db holds pipelines, entities, connections with their encrypted passwords, users, settings and the read positions of the agents. The notifications, AI Studio, data comparison and metrics history are in the other files

    Yes

    data/core-hub/agents/<agent-id>/

    The identity of each agent (bootstrap.json) and the state that some connectors keep next to it: the MySQL reader checkpoint and the cdc_arena source cache, the IBM i journal-checkpoints and the Couchbase dcp_cache folder

    Yes, taken at the same moment as the databases

    data/core-hub/connections/<connection-id>/

    Certificates that you uploaded for connections

    Yes

    data/chronos/scheduler.db

    The schedules of Chronos

    Yes

    bootstrap-core-hub.json

    The storage key that encrypts the stored credentials, and the secret that signs API tokens. It sits in the installation folder, not in data, and must match the databases

    Yes, never lose it

    shared/

    gs-license.dat, the certificate (gluesync-cert.pem, gluesync-key.pem, gluesync.com.jks), security-config.json and logback.xml

    Yes

    docker-compose.yml, .env, proxy/, prometheus/

    The image versions, your own changes to the services, and your environment variables

    Yes

    logs/

    Logs of the services

    Skip. The restore does not need them, and Gluesync creates the folders again

    Prometheus and Grafana

    The kit keeps no data folder for them: the metrics history of Prometheus starts again after a restore

    Nothing to back up

    Docker images

    Downloaded again from Docker Hub when the services start. docker save is not a backup of your data

    Skip

    /var/lib/gluesync

    Files of Gluesync Control (installation state, previous compose files kept by updates), created by the installer

    Skip

    Write down what you have now, so that you can compare after a restore:

    sudo gluesync status
sudo du -sh /opt/gluesync /opt/gluesync/data/* /opt/gluesync/logs
sudo ls -l /opt/gluesync/shared
    Expected result

    gluesync status shows Running and 7 of 7 running. The data folder of Core Hub is a few megabytes and the logs are the largest part. gs-license.dat is in shared.

    You can also back up the configuration with Gluesync Automator (see Platform configuration backup and restore), or let Core Hub send encrypted backups to Gluesync Connect (see Backups with Gluesync Connect and Restore Core Hub from a backup). The next steps use a copy of the folder, which needs no other product and restores the exact state of the files.

    Understand why the backup is taken with Gluesync stopped

    CLI

    Core Hub keeps its databases in SQLite files, and it writes to them all the time, even when no pipeline is busy. Recent changes sit in the -wal file next to each database and reach the main file later. In the lab, gluesync.db was 0.3 MB and its -wal file was 2.4 MB. Copying a running installation gives two kinds of result:

    • Copying only gluesync.db gives a file that passes the SQLite integrity check and holds nothing: the copy had 0 pipelines, 0 connections and 0 agents, while the live installation had 1, 2 and 2. A backup tool that skips files such as -wal produces this;

    • Copying the whole database folder with tar while Gluesync runs opened correctly in 10 of 10 attempts under a write load of one change per second. That does not make it safe: the five databases are copied one after another while Core Hub writes to all of them, so the copy can differ from any moment that existed. Support recommends the opposite: stop Gluesync, copy, start.

    Maintenance mode does not help here. It stops the entities, but Core Hub keeps its databases open.

    Two supported ways take a consistent copy of a running Core Hub: the Core Hub backup to Gluesync Connect, and a VM backup that quiesces the disk (see Restore from a VM-level backup).

    Expected result

    You will stop Gluesync for the time of the copy. In the lab the whole window was 17 seconds, from the stop to the services being started again, with a 12 MB folder. Pipelines that were active resume by themselves, and the source keeps the changes that happened in the meantime.

    Make a cold backup

    CLI

    1. Create a folder for the backups that only root can read:

      sudo install -d -m 700 /var/backups/gluesync

    2. Stop Gluesync. Your data and images are kept:

      sudo gluesync stop

    3. Copy the installation folder. --numeric-owner keeps the numeric owners of the files, which Core Hub needs, and the logs are left out:

      sudo tar --numeric-owner --exclude='gluesync/logs' \
  -czf /var/backups/gluesync/gluesync-$(date +%Y%m%d-%H%M%S).tar.gz -C /opt gluesync

    4. Start Gluesync:

      sudo gluesync start

    5. Write a checksum next to the archive, and copy both to another place:

      cd /var/backups/gluesync
sudo sh -c 'sha256sum gluesync-*.tar.gz > SHA256SUMS'
ls -lh
    Expected result

    The archive exists and is about 5 MB (5.3 MB in the lab for a 12 MB folder). gluesync status shows 7 of 7 running again. The Control Plane can answer 502 Bad Gateway for up to a minute after the start (45 to 75 seconds in the lab), until Core Hub is up.

    A backup that stays on the server it protects is lost with that server. Copy the archive and its checksum to another machine or to your backup system. Verify the copy with sha256sum -c SHA256SUMS on the destination.
    Optional: schedule the backup

    CLI

    Skip this step if your backup tool or your change process already runs the commands of the previous step at a time of your choice. Otherwise, put them in a script and let systemd run it every night.

    1. Save this script as /usr/local/sbin/gluesync-backup.sh:

      #!/bin/bash
# Cold backup of the Gluesync installation folder: stop, copy, start.
set -euo pipefail

SRC=/opt/gluesync
DEST=${DEST:-/var/backups/gluesync}
KEEP_DAYS=${KEEP_DAYS:-14}

install -d -m 700 "$DEST"
stamp=$(date +%Y%m%d-%H%M%S)
archive="$DEST/gluesync-$stamp.tar.gz"

# Start Gluesync again whatever happens below.
trap 'gluesync start' EXIT

gluesync stop
tar --numeric-owner --exclude="gluesync/logs" -czf "$archive.part" -C "$(dirname "$SRC")" "$(basename "$SRC")"
chmod 600 "$archive.part"
mv "$archive.part" "$archive"
( cd "$DEST" && sha256sum "$(basename "$archive")" > "$(basename "$archive").sha256" )

find "$DEST" -name 'gluesync-*.tar.gz*' -mtime +"$KEEP_DAYS" -delete
echo "backup written: $archive ($(du -h "$archive" | cut -f1))"

    2. Make it executable and run it once by hand:

      sudo chmod 750 /usr/local/sbin/gluesync-backup.sh
sudo /usr/local/sbin/gluesync-backup.sh

    3. Create /etc/systemd/system/gluesync-backup.service:

      [Unit]
Description=Cold backup of the Gluesync installation folder

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/gluesync-backup.sh

    4. Create /etc/systemd/system/gluesync-backup.timer, with a time when nobody needs Gluesync:

      [Unit]
Description=Nightly cold backup of Gluesync

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

    5. Enable the timer, and run the service once to prove that it works unattended:

      sudo systemctl daemon-reload
sudo systemctl enable --now gluesync-backup.timer
sudo systemctl start gluesync-backup.service
systemctl list-timers gluesync-backup.timer

    6. Add a step that copies the archives to your backup destination, for example with rsync at the end of the script, and check the destination from time to time.

    Expected result

    systemctl start returns when the backup is done, the unit result is success, and a new archive with its .sha256 file is in /var/backups/gluesync. The timer shows the next run. When the copy fails, for example because the disk is full, the script ends with an error and the trap still starts Gluesync again. In the lab, with a 1 MB destination, the script ended with exit code 2 and Gluesync was running again.

    The 17-second stop is a gap in the replication, not a loss. In the lab, one pipeline received an insert every second, an update every fifth second and a delete every eleventh second, during the failing run and during the systemd run. After both runs the source and the target held the same 718 rows with the same checksum.

    Do not schedule the backup while an initial snapshot of a large table is running: stopping Gluesync interrupts it. This was not tested.
    Restore on the same server after a loss

    CLI

    Use this when the data folder of Core Hub was deleted or damaged, or when you want an earlier state. The lab loss was rm -rf /opt/gluesync/data/core-hub with traffic still arriving at the source.

    If Gluesync was started on the empty folder, it looks like a new installation: no pipelines, and the sign-in admin with the default password asks for a new one. Do not configure anything there. Restore first.

    1. Check the backup file:

      cd /var/backups/gluesync
sha256sum -c SHA256SUMS

    2. Stop Gluesync:

      sudo gluesync stop

    3. Move the broken folder aside. Do not delete it yet:

      sudo mv /opt/gluesync /opt/gluesync.broken-$(date +%Y%m%d)

    4. Extract the backup. It brings back the whole folder, including the gluesync command:

      sudo tar --numeric-owner -xzf /var/backups/gluesync/gluesync-<date-time>.tar.gz -C /opt

    5. Start Gluesync:

      sudo gluesync start
    Expected result

    gluesync status shows Running, version and 7 of 7 running, and the Control Plane answers after up to a minute. In the lab the restore took 61 seconds from gluesync stop to the first answer of the Control Plane, and the Control Plane was unavailable for about 2 minutes in total, including the empty start. Sign in with your own password: the users and passwords are the ones of the backup.

    Everything you changed in Gluesync after the backup is gone: pipelines, users, passwords and settings. In the lab, a log retention of 14 days set after the backup was back at 3 days after the restore. The source and target databases are not touched.

    The logs are not in the backup. The folders in logs are created again when the services start, and Core Hub writes to them without any change of permissions.

    If the whole server is lost, install Gluesync again as in Install Gluesync on Linux and size the host, then continue with Move Gluesync to a new server.

    Check that the pipelines resumed and nothing is missing

    UI + SQL

    Core Hub continues from the position stored in the restored database. The changes made in the source since then are read again from the log of the source, and the target is updated with the same key, so the same change is not applied twice.

    1. Open the Control Plane and sign in;

    2. Open Data pipelines, open your pipeline and check that its status is Active and that the entities show CDC active;

    3. Compare the source and the target with SQL. The query below was used with PostgreSQL. Run it on the source and on the target, with the key and a few columns of your own tables:

      select count(*) as n,
       max(id) as max_id,
       md5(string_agg(id || '|' || status || '|' || credit_limit::text, ',' order by id)) as checksum
from shop.customers;      -- on the target: shop_copy.customers

    4. If the numbers differ, list the keys that are missing. Export the keys of both tables, and compare them:

      psql -Atc "select id from shop.customers" | sort > source-ids.txt
psql -Atc "select id from shop_copy.customers" | sort > target-ids.txt
comm -23 source-ids.txt target-ids.txt     # in the source, missing in the target
comm -13 source-ids.txt target-ids.txt     # in the target, deleted from the source

    5. On a PostgreSQL source, check that Gluesync is connected to its replication slot again:

      select slot_name, active from pg_replication_slots;
    Control Plane pipeline Backup demo with the status Active
    Figure 1. After the restore, the pipeline is Active and the entity shows CDC active.
    Expected result

    The pipeline is Active, the slot is active = t, and the count, the highest key and the checksum are equal on both sides. In the lab, the source received 226 inserts (keys 30018 to 30243) between the backup and the end of the restore, with an update on every fifth insert and a delete on every eleventh, while Gluesync was running, empty and stopped. At the end both tables held 613 rows with the same checksum, with 0 keys missing and 0 extra.

    The source has to keep the changes for as long as Gluesync is down and as old as your backup. With PostgreSQL, the replication slot keeps the WAL until Gluesync has read it, so look at the disk of the database server if Gluesync stays stopped for a long time. For other sources, keep the log (binary log, archive logs, journal receivers) longer than the age of your oldest backup. See Maintenance mode for the same rule.

    Restore from a VM-level backup

    VM + CLI

    A VM backup or snapshot (Veeam, VMware, Hyper-V and others) restores the whole server at once: the installation folder, the databases that run on it, and the Docker state. The lab used a Hyper-V checkpoint of type Production as a stand-in, taken while Gluesync and both databases were running and one change per second arrived. The VM was turned off and restored 1 minute later, then started.

    1. Restore the VM from your backup and start it;

    2. Wait for the services to start by themselves. Containers have the restart policy unless-stopped, so no command is needed;

    3. Check the clock, the services and the containers:

      timedatectl | grep -E 'Local time|synchronized'
sudo gluesync status
docker ps --format '{{.Names}} {{.Status}}'

    4. Check the entities in the Control Plane, and run the comparison of Check that the pipelines resumed and nothing is missing.

    Expected result

    The clock is synchronized, all seven services of Gluesync and your database containers are running, and the pipeline is Active without any action. In the lab the source and the target were identical (734 rows, the same checksum), the replication slot was active, and new changes inserted after the restore reached the target. The changes made between the checkpoint and the restore (about 45 inserts) were gone from both databases, because both were on the VM. The PostgreSQL log reported database system was not properly shut down; automatic recovery in progress, which is normal after a restore of a running VM.

    Check these points after any VM restore:

    • Where do the databases run? If they run on the same VM, source, target and Gluesync go back to the same moment and stay consistent. If they run elsewhere, they are ahead of the restored Gluesync: treat it like Restore on the same server after a loss and check the source log retention;

    • The time. A wrong clock breaks the license check and the TLS validity. Check timedatectl;

    • The order of the start. If a database was not ready when Core Hub started, an entity can show Active while nothing replicates. Pause the entity and start it again (see What holds the old address);

    • The kind of backup. Prefer a backup that quiesces the file system. Only the Hyper-V Production checkpoint type was tested.

    Move Gluesync to a new server

    CLI

    The lab moved a running installation from an Ubuntu 24.04 server to a Rocky Linux 9.8 server. The host name, the address, the operating system and the machine ID all changed. The databases moved too, and their data folders were copied the same way.

    On the old server

    1. Write down what changes: the host name, the address, the version (sudo gluesync status), and every connection that points to a host name or an address of this server;

    2. Stop what writes to the sources if you can, and wait until the target has caught up;

    3. Make a cold backup as in the step before, and leave Gluesync stopped:

      sudo gluesync stop
sudo tar --numeric-owner --exclude='gluesync/logs' -czf gluesync-move.tar.gz -C /opt gluesync
sha256sum gluesync-move.tar.gz

    4. Copy the archive to the new server and check the checksum there. Keep the old server switched off or with Gluesync stopped from now on: only one Core Hub must read the sources and write the targets.

    On the new server

    1. Install Gluesync with the same installer and your kit ID, and let it start once. Then stop it:

      sudo gluesync stop

    2. Move the new, empty installation aside and extract your backup in its place:

      sudo mv /opt/gluesync /opt/gluesync.fresh
sudo tar --numeric-owner -xzf gluesync-move.tar.gz -C /opt

    3. Start Gluesync:

      sudo gluesync start

    4. Start your databases, if they moved too. Gluesync creates the Docker network gluesync-net when it starts, so a database that joins that network is started afterwards.

    Expected result

    gluesync status shows the version of the backup (the restored docker-compose.yml holds the image tags, so the new server downloads those images on this first start: about 2 GB, several minutes), 7 of 7 running, and the new address. In the lab a fresh install of an older version became the version of the backup, and gluesync update --list reported the version of the backup as installed and offered nothing newer. The Control Plane opens with your pipelines, users and settings. If the server cannot reach Docker Hub, use the offline package, see Offline installation.

    Keep the installation folder at the same path, /opt/gluesync: Gluesync Control records it in /var/lib/gluesync/state.json. Another path was not tested.

    Table 1. What holds the old address
    Item What to do

    Connections of the sources and targets

    The connection stores the host name or address that you typed. In the lab the database file had the old address in the target connection. Change every connection that used the old server, as in the next step. A name that also resolves on the new server (a container name on gluesync-net, a DNS name) needs no change: the source connection of the lab used a container name and was not touched

    Database firewalls and allow lists

    The databases see a new client address. Allow the new server in your firewall and in the database access rules (for PostgreSQL, pg_hba.conf) before you start

    Certificates

    The certificate of the kit has no host name in it (CN = gluesync.com, no subject alternative names), so it works on any name. If you use your own certificate, check that its names match the new address

    Settings → Global settings → Grafana URL

    It is relative by default (/grafana/d/…) and moves with the server. If you entered a full address, change it

    OpenID Connect sign-in

    If you use it, the redirect address registered at the identity provider must point to the new address (/oidc/callback). This was not tested

    Gluesync Connect

    If the installation is paired, read Restore Core Hub from a backup before you start the new server, especially when the old one still runs
    Point the connections at the new database addresses

    UI

    After the first start on the new server, the pipeline can show an error for each connection that still uses the old address.

    1. Open the Control Plane on the new server and open your pipeline in Data pipelines. If a connection points to an address that is gone, the pipeline shows On error;

    2. Select the gear icon of the pipeline to open Pipeline settings;

    3. Select Edit target (or Edit source for a source connection);

    4. In Base settings, replace the Host name / Address with the one that works from the new server, and change the Port too if it changed;

    5. Select Test and continue;

    6. Pause the entity and start it again, so that it connects with the new settings. Select Stop sync, confirm with Pause entity, then select Start sync. Keep Where it stopped, and select Start entity. See Play and pause entities.

    Pipeline header with the status On error and the message that the target agent failed during initialization and that Gluesync will try again in 5 minutes
    Figure 2. A target that still points to the old server makes the pipeline fail at the start, with a retry in five minutes.
    Pipeline settings menu with the Edit target item marked by annotation 1
    Figure 3. Open Pipeline settings and select Edit target.
    Edit target page with the Host name or Address field marked by annotation 1 and the Test and continue button marked by annotation 2
    Figure 4. Replace the host name or address, then select Test and continue.
    Sync changes dialog with the Where it stopped option marked by annotation 1
    Figure 5. Continue from the last checkpoint: nothing is lost.
    Expected result

    The pipeline is Active and the entity shows CDC active. In the lab, saving the connection changed the target to the new address but did not restart the entity: five minutes after the change the replication slot was still inactive and a row inserted in the source had not arrived. After Pause entity and Start sync the slot became active and the missing row reached the target within seconds. Run the SQL of Check that the pipelines resumed and nothing is missing. In the lab the move ended with 771 rows on both sides after a new insert/update/delete load, the same checksum, 0 missing and 0 extra.

    If you move to a server with the same name and address, nothing in this step applies: the restore alone is enough.
    Check the license

    UI + CLI

    The license is the file shared/gs-license.dat, so it is part of the backup and moves with the folder. It is a signed token that Core Hub checks on the server, with a public key inside the product. It does not contain a host name, an address or a hardware identifier: the lab license carries only the issuer, the source, a license key, the plan and the expiration date. Nothing in the license check depends on the server, which is why a VM conversion from VMware to Hyper-V, a restore or a move to new hardware does not ask for a new activation.

    1. Compare the file on the old and the new server:

      sha256sum /opt/gluesync/shared/gs-license.dat

    2. In the Control Plane, select Settings (the gear icon) → License;

    3. Compare Type and Expiration date with what you had before the move.

    Settings window on the License tab with the Type Trial and the Expiration date marked by annotation 1
    Figure 6. The license on the new server shows the same type and expiration date.
    Expected result

    The checksum is the same on both servers, and License shows the type and the date of the file. In the lab the file was valid on a server with another host name, address, machine ID and operating system, and the banner and the Core Hub log reported the same plan and expiration date as before. Core Hub reads the file again every hour.

    Keep a copy of gs-license.dat outside the server. If you lose it, ask MOLO17 for a new copy; see Licensing and license management and License management. A license that expired is not renewed by a restore: the file that you restore has the expiration date it had.

    Verify the result

    You can protect and recover Gluesync if:

    • A cold backup of /opt/gluesync (without logs) with its checksum exists, and a copy is stored outside the server;

    • The backup runs on a schedule that you checked once by hand, and Gluesync starts again even when the copy fails;

    • After a restore, the pipelines are Active, and the source and the target have the same count, highest key and checksum, with no missing keys;

    • After a move, every connection points to an address that works from the new server, and the entities were paused and started again;

    • gs-license.dat is the same file on the old and the new server, and Settings → License shows its type and date;

    • The old server does not run Gluesync any more.

    Troubleshooting

    Problem Possible cause What to do

    After the restore, gluesync status shows Partly running, 6 of 7 running and gluesync-core-hub: created, and gluesync start ends with GS-CTL-003

    bootstrap-core-hub.json is missing from the installation folder. Docker created an empty folder with that name and cannot mount it as a file

    Stop Gluesync, remove the empty folder (sudo rmdir /opt/gluesync/bootstrap-core-hub.json), put back the file from your backup, then start

    Core Hub starts, but it logs Given final block not properly padded. Such issues can arise if a bad key is used during decryption., and calls that read connections fail

    bootstrap-core-hub.json comes from another installation. Its storageKey is not the one that encrypted the databases

    Use the file from the same backup as the databases. Do not edit its keys

    The Control Plane asks for the default admin sign-in and a new password, and there are no pipelines

    Core Hub started on an empty data/core-hub: the restore did not land

    Stop Gluesync, restore as in Restore on the same server after a loss, and start again

    The Control Plane answers 502 Bad Gateway after a start

    Core Hub is still starting

    Wait up to 75 seconds, as in the lab. If it lasts, check gluesync status

    After a move, the pipeline shows On error with Something went wrong during agent … during initialization. Check if the database is reachable.

    A connection still points to the old server, or the database does not accept the new server

    Change the connection (previous step), and check the firewall and the database access rules

    The pipeline is Active but the target does not change, and the replication slot is not active

    The entity started before the database was ready, or the connection was changed without a restart

    Pause the entity and start it again with Where it stopped

    The backup script ends with an error (exit code 2 in the lab)

    The disk of the backup folder is full

    Free space and run it again. Gluesync was started again by the script, check with gluesync status

    gluesync start on the new server takes many minutes

    The server is downloading the images of the backup’s version (about 2 GB)

    Wait, or use the offline package of Offline installation

    Related guides

    Contact MOLO17 Support if you need assistance with backing up, restoring or moving Gluesync.