Search docs.molo17.com
    Operational guide Deploy and run

    Restore Core Hub from a backup

    This guide shows how to put a backup made with Backups with Gluesync Connect back on the same host, or on a new machine. The restore is a command that runs while Core Hub is stopped.

    In this guide

    • Choose the backup and download it.

    • Prepare the host.

    • Run the restore with Docker or on a bare install.

    • Start Core Hub and check the result.

    • Fix the errors the command prints.

    Goal

    Bring a Core Hub installation back from an encrypted backup

    Audience

    Gluesync administrators with access to the Core Hub host

    Level

    Advanced

    Time

    20 to 60 minutes, depending on the size of the backup

    Area

    Deploy and run → Core Hub

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have the recovery key that the backup is sealed to. It starts with GSRK1-. If you rotated the key, use the key that was current when the backup was made. The key ID shown next to the backup in Gluesync Connect tells you which one;

    2. You can open the site in Gluesync Connect with a role that may download backups (owner or admin), or you already have the backup file;

    3. The host has Core Hub of the same release as the backup, or a newer one. The restore refuses a backup made by a newer release. See The release rule;

    4. You have free space in the Core Hub data folder: about twice the size of the unpacked backup, plus the size of the backup file when the command downloads it;

    5. You can stop Core Hub on the host.

    The restore replaces the databases and the configuration files of the installation. It moves the files that are in the way aside and does not delete them. Still, stop and read the next section before you run it on a host that is in use.

    Choose the case

    Case Options Notes

    Same host: Core Hub was damaged or you need an earlier state

    --force

    The data folder already holds an installation, so the restore asks for --force.

    New machine, the original is gone

    --force if Core Hub already started once on the new machine

    Install Core Hub, stop it, then restore. The restored installation keeps the identity of the original in Gluesync Connect.

    New machine, the original is still running

    --reset-connect, and --force if Core Hub already started once

    Without --reset-connect, two Core Hubs share one Gluesync Connect site and take each other’s tunnel down.

    --reset-connect clears the Gluesync Connect pairing stored in the restored database. The copy starts without a pairing and enrols as a new site when you pair it again. See When the original Core Hub still runs.

    Guide steps

    Get the backup

    UI

    1. Sign in to Gluesync Connect and open the site.

    2. Open the Backups tab.

    3. Find the backup you want. Each backup shows when it was made, its size, until when it is kept, its key and the Core Hub version.

    4. Select Download. Gluesync Connect gives you a link that is valid for 10 minutes.

    5. Either save the file to the host, or copy the link and give it to the restore command. The command downloads it itself, over HTTPS only.

    Pick the newest backup unless you want an earlier state. Gluesync Connect keeps the newest backup of a site even when it is older than your plan’s retention.
    Expected result

    You have the backup file on the host, with the extension .gsbk, or a download link that is still valid.

    Prepare the host

    CLI

    1. On a new machine, install Gluesync with the kit, in the same release as the backup or a newer one. See Install with the kit. You do not need to start it.

    2. Stop Core Hub and the other Gluesync services. On Linux: sudo gluesync stop. See Gluesync Control for Linux.

    3. Write down the address and the folder where Core Hub keeps its data. In a container that folder is /opt/gluesync/data.

    Expected result

    Core Hub is not running. Nothing else has the database files open.

    Set the recovery key

    CLI

    Pass the key through an environment variable. A key given as an argument is visible to every user of the machine.

    read -r -s -p "Recovery key: " GS_RESTORE_RECOVERY_KEY
export GS_RESTORE_RECOVERY_KEY

    The key is read in any case, with or without the dashes. If the command finds no key in the arguments or in GS_RESTORE_RECOVERY_KEY, and it runs in a terminal, it asks for the key and does not echo it.

    Expected result

    GS_RESTORE_RECOVERY_KEY is set in your shell. Close the shell when you are done.

    Run the restore with Docker

    CLI

    Use this form when Core Hub runs in a container from the Gluesync kit. The container entry point accepts the restore options. The command runs a one-off container from the Core Hub image, with the data folder of your installation mounted. Put the backup file in that folder first, so that the container sees it.

    docker run --rm \
  -v <data folder on the host>:/opt/gluesync/data \
  -e GS_RESTORE_RECOVERY_KEY \
  <Core Hub image> \
  ./entrypoint.sh --restore-backup /opt/gluesync/data/backup.gsbk --force

    <data folder on the host> is the folder that the core-hub service mounts as /opt/gluesync/data in the Compose file of your kit. <Core Hub image> is the image of that service. -e GS_RESTORE_RECOVERY_KEY passes the variable you set in the previous step, without writing the key in the command.

    To let the command download the backup, give it the link instead of the file, in quotes, and leave out the file:

    docker run --rm \
  -v <data folder on the host>:/opt/gluesync/data \
  -e GS_RESTORE_RECOVERY_KEY \
  <Core Hub image> \
  ./entrypoint.sh --restore-backup "https://<link from Gluesync Connect>" --force

    To run the restore from Compose, replace the command of the core-hub service for one run. GS_RESTORE_RECOVERY_KEY with no value passes the variable from your shell, so the key is not written in the file:

    services:
  core-hub:
    command: ["./entrypoint.sh", "--restore-backup", "/opt/gluesync/data/backup.gsbk", "--force"]
    environment:
      - GS_RESTORE_RECOVERY_KEY

    Remove the command line again before you start Gluesync.

    On Windows containers, the container entry point takes the options directly. Leave out ./entrypoint.sh and give the path inside the container, for example docker run <Core Hub image> --restore-backup C:\opt\gluesync\data\backup.gsbk.

    Add --reset-connect at the end of the options when the original Core Hub still runs. Drop --force when the data folder holds no installation.

    Run the restore on a bare install

    CLI

    Use this form when Core Hub runs directly on the host with Java 21. Run it from the folder Core Hub runs from, with the same environment variables that start Core Hub (DATA_DIR and, if you use it, COMMONS_CONFIG_DIR).

    export DATA_DIR=/opt/gluesync/data
java -jar <path to the Core Hub jar> --restore-backup ./backup.gsbk --force

    With a download link:

    java -jar <path to the Core Hub jar> --restore-backup "https://<link from Gluesync Connect>" --force

    Pass --data-dir <folder> to restore into a folder other than DATA_DIR. Add --reset-connect when the original Core Hub still runs.

    Read the result

    CLI

    The command checks the backup before it touches the installation:

    1. It decrypts the backup into a scratch folder inside the data folder.

    2. It checks that the backup comes from the same release as this Core Hub, or from an older one.

    3. It checks the size and the SHA-256 checksum of every file.

    4. It runs the SQLite integrity check on every database.

    5. Only then does it move the files that are in the way to .pre-restore-<timestamp> in the data folder, and put the restored files in their place. It removes the write-ahead log files of the databases.

    If any check fails, the installation is not changed. If placing the files fails halfway, the command puts the previous files back.

    A successful restore prints a summary and exits with status 0:

    Backup restored into /opt/gluesync/data
  taken:      <date and time> by Core Hub <version>
  files:      <number>
  database:   gluesync.db (schema <number>, integrity check passed)
  moved aside: previous files are in /opt/gluesync/data/.pre-restore-<timestamp>
Start Core Hub normally to use the restored installation.

    Lines starting with warning: can follow the file list. See Warnings after the restore.

    Exit status Meaning

    0

    The backup was restored.

    1

    The command line is wrong, or there is no usable recovery key.

    2

    The restore failed. The message after Restore failed: says why. See Troubleshooting.
    Start Core Hub and check the result

    UI

    1. Start Gluesync as you normally do. On Linux: sudo gluesync start. If the backup came from an older release, Core Hub migrates the databases at the first start.

    2. Sign in to the Control Plane with a user that existed when the backup was made. The password is the one it had at that time.

    3. Open Pipelines and check that your pipelines, agents and connections are there.

    4. Open Settings → License and check the license.

    5. Open a connection and test it.

    6. If you did not use --reset-connect, check that the Gluesync Connect tunnel is online. If you did, pair the new Core Hub with Gluesync Connect again.

    7. When you are sure about the result, delete the .pre-restore-<timestamp> folder in the data folder. It holds the files of the previous installation, with their credentials.

    Verify the result

    • Core Hub starts and you can sign in;

    • pipelines, agents, connections and user-defined functions are present;

    • the license is valid;

    • every connection test passes;

    • the Backup screen shows the state you expect: Waiting for Connect until the pairing is back, then Active.

    The restored installation is the one of the time of the backup. A change made after that time is not in it. Check each pipeline before you rely on it.

    When the original Core Hub still runs

    A Core Hub has one identity in Gluesync Connect, stored in its database. A restored copy has the same identity as the original. If both run, they take each other’s tunnel down.

    --reset-connect removes the stored pairing from the restored database. After the restore:

    • the backup settings are kept: backups stay on and the recovery key still works, because its public half is part of the restored settings;

    • the copy has no pairing, so the Backup screen shows Waiting for Connect until you pair it with Gluesync Connect;

    • the copy enrols as a new site;

    • on the free plan, only one site of the organization can back up. After pairing, the new site shows Unavailable with the site limit message while the original site holds the single place. It backs up once the original site releases the place in Gluesync Connect, or you upgrade the plan.

    The release rule

    The restore accepts a backup made by the same release or an older one. It refuses a backup made by a newer release, because its databases can use a schema this release cannot open, and Core Hub never migrates a database to an older schema.

    To restore a backup made by a newer release, install that release or a newer one first. A newer Core Hub migrates the restored databases when it starts. After that, you cannot go back to the older release with these databases.

    Warnings after the restore

    The summary can show these warnings. The restore succeeded in each case.

    Warning What to do

    The SSL keystore was restored to <path>, but the security configuration still points to <path>. Copy the file there, or change sslCertificatePath

    The keystore was restored inside the data folder and security-config.json names another path. Copy the keystore to the path in the file, or change sslCertificatePath.

    gluesync.db has a write-ahead log, which means Core Hub is running or was stopped abruptly. Make sure it is stopped before it is started again on the restored data

    Stop Core Hub before you start it again on the restored data.

    The storage key is still the default one…​

    The backup was made with the placeholder storage key. See Move to production.

    Core Hub runs in DEBUG mode…​

    The backup came from a development installation.

    The version of this Core Hub ('<version>') or of the backup ('<version>') could not be compared, so the check that refuses a backup from a newer release did not run

    A development build reports no release number. Use a Core Hub release at least as new as the backup.

    Troubleshooting

    Errors that start with Error: mean the command did not start. Exit status 1. Errors that start with Restore failed: mean the restore stopped. Exit status 2. The installation is unchanged, except for the case in Placing the files.

    The command line and the key

    Problem Possible cause What to do

    Error: unknown argument '<argument>'

    A flag the command does not know.

    Use only --restore-backup, --recovery-key, --force, --reset-connect and --data-dir.

    Error: unexpected value

    A value that no flag asked for, for example an unquoted download link split by a shell.

    Put the link in quotes. The command never prints the value, so a key pasted in the wrong place does not reach your log.

    Error: --restore-backup needs the backup file or URL

    The flag has no value.

    Give the path of the backup file, or the link.

    Error: --recovery-key needs a value, Error: --data-dir needs a value

    The flag is the last argument, or the next argument starts with --.

    Give the value.

    Error: no recovery key. Set GS_RESTORE_RECOVERY_KEY, or pass --recovery-key

    There is no key and no terminal to ask for it.

    Set GS_RESTORE_RECOVERY_KEY. With docker compose run, pass it with -e GS_RESTORE_RECOVERY_KEY.

    Error: This is not a recovery key: it must start with GSRK1-

    The text is not a recovery key.

    Copy the recovery key again. Do not use the key ID or a public key.

    Error: The recovery key is malformed: it has the wrong length or contains characters that cannot be part of it

    The key was cut or has extra characters.

    Copy the whole key. Letters I, L, O and U are not part of a key.

    Error: The recovery key is not valid: the checksum does not match. Check it for typos

    A character is wrong.

    Compare the key with your copy.

    Warning: --recovery-key is visible in the process list. Use GS_RESTORE_RECOVERY_KEY next time.

    You passed the key as an argument.

    No action for this run. Use the environment variable next time.

    Opening the backup

    Problem Possible cause What to do

    Restore failed: The backup file <path> does not exist

    Wrong path, or the file is not visible inside the container.

    Check the path. With Docker, mount the file with -v.

    Restore failed: This backup was sealed to recovery key <id>, but the key you provided is <id>. Use the recovery key that was shown when backups were enabled or last rotated at the time this backup was taken

    The key is valid but it is not the one that sealed this backup. Typical after a key rotation.

    Use the key with the first ID. The Backups tab in Gluesync Connect shows the key ID of each backup.

    Restore failed: This is not a Gluesync backup: the file does not start with the GSBK signature

    The file is not a backup, or the download saved an error page.

    Download the backup again.

    Restore failed: This backup uses format version <n>, which this Core Hub cannot read. It was probably made by a newer release

    Restore failed: This backup uses the encryption scheme '<name>', which this Core Hub cannot read

    The backup uses a newer container format.

    Restore with a newer Core Hub release.

    Restore failed: The backup header is not valid: the file is damaged

    Restore failed: The backup header has an impossible size (<n> bytes): the file is damaged

    Restore failed: The backup declares an impossible chunk size (<n>): the file is damaged

    The file is damaged.

    Download the backup again. If it fails again, choose another backup.

    Restore failed: The backup is truncated: it ends inside <part>

    Restore failed: The backup is truncated: it has no data

    Restore failed: The backup is truncated in the middle of a chunk header

    The download stopped early or the file was cut.

    Download the backup again and compare the size with the one in Gluesync Connect.

    Restore failed: Chunk <n> failed authentication: the backup is corrupted, truncated, reordered or was modified

    Restore failed: Chunk <n> has the wrong size: the backup is damaged or was modified

    Restore failed: A chunk declares an impossible size (<n> bytes): the backup is damaged or was modified

    The file is damaged or was changed after it was written.

    Download the backup again. If it fails again, choose another backup.

    Restore failed: The backup’s key wrapping does not authenticate: the file is damaged or was modified

    Restore failed: The backup carries an invalid ephemeral key: the file is damaged or was modified

    The part of the file that holds the locked data key is damaged. The recovery key is the right one.

    Download the backup again. If it fails again, choose another backup.

    Checking the contents

    Problem Possible cause What to do

    Restore failed: The backup decrypted correctly but its contents are not a valid archive: <reason>

    The archive inside the backup is damaged.

    Choose another backup.

    Restore failed: This is not a Core Hub backup: it has no manifest.json

    Restore failed: The backup manifest cannot be read: <reason>

    Restore failed: The backup manifest is implausibly large: the backup is damaged

    The file is encrypted correctly but it is not a Core Hub backup, or it is damaged.

    Choose another backup.

    Restore failed: This backup uses bundle format <n>, but this Core Hub reads format 1. Restore it with the Core Hub release that made it, or a newer one

    The backup uses a newer layout.

    Use a newer Core Hub release.

    Restore failed: This backup was taken by Core Hub <version>, which is newer than this one (<version>). Its databases may use a schema this release cannot open, and Core Hub cannot migrate downwards. Restore it with Core Hub <version> or newer

    The Core Hub on this host is older than the one that made the backup.

    Install that release or a newer one. See The release rule.

    Restore failed: <path> already holds a Core Hub installation (<files>). Restoring replaces it: pass --force to go ahead, and the existing files will be moved aside, not deleted

    The data folder has databases or a bootstrap file. This is the case on the same host, and on a new machine where Core Hub already started.

    Add --force. Check --data-dir first, so you do not replace the wrong installation.

    Restore failed: Not enough free space in <path> to restore: <n> MiB are needed besides the decrypted copy, <m> MiB are free

    The disk of the data folder is too full.

    Free space, or restore into another folder with --data-dir.

    Restore failed: <file> does not match its checksum: the backup is damaged or was modified

    Restore failed: <file> is larger than its manifest says: the backup was modified

    Restore failed: The backup is incomplete: <file> is in its manifest but not in the archive

    Restore failed: The backup archive contains entries its manifest does not list (<n>): the backup was modified

    Restore failed: The backup manifest lists the same file twice: the backup is damaged

    Restore failed: The backup archive contains the same entry twice: the backup is damaged

    Restore failed: The backup manifest has an invalid record for <file>: the backup is damaged

    The backup is damaged or was modified.

    Choose another backup. Do not try to repair the file.

    Restore failed: The backup contains an entry with an unsafe path and was refused: <path>

    Restore failed: The backup contains an entry this Core Hub does not know how to restore: <path>

    The backup holds a file the restore does not accept.

    Choose another backup, or restore with a newer Core Hub release.

    Restore failed: The backup holds no main database (gluesync.db): it cannot restore an installation

    The backup was made without the main database.

    Choose another backup.

    Restore failed: <database> in the backup is not a readable database: <reason>

    Restore failed: <database> is at schema version <n>, but the manifest says <m>: the backup is inconsistent

    Restore failed: <database> in the backup failed SQLite’s integrity check, so nothing was restored: <problems>

    Restore failed: <database> still has an unmerged write-ahead log after its check: nothing was restored

    A database in the backup is damaged. The backup was made from a damaged database, or it was damaged since.

    Choose an earlier backup. If all fail, contact MOLO17 Support.

    Restore failed: --reset-connect needs the main database, which this backup does not hold

    Restore failed: Could not clear the Connect pairing in the restored database: <reason>

    The pairing cannot be removed.

    Run the restore again without --reset-connect, and keep the original Core Hub stopped.

    Placing the files

    Problem Possible cause What to do

    Restore failed: Could not place the restored files (<reason>). The previous installation was put back where possible; anything that could not be is in <folder>

    A file could not be moved, for example for a permission problem or a full disk.

    Fix the cause named in the message. Check the folder in the message for files that were not put back, and copy them back by hand. Then run the restore again.

    Restore failed: Could not create the data directory <path>

    Restore failed: Could not create the scratch directory <path>

    The user running the command cannot write there.

    Run the command as the user that owns the data folder.

    Downloading

    Problem Possible cause What to do

    Restore failed: The download from <address> failed with HTTP 403. A presigned link is only valid for ten minutes: request a new one

    The link expired.

    Select Download again in Gluesync Connect and run the command at once.

    Restore failed: The download from <address> failed with HTTP <status>

    Gluesync Connect or the storage refused the request.

    Check that the link is complete and in quotes. Request a new one.

    Restore failed: Backups can only be downloaded over https (<address> is not): refusing to send the request

    The link starts with http://.

    Use the https:// link from Gluesync Connect.

    Restore failed: The backup address is not a valid URL

    The link is cut or has characters a shell changed.

    Copy the link again, in single quotes.

    Restore failed: Could not download the backup from <address>: <reason>

    A network error.

    Check that the host reaches the address, through your proxy or firewall. Then run the command again.

    Restore failed: The download ended after <n> of <m> bytes

    Restore failed: The download exceeded <n> bytes and was stopped

    Restore failed: The backup is <n> bytes, more than the <m> this restore accepts

    The connection broke, or the file is larger than the 32 GiB limit of the command.

    Run the command again. If it persists, download the file with a browser and restore from the file.

    Restore failed: Too many redirects while downloading the backup

    Restore failed: The server redirected to nowhere while downloading from <address>

    Restore failed: The server redirected to an invalid address

    The address does not lead to the backup.

    Request a new link.

    Restore failed unexpectedly: <class>: <message>

    Something the command did not foresee. The Core Hub log has the details.

    Contact MOLO17 Support with the message and the Core Hub log.

    Contact MOLO17 Support if you need assistance with a restore.