Search docs.molo17.com
    Operational guide Deploy and run

    Upgrade a disaster-recovery pair

    This guide shows how to move both sites of a Gluesync disaster-recovery (DR) pair to a new build without ever leaving the pair unable to fail over to a site that cannot open its databases. You upgrade the standby first, hand over to it with a planned switchover, upgrade the old site last, and know how to go back to the old build while you are still testing the new one.

    In this guide

    You will learn why a DR pair is upgraded in a fixed order, what to prepare on each site, how to upgrade the standby with Gluesync Control, how to run a planned switchover to the new build and read its verification, how long to keep the new build under observation and what that costs in protection, how to finish by upgrading the old site, how to roll back to the old build during that observation window, and what changes in witness mode.

    Quick information

    Goal

    Upgrade Core Hub and Chronos on both sites of a DR pair, keeping a site that can take over at every step that you control

    Audience

    Gluesync administrators who run a DR pair, with shell access to both hosts and the operator token

    Level

    Advanced

    Time required

    Two Gluesync Control updates, one planned switchover and the soak you choose. Plan a maintenance window for each update

    Area

    Control Plane → Settings → Disaster Recovery; Gluesync Control (gluesync update); the DR operator API on port 1719

    Applies to

    Gluesync 2.3

    Before you begin

    This guide upgrades a pair that is already set up and healthy. To pair two sites, see Set up a disaster-recovery pair.

    Make sure that:

    1. Both sites are healthy and the standby follows every database: in Settings → Disaster Recovery, the Replication table is up to date on the standby;

    2. No switchover is running;

    3. You have a backup of both sites, see Back up, restore and move Core Hub;

    4. The Core Hub service of the compose file of each site has stop_grace_period: 60s, so that replication can push its last writes when Core Hub restarts:

      services:
  <core hub service>:
    stop_grace_period: 60s

    5. You have the operator token (GS_DR_ADMIN_TOKEN) of both sites, and you can reach port 1719 of each site (the default control port) from where you run curl;

    6. You can run Gluesync Control on the host of each site, see Gluesync Control for Windows or Gluesync Control for Linux.

    Do not update Core Hub or Chronos of a site with DR enabled from the Control Plane or from Conductor. Both refuse, with the message "Disaster recovery is enabled on this Core Hub", and point to this guide. Upgrade each site with Gluesync Control on its host, one site at a time, in the order below.

    The active site is the one that serves Gluesync now, and the standby site only follows it. In the commands, replace <site>, <active site> and <old site> with the host names of your sites, and <operator token> with the operator token of the site you call.

    Guide steps

    Check the pair before you start

    UI + API

    1. Open Settings → Disaster Recovery on either site;

    2. Check that the active site and the standby site both show their build, and that the Replication table shows every database followed by the standby;

    3. Read the status of each site, and write down its build:

      curl -H "Authorization: Bearer <operator token>" http://<site>:1719/dr/status

    4. Check that no "cannot take over" notice is shown and that no switchover is running.

    Expected result

    Both sites report the same build, the standby follows all the databases (the five Core Hub databases and the Chronos database), and the status of neither site reports takeoverBlocked.

    Upgrade the standby site first

    CLI

    The standby never opens its databases while it is the standby, so an upgrade of the standby migrates nothing. The active site keeps serving, and DR protection stays complete during this step.

    1. On the host of the standby site, update Gluesync with Gluesync Control. Core Hub and Chronos are updated with it:

      sudo gluesync update

    2. Answer Yes to the confirmation, or add -y in a script;

    3. Wait until the command reports that Gluesync was updated;

    4. In Settings → Disaster Recovery, wait for the standby to follow again. A standby that cannot resume re-restores its copies from the replica by itself.

    On Windows Server, run gluesync update from an elevated terminal, without sudo.

    Expected result

    Settings → Disaster Recovery shows the standby on the new build and the active site on the old one, with a warning that the two sites run different builds ("expected only while you upgrade the pair"). Every database is followed again, and no "cannot take over" notice is shown.

    Do not go on until the standby follows every database. If it shows "cannot take over", you upgraded the wrong site first: see Troubleshooting.
    Switch over to the upgraded standby

    UI or API

    A planned switchover hands the active role from the old site to the new one without losing a write. The new site verifies every database against the old one before it takes over.

    1. On the active site, open Settings → Disaster Recovery;

    2. Select Planned switchover.

    You can do the same through the API, on the active site:

    curl -X POST -H "Authorization: Bearer <operator token>" http://<active site>:1719/dr/switchover
    Expected result

    The Last switchover verification panel lists every database as identical. The new site then starts and migrates its databases, and becomes active. The old site becomes the standby. It now shows "cannot take over", which is expected because it runs the old build, and it keeps a rollback point (see the Rollback point panel).

    From now until you upgrade the old site, DR protection is reduced. If the new active site fails, the old site cannot take over: it holds databases that the new site migrated, and its build cannot open them.
    Soak the new build

    UI

    Run the new build for as long as you need to be sure about it. Keep this window short, because the old site cannot take over while it lasts.

    1. Check your pipelines and schedules on the new active site, as you would after any upgrade;

    2. Open Settings → Disaster Recovery and check that the old site is the standby, that it shows "cannot take over", and that the Rollback point panel shows a rollback point.

    Expected result

    The new build runs your pipelines. The Rollback point panel is available, and rollback in /dr/status of the old site reports available.

    If the new build is not acceptable, go to Roll back during the soak. If it is, finish the upgrade.

    Upgrade the old site

    CLI

    1. On the host of the old site, which is now the standby, update Gluesync with Gluesync Control:

      sudo gluesync update

    2. Answer Yes to the confirmation, or add -y in a script;

    3. Wait until the command reports that Gluesync was updated;

    4. In Settings → Disaster Recovery, wait until the site follows every database.

    Expected result

    The old site follows the migrated databases, and the "cannot take over" notice is gone. Both sites run the new build and the warning about different builds disappears. DR protection is complete again.

    If you want the original site to be the active one again, run a planned switchover back, as in the previous step on the site that is active now.

    Roll back during the soak

    The site that handed over keeps its exact state at the switchover. That is the rollback point. You can use it as long as the pair has not moved on, to return to the old build. Do this only if the new build does not work for you.

    Stop the new site

    CLI

    A rollback never runs beside an active site: the old site refuses it while the other site is active.

    1. On the host of the new site, which is the active one, stop Core Hub and Chronos. Gluesync Control stops the whole installation, which includes both:

      sudo gluesync stop
    Expected result

    Neither Core Hub nor Chronos of the new site runs.

    Roll back on the old site

    API

    1. Check that the rollback point is available: rollback in /dr/status of the old site reports available and no problem;

    2. Call the rollback on the old site, which is the standby:

      curl -X POST -H "Authorization: Bearer <operator token>" http://<old site>:1719/dr/rollback
    Expected result

    The call returns 202. The old site verifies the kept copy against the digests taken at the switchover, takes over at the next epoch from it, with its own Chronos database, and becomes active. Pipelines restart from the checkpoints of the switchover, and the source changes made during the soak are replayed to the target.

    In the lab, 224 source rows were written during the soak. After the rollback, none was missing, extra or changed on the target.

    Core Hub and Chronos changes made during the soak are lost: configuration edits, new pipelines and new schedules. The pair returns to its state at the switchover.
    Bring the new site back on the old images

    CLI

    1. On the host of the new site, pin Core Hub and Chronos to the previous version with Gluesync Control, as described in Pin versions and update one service:

      sudo gluesync update --pin core-hub=<previous version> --i-understand-pinning --allow-downgrade -y
sudo gluesync update --pin gluesync-chronos=<previous version> --i-understand-pinning --allow-downgrade -y

    2. Start Gluesync:

      sudo gluesync start

    3. In Settings → Disaster Recovery, check that the site is the standby and follows every database.

    Expected result

    The new site joins as the standby on the old build, follows every database, and does not show "cannot take over". The pair is complete again, on the old build.

    When you are ready to try again, remove the pins with sudo gluesync update --unpin all and start from the first step.

    When the rollback is not available

    The Rollback point panel, and rollback in /dr/status, tell you whether the point can be used. The fields are fromEpoch, toEpoch, handedOverAtMillis, build, available and problem. A rollback is not available when:

    • The pair moved on: another failover or switchover happened since, and the point is "stale";

    • The kept copy no longer matches the digests taken at the switchover: problem says why.

    In both cases, restore from your backup instead.

    Understand the order

    A new build can migrate its databases when it starts, and a migration is one-way: an older build cannot open the databases afterwards. If you updated the active site in place, it would migrate the active databases, and the standby, still on the old build, would follow databases that it cannot open. It could not take over.

    So the standby goes first, because it never opens its databases. Then a planned switchover moves the active role to it, and the old site is upgraded last.

    Stage Active site Standby site Protection

    Before

    Old build

    Old build

    Complete

    After you upgrade the standby

    Old build

    New build

    Complete

    After the switchover, and during the soak

    New build

    Old build, "cannot take over", rollback point kept

    Reduced: the old site cannot take over until it is upgraded

    After you upgrade the old site

    New build

    New build

    Complete

    What protects you if the order is wrong

    A standby whose follower copies have a newer schema than its build supports reports "takeover blocked": takeoverBlocked in /dr/status, and the error notice "Site <id> cannot take over" in the UI. It refuses promotion (forced promotion too), automatic takeover in witness mode and planned handover, and gives the reason, for example "database gluesync.db is at schema 51, this build (…​) supports up to 50: upgrade this site before it can take over". It keeps following and does not crash-loop. Upgrading the site clears the block.

    Where to see the build and the schema

    /dr/status reports the build of the site and, for each database, a schema with name, supported and current. In the Control Plane, the Replication table of Settings → Disaster Recovery has a Schema column, and a warning when the two sites run different builds.

    The operator API

    Each site has a control port (1719 by default) with an operator API that you call with the operator token (GS_DR_ADMIN_TOKEN) as a bearer token. DR is enabled with GS_DR_ENABLED=true on the Core Hub service.

    Call Where What it does

    GET /dr/status

    Either site

    Reports the state, the build, the schema of every database, takeoverBlocked and the rollback point

    POST /dr/switchover

    The active site

    Hands the active role to the standby

    POST /dr/promote and POST /dr/promote?force=true

    The standby

    Promotes the standby. It is refused while takeover is blocked, forced or not

    POST /dr/rollback

    The standby

    Returns to the rollback point of a switchover. It is refused while the other site is active

    Witness mode

    The procedure is the same in witness mode. During the soak, if the new active site fails, the old standby does not take the lease because takeover is blocked. So there is no automatic takeover onto databases that the old site cannot open.

    A rollback in witness mode also needs the witness to grant the next epoch. Stop the new site first, so that its lease expires.

    Verify the result

    You have upgraded the pair if:

    • Both sites report the new build in /dr/status and in Settings → Disaster Recovery;

    • The standby follows every database, and the Schema column agrees on both sites;

    • No "cannot take over" notice is shown, and takeoverBlocked is not set;

    • The Last switchover verification panel showed every database as identical;

    • Your pipelines and schedules run on the active site.

    How this procedure is verified

    The procedure was run end to end in a lab with two sites, Chronos, and a PostgreSQL pipeline under load, going to a build with a schema change. The runs covered the upgrade of the standby, the switchover with every database verified, the rollback with no target row lost, a full upgrade followed by a switch back with all six databases identical, and the wrong order: with the active site updated first, the standby blocked takeover cleanly and did not restart.

    Troubleshooting

    Problem Possible cause What to do

    The update from the Control Plane or Conductor reports "Disaster recovery is enabled on this Core Hub"

    Updates of Core Hub and Chronos are refused on a site with DR enabled

    Upgrade the site with Gluesync Control on its host, in the order of this guide

    The standby shows "Site <id> cannot take over" and takeoverBlocked is set

    Its follower copies have a newer schema than its build supports. After a switchover this is expected on the old site. Before one, the active site was updated first

    Upgrade the standby with Gluesync Control. The block clears when it runs a build that supports the schema. The standby keeps following in the meantime

    POST /dr/promote?force=true is refused

    Takeover is blocked on that site, and force does not override it

    Upgrade the site first, then promote it

    The UI warns that the two sites run different builds

    You are in the middle of the upgrade

    Nothing to do. It is expected only while you upgrade the pair, and it disappears when both sites run the same build

    The rollback is refused

    The new site is still active

    Stop Core Hub and Chronos on the new site, then call /dr/rollback again

    The rollback point is not available, or problem has a text

    The pair moved on (the point is "stale"), or the kept copy no longer matches

    Restore from your backup

    In witness mode, the rollback does not start

    The witness has not granted the next epoch, because the lease of the new site has not expired

    Make sure that the new site is stopped, and wait for its lease to expire

    Related guides

    Contact MOLO17 Support if you need assistance with upgrading a disaster-recovery pair.