Disaster recovery on Kubernetes

From GKO 0.3.0, the Gluesync Kubernetes Operator (GKO) builds a disaster-recovery site the way Conductor does on Docker: a Super Admin pairs two sites from Settings → Disaster Recovery in the Core Hub UI. This page explains what GKO changes in the cluster, how to shape the Service the other site reaches, what Argo CD needs, and how to set a pair up by hand with the Kubernetes kit instead.

How a pair works, its ports, the pairing code and every GS_DR_* variable are described in Set up a disaster-recovery pair. Core Hub and Chronos replicate, fail over and fence; GKO only builds each site.

Before you begin

Make sure that:

GKO 0.3.0 or later runs in both namespaces, with the custom resource definition of its release: Helm installs a definition but never upgrades it, so on an existing install apply crds/ of the chart first, see Upgrade GKO;
Chronos keeps its data in a PersistentVolumeClaim, as the kit does by default (persistence.chronos.data.enabled), not ReadWriteOncePod;
Each site has an address that the other site can reach on ports 1718 and 1719, and you know it before you start: the pairing code carries it. Give each site a DNS name and point it at its load balancer once the load balancer has an address, or ask for a fixed address with loadBalancerIP or your provider’s annotations, see The Service the other site reaches;
No update is running, and Core Hub has no GS_DR_* variable set by hand.

Guided setup from the Core Hub UI

The steps in the UI are the ones of Set up a disaster-recovery pair: Start a new pair on the site that stays active, copy the pairing code and the operator token, then Join a pair with that code on the other site. On Kubernetes, enter in Address of this site and Address of the other site the addresses that publish ports 1718 and 1719, with port 1717, for example https://gluesync-a.example.com:1717: only the host is used.

When you select Set up or Join, GKO:

checks the namespace. If the setup cannot run, the Set up disaster recovery button is disabled and its reason is shown, for example a Chronos that keeps its data in an emptyDir;
backs up the pod templates of Core Hub and Chronos, the previous disaster-recovery Secret and whether its Service existed, in the Secret gko-disaster-recovery-backup;
writes the Secret gko-disaster-recovery with the GS_DR_* variables, GS_DR_MANAGED_BY=gko, the shared secret and the operator token, and answers the UI with the pairing code and the operator token;
two seconds later, once the answer has left through Core Hub, changes the workloads:
- Core Hub reads the Secret (envFrom) and mounts the Chronos claim at /opt/gluesync/chronos (GS_DR_CHRONOS_DIR). Started as root, as in the kit, the Core Hub image gives everything under /opt/gluesync to its user 1017; when Core Hub runs with a runAsUser, an init container that runs as root gives the Chronos data to that user;
- the readiness, liveness and startup probes of Core Hub move to GET /health on the control port 1719, because a standby Core Hub serves nothing on 1717;
- the grace period of Core Hub becomes at least 60 seconds, so that replication pushes its last writes when it stops;
- with a claim that is not ReadWriteMany, Core Hub must run on the node of Chronos;
- Chronos gets GS_DR_CONTROL_URL, the control port of its Core Hub;
- the Service gko-disaster-recovery publishes ports 1718 and 1719;
checks that Core Hub stays ready, with the stability check of updates (spec.stability). If it does not, GKO restores the backup, replaces the broken Core Hub pod, and records the failure.

Expected result

Core Hub and Chronos restart, as on Docker, and the UI asks you to sign in again. kubectl get gsp gluesync -o jsonpath='{.status.disasterRecovery}' shows the node id, the peer, the address of the Service and the outcome of the change in lastChange:

kubectl -n gluesync get gsp gluesync -o jsonpath='{.status.disasterRecovery}{"\n"}'
kubectl -n gluesync get events --field-selector involvedObject.kind=GluesyncPlatform

What GKO replaced on Core Hub is kept in the annotation gluesync.molo17.com/disaster-recovery-original of its StatefulSet, and the change in progress in the annotation gluesync.molo17.com/disaster-recovery-change of the GluesyncPlatform. While a change runs, no update starts; while an update runs, the setup is refused.

Show the pairing code, disable

Show pairing code on the first site rebuilds the code from the Secret gko-disaster-recovery. Disable disaster recovery removes exactly what the setup added and puts back what it replaced; the Secret, the Service and the backup are deleted once Core Hub runs without them. GKO never changes or disables a pair set up by hand.

Turn a pair off as described in Turn the pair off: disable the active site first. The Core Hub of a standby does not serve, and GKO, like Conductor, authenticates every call with it, so a Disable sent to the standby is refused: promote the standby and disable it from its own Control Plane, or uninstall it.

If you lose the operator token, read it from the Secret:

kubectl -n gluesync get secret gko-disaster-recovery -o jsonpath='{.data.GS_DR_ADMIN_TOKEN}' | base64 -d

A Kubernetes site and a Docker site

The pairing code is the same on Kubernetes and on Docker. A Kubernetes site joins a pair started by Conductor on Docker, and the other way round.

The Service the other site reaches

By default the Service gko-disaster-recovery is a LoadBalancer. Shape it in spec.disasterRecovery of the GluesyncPlatform, with the kit value operator.disasterRecovery or the GKO chart value platform.disasterRecovery, before you start the setup:

spec:
  disasterRecovery:
    service:
      type: LoadBalancer                 # or NodePort, or ClusterIP to publish 1718/1719 yourself
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      loadBalancerSourceRanges: ["198.51.100.0/24"]   # the other site
    # When the OTHER site publishes 1718/1719 on node ports:
    peerSink: "gluesync-b.example.com:31718"
    peerControlURL: "https://gluesync-b.example.com:31719"

Field Default Description

Field	Default	Description
`service.type`	`LoadBalancer`	`LoadBalancer`, `NodePort`, or `ClusterIP` when you publish ports 1718 and 1719 yourself, for example with a TCP route of your ingress controller.
`service.annotations`	empty	Annotations of the Service, for example an internal load balancer of your cloud.
`service.loadBalancerClass`, `service.loadBalancerIP`, `service.loadBalancerSourceRanges`	empty	For a `LoadBalancer`: its class, a fixed address where your provider supports it, and the addresses allowed in.
`service.sinkNodePort`, `service.controlNodePort`	allocated	For a `NodePort`: the node ports of 1718 and 1719.
`peerSink`, `peerControlURL`	empty	`GS_DR_PEER_SINK` and `GS_DR_PEER_CONTROL_URL`: where this site reaches the other site’s sink and control port, when the other site does not publish them on ports 1718 and 1719 of the host of its URL, for example on node ports.
`chronosDataOwner`	`true`	Only for a Core Hub with a `runAsUser`: `false` leaves out the root init container that gives the Chronos data to that user, where Pod Security forbids it. Make the data writable for that user yourself then, for example with an `fsGroup`.

service.type

LoadBalancer

LoadBalancer, NodePort, or ClusterIP when you publish ports 1718 and 1719 yourself, for example with a TCP route of your ingress controller.

service.annotations

empty

Annotations of the Service, for example an internal load balancer of your cloud.

service.loadBalancerClass, service.loadBalancerIP, service.loadBalancerSourceRanges

empty

For a LoadBalancer: its class, a fixed address where your provider supports it, and the addresses allowed in.

service.sinkNodePort, service.controlNodePort

allocated

For a NodePort: the node ports of 1718 and 1719.

peerSink, peerControlURL

empty

GS_DR_PEER_SINK and GS_DR_PEER_CONTROL_URL: where this site reaches the other site’s sink and control port, when the other site does not publish them on ports 1718 and 1719 of the host of its URL, for example on node ports.

chronosDataOwner

true

Only for a Core Hub with a runAsUser: false leaves out the root init container that gives the Chronos data to that user, where Pod Security forbids it. Make the data writable for that user yourself then, for example with an fsGroup.

A site behind a NodePort is reached on node ports, not on 1718 and 1719: set peerSink and peerControlURL on the other site.

Argo CD

GKO changes the Core Hub and Chronos StatefulSets that the kit renders. Add these to the ignoreDifferences of the Application of the kit, next to the ones of Update from Git and Argo CD, or a sync undoes the setup:

spec:
  ignoreDifferences:
    - group: apps
      kind: StatefulSet
      name: gluesync-gluesync-core-hub
      jqPathExpressions:
        - .metadata.annotations["gluesync.molo17.com/disaster-recovery-original"]
        - .spec.template.spec.containers[].envFrom
        - .spec.template.spec.containers[].volumeMounts[] | select(.name == "gko-dr-chronos")
        - .spec.template.spec.containers[].readinessProbe
        - .spec.template.spec.containers[].livenessProbe
        - .spec.template.spec.containers[].startupProbe
        - .spec.template.spec.volumes[] | select(.name == "gko-dr-chronos")
        - .spec.template.spec.terminationGracePeriodSeconds
        - .spec.template.spec.affinity
    - group: apps
      kind: StatefulSet
      name: gluesync-gluesync-chronos
      jqPathExpressions:
        - .spec.template.spec.containers[].env[] | select(.name == "GS_DR_CONTROL_URL")
    - group: gluesync.molo17.com
      kind: GluesyncPlatform
      name: gluesync
      jqPathExpressions:
        - .metadata.annotations["gluesync.molo17.com/disaster-recovery-change"]
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

The probes, the grace period and the affinity of Core Hub are then ignored also when you change them in Git. The Secrets and the Service that GKO creates are not in Git and carry no Argo CD tracking label, so Argo CD does not prune them. To keep the whole pair in Git instead, set it up by hand with the kit values below, and do not use the guided setup.

A pair set up by hand with the kit

To keep the pair in Git, or without GKO, set it in the values of the kit on each site. GKO then sees the variables and leaves them alone: the UI shows the pair as set with the GS_DR_* environment variables.

Create the Secret on both clusters, with the same shared secret and an operator token per site:

kubectl -n gluesync create secret generic gluesync-dr \
  --from-literal=shared-secret="<the shared secret, the same on both sites>" \
  --from-literal=admin-token="$(openssl rand -hex 32)"

On site A, the first and active site:

disasterRecovery:
  enabled: true
  nodeId: site-a
  peerUrl: https://gluesync-b.example.com:1717
  bootstrapActive: true        # the first start only: remove it once the pair is up
  existingSecret: gluesync-dr
  service:
    type: LoadBalancer

On site B, the same with nodeId: site-b, peerUrl: https://gluesync-a.example.com:1717, and without bootstrapActive;
Install or upgrade both sites, site A first, and check the pair as in Check the pair.

The kit then sets on Core Hub the GS_DR_* variables, ports 1718 and 1719 and the probes on /health of 1719, mounts the Chronos claim at disasterRecovery.chronosDir (under /opt/gluesync, which the Core Hub image gives to its user 1017 when it starts), keeps Core Hub on the node of Chronos unless persistence.accessMode is ReadWriteMany, sets GS_DR_CONTROL_URL on Chronos, and publishes ports 1718 and 1719 with the Service <release>-gluesync-dr. For a witness, set failoverMode: witness, witnessUrl and leaseTtlSeconds; for ports published elsewhere, peerSink and peerControlUrl; for any other variable, extraEnv.

Update a pair

While disaster recovery is enabled, Core Hub and Chronos cannot be updated from the Core Hub UI, as with Conductor: the UI shows the update as blocked. Update the sites one at a time, standby first, then switch over, see Upgrade a disaster-recovery pair.

On Kubernetes, change spec.version of the GluesyncPlatform of each site. A standby Core Hub does not answer its API, so GKO cannot put its pipelines into maintenance mode: set spec.maintenance.disabled: true on the standby before its update, and put it back afterwards.

How this is verified

The guided setup was run on a kind cluster with two sites in two namespaces, each the Kubernetes kit with Core Hub 2.3.2.1 on SSL, Chronos 0.3.4 and GKO, the Service as ClusterIP: the setup of the first site, which started active with the Chronos data shared and owned by Core Hub; the join of the second site, which started as the standby, both sites reporting PINNED, and the standby following the six databases, Chronos scheduler included; the pairing code shown again; Core Hub and Chronos updates blocked in the UI API; a second setup refused; the disable of the active site, which put both workloads back exactly as before the setup and removed the Secrets and the Service; and a setup made to fail, with Core Hub refusing its settings, rolled back in about 30 seconds. A pair set up by hand with the kit values was run on the same cluster. Load balancers, a witness, a switchover and two clusters were not part of the run.

Troubleshooting

Symptom What to do

Symptom	What to do
Set up disaster recovery is disabled with "GKO may not read PersistentVolumeClaims"	GKO updated itself to 0.3.0 but runs with the RBAC of an older chart. Upgrade the kit or the GKO chart.
"Chronos does not keep its data (/app/data) in a PersistentVolumeClaim"	Enable `persistence.chronos.data` in the kit, or mount a claim at the `DATA_DIR` of Chronos.
`lastChange` says `Failed … rolled back` with `Unschedulable` or `not ready within`	With a `ReadWriteOnce` Chronos claim, Core Hub must move to the node of Chronos; if its own volume cannot follow (a local volume), it cannot start there. Use a `ReadWriteMany` claim for the Chronos data, or keep both pods on one node.
`lastChange` says `Failed … rolled back` with `CrashLoopBackOff`	Core Hub refused its settings. Read the log of the failed pod for "Invalid disaster recovery settings", fix the request or `spec.disasterRecovery`, and set up again.
The init container `gko-dr-chronos-owner` is refused	Core Hub runs with a `runAsUser` and Pod Security forbids a root container in the namespace. Set `chronosDataOwner: false` and make the Chronos data writable for that user yourself.
The other site stays in "Waiting for the other site"	Check that ports 1718 and 1719 of each site are reachable from the other one at the host of its URL, and that `status.disasterRecovery.address` has an address.

Set up disaster recovery is disabled with "GKO may not read PersistentVolumeClaims"

GKO updated itself to 0.3.0 but runs with the RBAC of an older chart. Upgrade the kit or the GKO chart.

"Chronos does not keep its data (/app/data) in a PersistentVolumeClaim"

Enable persistence.chronos.data in the kit, or mount a claim at the DATA_DIR of Chronos.

lastChange says Failed … rolled back with Unschedulable or not ready within

With a ReadWriteOnce Chronos claim, Core Hub must move to the node of Chronos; if its own volume cannot follow (a local volume), it cannot start there. Use a ReadWriteMany claim for the Chronos data, or keep both pods on one node.

lastChange says Failed … rolled back with CrashLoopBackOff

Core Hub refused its settings. Read the log of the failed pod for "Invalid disaster recovery settings", fix the request or spec.disasterRecovery, and set up again.

The init container gko-dr-chronos-owner is refused

Core Hub runs with a runAsUser and Pod Security forbids a root container in the namespace. Set chronosDataOwner: false and make the Chronos data writable for that user yourself.

The other site stays in "Waiting for the other site"

Check that ports 1718 and 1719 of each site are reachable from the other one at the host of its URL, and that status.disasterRecovery.address has an address.