How an update runs

The Gluesync Kubernetes Operator (GKO) runs every update the same way. An update moves Core Hub to a new version first, behind maintenance mode and a stability check, and then moves the modules one by one. Anything that does not come up on its new version is rolled back.

Start an update

From the Core Hub UI

Open Platform updates and click Update all, or update one component. GKO writes the new version into the GluesyncPlatform and runs the update. The UI waits for the result, up to 30 minutes; a longer update goes on and its result is in the GluesyncPlatform status.

From the command line or Git

Change spec.version, or a module pin in spec.modules:

kubectl -n gluesync-dev patch gsp gluesync --type merge -p '{"spec":{"version":"2.3.2.2"}}'
kubectl -n gluesync-dev get gsp gluesync -w

With updatePolicy: gitops, this is the only way, see Update from Git and Argo CD.

Only one update runs at a time. A UI request made while an update runs is refused, unless it asks for the same update; then it waits for that one.

Steps

# Step What happens

#	Step	What happens
1	Preflight	GKO resolves the target version of each module: its pin in `spec.modules`, or else the latest version on the release channel. It checks that it can reach Core Hub. Nothing changes yet.
2	Snapshot	It records the current image of every component it will change, in `status.update.previousImages`. Rollbacks restore these images.
3	Pipelines into maintenance mode	It lists the pipelines that are configured, running and not already in maintenance mode, records them, and puts each one into maintenance mode. Then it waits until all of them report it, for up to `spec.maintenance.drainTimeoutSeconds` (600 s).
4	Core Hub	It changes the Core Hub image and runs the Stability check.
5	Resume pipelines	As soon as Core Hub passes, it takes the recorded pipelines out of maintenance mode. Pipelines that were already in maintenance mode before the update stay there.
6	Modules	It updates the modules one at a time, in name order, each with its own stability check and rollback.
7	Finish	It writes the outcome to `status.lastAttempt` and `status.history`, and records a Kubernetes event. When GKO itself has a new version, it is reported as `SCHEDULED`: GKO changes its own image only after the update has ended, see GKO’s own update.

Preflight

GKO resolves the target version of each module: its pin in spec.modules, or else the latest version on the release channel. It checks that it can reach Core Hub. Nothing changes yet.

Snapshot

It records the current image of every component it will change, in status.update.previousImages. Rollbacks restore these images.

Pipelines into maintenance mode

It lists the pipelines that are configured, running and not already in maintenance mode, records them, and puts each one into maintenance mode. Then it waits until all of them report it, for up to spec.maintenance.drainTimeoutSeconds (600 s).

Core Hub

It changes the Core Hub image and runs the Stability check.

Resume pipelines

As soon as Core Hub passes, it takes the recorded pipelines out of maintenance mode. Pipelines that were already in maintenance mode before the update stay there.

Modules

It updates the modules one at a time, in name order, each with its own stability check and rollback.

Finish

It writes the outcome to status.lastAttempt and status.history, and records a Kubernetes event. When GKO itself has a new version, it is reported as SCHEDULED: GKO changes its own image only after the update has ended, see GKO’s own update.

Steps 3 and 5 run only when Core Hub changes. An update of modules only, started by a new pin in spec.modules, skips maintenance mode.

GKO keeps the progress of the update in the GluesyncPlatform status. If GKO restarts in the middle of an update, it carries on from the last finished step, and it still resumes the pipelines it paused.

Stability check

After it changes an image, GKO waits for the component to pass:

every pod of the workload runs the new image and is Ready;
no container restarts for spec.stability.windowSeconds (20 s); a restart starts the window again;
all of this within spec.stability.maxWaitSeconds (180 s) of the image change, including the time the image takes to download.

Errors that fail at once

GKO does not wait for maxWaitSeconds when the new image cannot come up by itself. The check fails at once when a pod of the new version is waiting with one of these reasons:

CrashLoopBackOff;
InvalidImageName;
CreateContainerConfigError or CreateContainerError;
ErrImagePull or ImagePullBackOff with an error saying that the image is not found (not found, manifest unknown) or that access is refused (unauthorized, denied, forbidden).

Errors that are retried

Other image pull errors are retried until maxWaitSeconds: Kubernetes keeps pulling, and GKO keeps waiting. These are usually temporary, for example:

a registry or network timeout;
a pull-through cache or registry mirror that has not cached the new image yet and times out on the first pulls;
a dropped connection to the registry.

When maxWaitSeconds runs out first, the check fails with not ready within <n>s, followed by the last pull error.

Operator 0.1.0 treated every image pull error as final, so a single registry timeout rolled the update back. Since operator 0.1.1, only not found and access errors fail at once.

Rollback

Core Hub

When Core Hub fails its stability check, GKO:

puts the previous Core Hub image back;
deletes the Core Hub pod that is not Ready on the new image, because a StatefulSet does not replace it by itself;
waits until Core Hub is Ready on the previous version, for up to maxWaitSeconds;
takes the paused pipelines out of maintenance mode;
ends the update as Failed, with the reason and rolled back to <previous version>.

Modules are not updated after a Core Hub rollback.

If Core Hub does not come back on the previous version within maxWaitSeconds, the update ends as Failed and the pipelines stay in maintenance mode. The message lists them. See Core Hub does not come back after a rollback.

Modules

When a module fails its stability check, GKO puts its previous image back and moves on to the next module. The other modules and Core Hub keep their new versions, and the update ends as Degraded.

Results

Result Meaning

Result	Meaning
`Succeeded`	Every component runs its target version. Nothing was rolled back.
`Degraded`	Core Hub was updated, but at least one module was rolled back, or some pipelines could not be taken out of maintenance mode.
`Failed`	Core Hub did not pass its stability check and was rolled back.
`Refused`	The update did not start, and no image was changed. For example: the pipelines did not reach maintenance mode in time, GKO could not reach Core Hub, or no workload is labelled `core-hub`.

Succeeded

Every component runs its target version. Nothing was rolled back.

Degraded

Core Hub was updated, but at least one module was rolled back, or some pipelines could not be taken out of maintenance mode.

Failed

Core Hub did not pass its stability check and was rolled back.

Refused

The update did not start, and no image was changed. For example: the pipelines did not reach maintenance mode in time, GKO could not reach Core Hub, or no workload is labelled core-hub.

Each component also gets its own result in status.lastAttempt.results: OK, ERROR, ROLLED_BACK or SKIPPED. GKO itself is SCHEDULED until its own update ends, then OK, ROLLED_BACK or ERROR; a GKO rollback turns a Succeeded update into Degraded. A module is SKIPPED when the version catalog publishes no version for its image.

GKO does not retry a Failed or Refused update by itself: it waits until the spec changes. A new request from the UI changes it. From the command line, see Retry an update.

GKO’s own update

From GKO 0.2.0, GKO updates itself. Platform updates lists it as a module, with the version available on the release channel set in Core Hub, and Update all includes it.

Target version

A pin in spec.modules, under gluesync-operator or operator. The UI sets this pin when it updates GKO.
Without a pin, the latest GKO version on the release channel, whenever Core Hub is updated.

Always last

GKO never changes its own image while an update runs:

The update runs to its end: Core Hub, pipelines, modules.
GKO is reported as SCHEDULED, the result is saved, and the UI gets its answer. The UI shows GKO as updated.
GKO waits until no update has run for 15 seconds. The UI updates one component per request, so every request still runs on the current GKO, and GKO restarts after the last one.
GKO changes its own image.

While the new GKO pod starts, no update starts, and the UI gets the answer that GKO is restarting. Try again a few seconds later.

Hand-over and rollback

The new GKO pod starts next to the old one. The old pod keeps running, and stays in charge, until the new pod has been Ready for 10 seconds. Then the old pod stops, and the new one finishes the Stability check.

The old pod puts its own image back when the new pod:

is in CrashLoopBackOff, or its image cannot be pulled (not found, access refused);
is not Ready within spec.stability.maxWaitSeconds (180 s);
makes the Deployment pass its progressDeadlineSeconds (300 s).

GKO’s result is then ROLLED_BACK, and the update becomes Degraded. GKO does not try the same version again until the spec changes.

A new GKO that fails after it has taken over is the only pod left, and cannot roll itself back. Roll it back by hand, see GKO fails after its own update.

When GKO does not update itself

With updatePolicy: gitops: Git sets GKO’s image tag, see Update from Git and Argo CD.
With the self-update turned off (selfUpdate.enabled: false), see GluesyncPlatform reference.
When the GKO Deployment does not carry the label com.molo17.conductor.type: operator, as in kits older than GKO 0.2.0.

In these cases, upgrade GKO with Helm, see Upgrade GKO.

Pipeline resume

GKO retries taking each paused pipeline out of maintenance mode for up to five minutes. A pipeline it cannot resume in that time stays in maintenance mode, appears as not resumed in the results, and makes the update Degraded. Take it out of maintenance mode from the Core Hub UI, see Pipeline maintenance mode.