Troubleshoot GKO

When the Gluesync Kubernetes Operator (GKO) does not update as expected, start with the GluesyncPlatform status and events, then the GKO log. Most problems show their reason in status.lastAttempt.message.

Where to look

kubectl -n gluesync-dev get gsp
kubectl -n gluesync-dev describe gsp gluesync
kubectl -n gluesync-dev get gsp gluesync -o jsonpath='{.status.lastAttempt.message}{"\n"}'

The GKO log, with the kit:

kubectl -n gluesync-dev logs deployment/gluesync-gluesync-operator -f

With the GKO chart, the Deployment is gluesync-operator.

See Update status and history for the status fields and the events.

The Core Hub UI

Platform updates lists nothing, or reports that it cannot reach Conductor

The UI reaches GKO through Core Hub on /conductor/api. Check, in this order:

The GKO pod is Running and Ready:

kubectl -n gluesync-dev get pods -l app.kubernetes.io/name=gluesync-operator

GKO is connected to Core Hub. A log line module connection to Core Hub closed, reconnecting that repeats means it cannot connect: check spec.coreHub.url and the licence Secret in spec.coreHub.moduleAuth.licenseSecretRef.
The route exists:
```
kubectl -n gluesync-dev get ingressroute | grep conductor-api
```
With the GKO chart, set api.ingressRoute.enabled=true.
The catalog check has run: the AVAILABLE column of kubectl get gsp is filled. See AVAILABLE stays empty.

The UI answers that updates are managed in Git

The GluesyncPlatform has updatePolicy: gitops. Change spec.version in Git, see Update from Git and Argo CD, or set updatePolicy: ui.

The UI answers that the role may not manage updates

Only users with the SUPER_ADMIN or MANAGER role can start updates.

The UI answers that an update is running

Only one update runs at a time. Wait for it to finish: kubectl get gsp -w shows the phase leave Updating.

The UI stops waiting before the update ends

The UI waits up to 30 minutes for the result, then reports the update as pending. The update goes on: follow it with kubectl get gsp -w.

Refused updates

A Refused update changed no image.

Pipelines did not reach maintenance mode

The message is pipelines did not reach maintenance mode within <n>s (still draining: <pipeline ids>); no image was changed. GKO has already taken those pipelines out of maintenance mode again.

Check the listed pipelines in the Core Hub UI: a pipeline that is busy, for example with a large snapshot or a long transaction, can take longer to reach maintenance mode. Retry when they are quieter, or raise spec.maintenance.drainTimeoutSeconds.

Maintenance mode needs Core Hub access

GKO could not reach Core Hub’s REST API before the update. Check spec.coreHub.url, spec.coreHub.insecureSkipVerify, and the licence or token Secret. The Secret must be in the same namespace and its key must match the one in the GluesyncPlatform.

No workload labelled core-hub

No StatefulSet or Deployment in the namespace has the label com.molo17.conductor.type=core-hub. Add it, see Label the Gluesync workloads.

Failed updates

A Failed update rolled Core Hub back to its previous version. The message gives the reason after Core Hub <version>:.

Not ready within the maximum wait

The message is not ready within <n>s, sometimes followed by the last image pull error.

With a pull error such as a timeout: the registry, mirror or pull-through cache was too slow to serve the new image. Pull the image once through your registry so it is cached, or raise spec.stability.maxWaitSeconds, then retry.
Without a pull error: Core Hub started but did not become Ready in time. Raise spec.stability.maxWaitSeconds if Core Hub needs longer to start on your storage, and check the Core Hub log.

Image not found, unauthorized or denied

The registry refused the image, so the check failed at once.

not found or manifest unknown: the version does not exist in the registry the workload pulls from. With spec.imageRegistry, copy the image to your mirror first.
unauthorized, denied or forbidden: the workload has no valid pull secret for that registry. Add one to the Core Hub StatefulSet through the kit values, then retry.

CrashLoopBackOff or container errors

Core Hub exited on the new version. Read the events and the log of the Core Hub pod; after the rollback the pod runs the previous version, so look at the events first:

kubectl -n gluesync-dev get events --sort-by=.lastTimestamp | grep core-hub
kubectl -n gluesync-dev logs statefulset/gluesync-gluesync-core-hub --previous

Open a support request with the message and the log, see Enterprise support.

Core Hub does not come back after a rollback

The message ends with Core Hub did not come back after the rollback, and lists the pipelines left in maintenance mode. Core Hub is on its previous image but not Ready.

Check the Core Hub pod: kubectl -n gluesync-dev describe pod gluesync-gluesync-core-hub-0.
Fix the cause, for example storage or resources, until Core Hub is Ready.
Take the listed pipelines out of maintenance mode from the Core Hub UI, see Pipeline maintenance mode.

Degraded updates

Core Hub runs its new version, but something else did not complete. Read status.lastAttempt.results:

A module with ROLLED_BACK failed its stability check and runs its previous version again. Its message gives the reason, as for Core Hub. Retry later, or pin the module to a version that works in spec.modules.
An entry pipelines with not resumed: <ids> lists pipelines still in maintenance mode. Take them out of maintenance mode from the Core Hub UI.

Retry an update

GKO does not retry a Failed or Refused update by itself. It starts again when the spec changes. A new request from the Core Hub UI changes it. From the command line, set spec.requestID to any new value:

kubectl -n gluesync-dev patch gsp gluesync --type merge \
  -p "{\"spec\":{\"requestID\":\"retry-$(date +%s)\"}}"

Nothing happens after changing spec.version

The phase is still Failed or Refused for this version: the same spec was already tried. See Retry an update.
An update is already running: status.update.step is set.
The new value equals the running version: VERSION and DESIRED already match.
GKO is not running, or watches another namespace: it manages only the namespace it runs in.

A helm upgrade or a sync put the old version back

With a kit that does not include GKO, helm upgrade writes the image tags of its own values.yaml. Set them to the running versions before each upgrade, or move to a kit that includes GKO.
With Argo CD, add the ignoreDifferences and RespectIgnoreDifferences=true described in Update from Git and Argo CD.

GKO’s own update

The UI answers that GKO is restarting

GKO is changing its own image, and refuses new updates until the new pod has taken over, usually within a few seconds. Try again.

GKO was rolled back

status.lastAttempt.results shows GKO as ROLLED_BACK, and the update is Degraded: the new GKO did not come up, and the old one put its own image back. The message says why, for example CrashLoopBackOff or not ready within 180s. Look at the events of the GKO pods of the new version:

kubectl -n gluesync-dev get events --field-selector involvedObject.kind=Pod | grep operator

GKO does not try that version again until the spec changes. To retry it from the UI, update GKO again.

GKO fails after its own update

When a new GKO passes the hand-over and fails later, no other GKO pod is left to roll it back. Put the previous version back with Kubernetes:

kubectl -n gluesync-dev rollout undo deployment/gluesync-gluesync-operator
kubectl -n gluesync-dev rollout status deployment/gluesync-gluesync-operator

The Deployment is gluesync-gluesync-operator with the kit and the release name gluesync, and gluesync-operator with GKO’s chart. Then pin the GKO version that works in spec.modules, so that the next Core Hub update does not move GKO again:

kubectl -n gluesync-dev patch gsp gluesync --type merge -p '{"spec":{"modules":{"gluesync-operator":"0.2.0"}}}'

GKO is not listed in Platform updates

The GKO Deployment needs the label com.molo17.conductor.type: operator and GKO 0.2.0 or later. Upgrade once with Helm, see Upgrade GKO.

AVAILABLE stays empty

spec.offline is true: GKO never checks the catalog.
GKO cannot reach api.backoffice.molo17.com over HTTPS. Allow it in your egress rules or proxy.
The check has not run yet. Force it, see Check for updates now.

A module whose image is not in the catalog is SKIPPED during updates; it keeps its version unless you pin one in spec.modules.