Upgrading Red Hat OpenShift isn’t just about running oc adm upgrade.
Behind the scenes, a carefully orchestrated process ensures your cluster stays available while everything gets updated.
In this guide, I’ll walk through:
- Real Upgrade path 4.19 to 4.20
- Handling AdminAckRequired blockers
- Setting upgrade channels properly
- Understanding how OpenShift upgrades nodes (masters + workers)
- Why Pod Disruption Budgets (PDBs) can make or break your upgrade
Pre-Upgrade:
- Check that none of the operators are "Degraded"
- Check PDB's.
- Check Failing Pods.
- Run a must gather (oc adm must-gather).
1. Check current status
#oc adm upgrade
The output will look something like:
Cluster version is 4.19.27
Upgradeable=False
Reason: AdminAckRequired
Message: The admissionregistration.k8s.io/v1beta1 group version is deprecated in 4.19 and will be removed in 4.20.Any clients using the v1beta1 version of these resources must be updated to use the corresponding v1 version instead. See https://access.redhat.com/articles/7130599 for more information.
This essentially means our upgrade, usually Redhat will compliment the message with a knowledgebase as you can see in the above output.That knowledgebase will actually give you pointers on how to unblock.
2. Acknowledgement and Approval
The knowledgebase actually asked use to patch a configmap named admin-acks.
oc -n openshift-config patch cm admin-acks \
--patch '{"data":{"ack-4.19-admissionregistration-v1beta1-api-removals-in-4.20":"true"}}' \
--type=merge
3. Setting the upgrade channel
At the time when I ran this upgrade updates were only available in the fast channel but In production consider using stable or EUS channels.
oc adm upgrade channel fast-4.20
Verify by running the "oc adm upgrade" command again. Output should show something on these lines:
Channel: fast-4.20
VERSION IMAGE
4.20.18 quay.io/openshift-release-dev/ocp-release@sha256:2dab927fd20984e247301b2483083b71f942a1f550f5d8a1db42897edc042e39
If I could explain the relevance of channels in one word:
Fast channels - Latest features but mostly used for Dev/Test environments
Stable channels - Production grade upgrades
EUS channels - LTS upgrades
4. Trigger the upgrade
oc adm upgrade --to=4.20.18
Now, this is where most articles/guides stop. Because the next steps are taken care by openshift and most Engineers leave their desk for coffee breaks here. If you ask me, this part is equally important to understand, I will briefly walk though what openshift does "Under the Hood".
4.1 Upgrades the control planes first.
Suppose you have 3 masters, It will upgrade one master at a time, so that the cluster quorum is maintained. Each master goes through the below process:
- Node gets cordoned.
- Node gets drained.
- Update is applied.
- Node is rebooted.
- API restored.
- Uncordoned.
- Moves on to the next Master.
4.2. Next it moves on the Worker nodes.
The worker nodes also go through a similar process:
- Cordon
- Drain
- Update
- Reboot
- Uncordoned
Another topic that's less spoken about when an upgrade gets stuck is PDB's. If your PDB's are too strict your upgrade gets stuck and if it's too generous, the application is impacted. Be very careful while budgeting PDB's.
And lastly, Skip the coffee breaks and monitor the upgrade:
oc get clusterversion -w
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.19.27 True True 11s Working towards 4.20.18: 10 of 959 done (1% complete)
version 4.19.27 True True 11s Working towards 4.20.18
version 4.19.27 True True 11s Working towards 4.20.18: 1 of 959 done (0% complete)
version 4.19.27 True True 11s Working towards 4.20.18: 4 of 959 done (0% complete)
version 4.19.27 True True 11s Working towards 4.20.18: 6 of 959 done (0% complete)
version 4.19.27 True True 11s Working towards 4.20.18: 8 of 959 done (0% complete)
version 4.19.27 True True 11s Working towards 4.20.18: 10 of 959 done (1% complete)
5. Verify completion
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.20.18 True False 10m Cluster version is 4.20.18
My verdict on the upgrade process is, the upgrade is straight forward only when:
- you handle any deprecations properly
- Set correct channels.
- Don't skip prechecks.
If this helped you, feel free to share or drop your upgrade experiences too!