Upgrade a Kubernetes cluster on Destination Earth OpenStack Magnum
You can upgrade a Destination Earth OpenStack Magnum cluster to the next minor Kubernetes version when compatible templates are available.
This article shows how to upgrade a Magnum Kubernetes cluster from 1.30 to 1.31.
What we are going to cover
How the Magnum rolling upgrade works
The target cluster template specifies the Kubernetes version for the upgrade. After you select the template, Magnum upgrades the cluster nodes through its rolling upgrade mechanism. For a supported pair of consecutive Kubernetes versions, Magnum:
Upgrades the control plane nodes first.
Upgrades worker nodes one by one to minimize downtime.
Adds an extra node before upgrading each node (in addition to the nodes you specified in the cluster configuration).
Maintains API compatibility during the upgrade and thus ensures that existing workloads continue running smoothly.
It is not possible to skip minor versions when upgrading.
Prerequisites
No. 1 Hosting
You need a Destination Earth hosting account with Horizon interface https://cloud.central.data.destination-earth.eu.
No. 2 Access to the cloud and cluster
Commands openstack and kubectl must be up and running:
How To Install OpenStack and Magnum Clients for Command Line Interface to Destination Earth Horizon
How To Access Kubernetes Cluster Post Deployment Using Kubectl OpenStack Magnum
No. 3 Availability of upgradeable cluster templates
There is a direct correspondence between cluster templates available on Destination Earth and the Kubernetes versions with the identical minor version numbers:
Upgradeable Kubernetes versions |
calico cluster template |
cilium cluster template |
|---|---|---|
1.30 |
k8s-v1.30.10-1.0.0 |
k8s-v1.30.10-1.0.0-cilium |
1.31 |
k8s-v1.31.10-1.0.0 |
k8s-v1.31.10-1.0.0-cilium |
The upgrade path used in this article is 1.30 to 1.31.
To test the upgrade, you can create a new cluster with the Calico template for Kubernetes 1.30. The command may look like this:
openstack coe cluster create \
--cluster-template k8s-v1.30.10-1.0.0 \
--docker-volume-size 50 \
--labels eodata_access_enabled=false,floating-ip-enabled=true \
--merge-labels \
--keypair sshkey \
--master-count 3 \
--node-count 5 \
--timeout 190 \
--master-flavor eo2a.large \
--flavor eo2a.medium \
eumetsat-automatic-upgrade
No. 4 Ensuring apps compatibility with target K8s version
In this particular case, check Kubernetes 1.31 Release Notes to ensure your applications are compatible. Be sure to always check release notes for the target Kubernetes version you are upgrading the cluster to.
Backup and observe the state of the cluster before the upgrade
Before the upgrade, compare the state of the cluster
before,
during and
after the upgrade.
Ideally, everything should work right out of the box, however, it is preferable to check and verify.
Kubernetes cluster backup
It is highly recommended to make a backup of your 1.30 cluster before the update.
This backup will serve as the state “before” the update.
Monitor the upgrade process with cluster dashboard
The simplest way to observe a cluster is through cluster dashboard. See Using Dashboard To Access Kubernetes Cluster Post Deployment OpenStack Magnum.
To see node versions, select option Nodes in the left side menu, click on a node name and scroll down a bit to find kubelet version.
Other tools for comparisons of clusters might include CI/CD tests, observing cluster with Prometheus and Grafana, storing cluster statistics in a database as a time series and so on.
Prepare the upgrade
Verify cluster version
The following command prints cluster version:
echo $(openstack coe cluster show eumetsat-automatic-upgrade | awk '/ coe_version /{print $4}')
In this article, the version before the update is 1.30.10.
Identify cluster ID
Retrieve the ID of the Kubernetes cluster you want to upgrade:
openstack coe cluster list
Record the cluster ID shown in the uuid column. In commands that follow, you can use either the cluster name or its uuid.
Trigger the upgrade
The command to initiate the upgrade is:
openstack coe cluster upgrade <cluster-id> <template-version>
For this example:
openstack coe cluster upgrade eumetsat-automatic-upgrade k8s-v1.31.10-1.0.0
Because the eumetsat-automatic-upgrade cluster uses the k8s-v1.30.10-1.0.0 template, you must upgrade it with the k8s-v1.31.10-1.0.0 template listed in Prerequisites No. 3.
The command should confirm that the request to upgrade the cluster has been accepted. Conversely, an invalid template name results in an HTTP 404 error.
Commands to monitor the upgrade progress
A GUI way of monitoring the upgrade process would be using the dashboard for the cluster, as mentioned above. A CLI way would be to issue the specific commands, for example:
Show only the version of the cluster
echo $(openstack coe cluster show eumetsat-automatic-upgrade | awk '/ coe_version /{print $4}')
During the upgrade, health_status field might temporarily get into UNHEALTHY status, until the upgrade stops. Regardless of that, the cluster remains responsive throughout the entire upgrade process.
Monitor the nodes
kubectl get nodes -o wide
The following observed upgrade from Kubernetes v1.30.10 to v1.31.10 shows the important stages of the rolling replacement process. The kubectl excerpts show only the node name, status, role, and Kubernetes version from the full output.
Before the upgrade: The worker and control-plane nodes are Ready and run v1.30.10.
NAME STATUS ROLES VERSION kube-23fn7-default-worker-gmvfx-kpps2-6vc98 Ready worker v1.30.10 kube-23fn7-zvcpq-gqnpx Ready control-plane,master v1.30.10
Upgrade request accepted: The existing nodes initially remain Ready on v1.30.10. Magnum performs the accepted upgrade asynchronously, so node replacement may not be visible immediately.
$ openstack coe cluster upgrade eumetsat-automatic-upgrade k8s-v1.31.10-1.0.0 Request to upgrade cluster eumetsat-automatic-upgrade has been accepted.
New control plane created: A second control-plane node appears as NotReady on v1.31.10. The original control-plane node remains Ready on v1.30.10.
NAME STATUS ROLES VERSION kube-23fn7-default-worker-gmvfx-kpps2-6vc98 Ready worker v1.30.10 kube-23fn7-zvcpq-gqnpx Ready control-plane,master v1.30.10 kube-23fn7-zvcpq-qddr7 NotReady control-plane,master v1.31.10
New control plane ready: Both control-plane nodes are Ready, but temporarily run different Kubernetes versions.
NAME STATUS ROLES VERSION kube-23fn7-default-worker-gmvfx-kpps2-6vc98 Ready worker v1.30.10 kube-23fn7-zvcpq-gqnpx Ready control-plane,master v1.30.10 kube-23fn7-zvcpq-qddr7 Ready control-plane,master v1.31.10
Original control plane cordoned: The original control-plane node changes to Ready,SchedulingDisabled while the replacement remains Ready. Kubernetes stops scheduling new workloads on the original node before Magnum removes it.
NAME STATUS ROLES VERSION kube-23fn7-zvcpq-gqnpx Ready,SchedulingDisabled control-plane,master v1.30.10 kube-23fn7-zvcpq-qddr7 Ready control-plane,master v1.31.10
Upgrade complete: The original nodes are no longer listed. The replacement control-plane and worker nodes are Ready on v1.31.10.
NAME STATUS ROLES VERSION kube-23fn7-default-worker-gmvfx-6q6ts-cm6h5 Ready worker v1.31.10 kube-23fn7-zvcpq-qddr7 Ready control-plane,master v1.31.10
The crucial change is that Magnum replaces the nodes instead of upgrading them in place. The node names and internal IP addresses therefore change during the upgrade.
In this observation, the node image changed from Ubuntu 22.04.5 LTS with kernel 5.15.0-134-generic to Ubuntu 24.04.2 LTS with kernel 6.8.0-63-generic. The container runtime remained containerd 1.7.20.
Continue monitoring until every intended node is Ready and reports the target Kubernetes version. For the v1.30.10 to v1.31.10 example used in this article, every node must report v1.31.10.
As a rule of thumb, you can assume that each node will take a few minutes to upgrade. The actual time will depend on way too many factors to list and that consideration is out of scope of this article.
Verify the upgrade
If you performed both
the backup of the old cluster and
upgraded to the next minor Kubernetes version,
you will have two clusters and be able to compare them.
Use the dashboard to compare the results “before” and “after”. Hunt down the eventual differences, learn what they mean and, if needed, resolve them.
If something is still not right, examine whether there are breaking changes in code from version 1.30 to version 1.31.