Node OS Upgrades
Background¶
For Upstream Kubernetes clusters provisioned and managed by the controller for bare metal and VM based environments, the user has complete control over the lifecycle mgmt and operations for the Operating System (OS) powering the control plane (master) and worker nodes of the cluster. Users are expected to keep the OS patched and updated. OS patches and updates frequently require rebooting the node.
The documentation below describes the ideal process that should be employed by users when they wish to update/upgrade the node operating system in a graceful manner. Following this process will ensure that impact to the applications operating in the Kubernetes cluster is minimized or eliminated.
Important
This process does not apply single node clusters because there is only a single node that will be patched and there will be impact due to reboot.
Steps per Node¶
Perform the following steps for every node in a sequence.
Important
If you are using any pod scheduling conditions such as "affinity/anti-affinity rules", you may run into an issue where Kubernetes may not have any nodes available to schedule the pods. These pods might remain in pending state until the rebooted node is back in service.
Step 1: Cordon Node¶
Mark the node as unscheduleable/cordon:
kubectl cordon <node1>
Step 2: Drain Node¶
Drain the node to remove all the running pods. Ensure that you ignore all the daemonsets.
kubectl drain <node1> --ignore-daemonsets
Step 3: Apply Update/Patch & Reboot¶
Update/patch the OS in the node and reboot the node. After the reboot is complete, make sure that kubelet and container runtime engine is up and running. Then verify that the node is ready.
kubectl get node <node1>
Uncordon Node¶
Uncordon the node so that pods can be scheduled on it.
kubectl uncordon <node1>
Once the node is ready and uncordoned, perform the same steps on the remaining nodes one by one.
Pre/Post Node OS Upgrade¶
Below is an example of the Node OS version before the upgrade
Users must manually update the OS of their machine and patch upgrade. Once the patch upgrade is successful, you can view the OS version upgrade on the node overview page as shown below in the below example
