Skip to content

Node OS Upgrades

Background

For Upstream Kubernetes clusters provisioned and managed by the controller for bare metal and VM based environments, the user has complete control over the lifecycle mgmt and operations for the Operating System (OS) powering the control plane (master) and worker nodes of the cluster. Users are expected to keep the OS patched and updated. OS patches and updates frequently require rebooting the node.

The documentation below describes the ideal process that should be employed by users when they wish to update/upgrade the node operating system in a graceful manner. Following this process will ensure that impact to the applications operating in the Kubernetes cluster is minimized or eliminated.

Important

This process does not apply single node clusters because there is only a single node that will be patched and there will be impact due to reboot.

Approaches to Node OS Upgrades

There are two main approaches to consider when upgrading the OS on nodes within your Cluster:

  • In-Place OS Upgrades Without Node Replacement
  • Node Replacement

Approach 1: In-Place OS Upgrades Without Node Replacement

This approach involves upgrading the operating system on existing nodes without replacing them.

Step 1: Cordon Node

Mark the node as unscheduleable/cordon:

kubectl cordon <node1>

Step 2: Drain Node

Drain the node to remove all the running pods. Ensure that you ignore all the daemonsets.

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Step 3: Apply Update/Patch & Reboot

Update/patch the OS in the node and reboot the node. After the reboot is complete, make sure that kubelet and container runtime engine is up and running. Then verify that the node is ready.

kubectl get node <node1>

Step 4: Uncordon Node

Uncordon the node so that pods can be scheduled on it.

kubectl uncordon <node1>

Once the node is ready and uncordoned, perform the same steps on the remaining nodes one by one.

Step 4: Pre/Post Node OS Upgrade

Below is an example of the Node OS version before the upgrade

Version Selection

Users must manually update the OS of their machine and patch upgrade. Once the patch upgrade is successful, you can view the OS version upgrade on the node overview page as shown below in the below example

Version Selection

Steps per Node

Perform the following steps for every node in a sequence.

Important

If you are using any pod scheduling conditions such as "affinity/anti-affinity rules", you may run into an issue where Kubernetes may not have any nodes available to schedule the pods. These pods might remain in pending state until the rebooted node is back in service.

Approach 2: Node Replacement

This approach involves replacing nodes with new ones running the desired operating system version.

Step 1: Add New Nodes:

Provision new nodes with the desired operating system version. Ensure these nodes join the Kubernetes cluster correctly.

Step 2: Cordon and Drain Old Nodes:

Follow the same steps as in Approach 1 to cordon and drain the old nodes.

Step 3: Delete Old Nodes:

Once the old nodes are drained, delete them from the Kubernetes cluster:

Step 3 Verify Cluster Health:

Ensure the cluster is healthy and all applications are running as expected.

By following these guidelines and carefully considering the specific requirements of your environment, you can successfully upgrade the operating system on your Kubernetes nodes while minimizing disruptions to your applications.