upgrades.md

Upgrading Kubernetes in Kubespray

Kubespray handles upgrades the same way it handles initial deployment. That is to say that each component is laid down in a fixed order.

You can also individually control versions of components by explicitly defining their versions. Here are all version vars for each component:

• docker_version
• docker_containerd_version (relevant when container_manager == docker)
• containerd_version (relevant when container_manager == containerd)
• kube_version
• etcd_version
• calico_version
• calico_cni_version
• weave_version
• flannel_version
• kubedns_version

Warning Attempting to upgrade from an older release straight to the latest release is unsupported and likely to break something

See Multiple Upgrades for how to upgrade from older Kubespray release to the latest release

Unsafe upgrade example

If you wanted to upgrade just kube_version from v1.18.10 to v1.19.7, you could deploy the following way:

ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.18.10 -e upgrade_cluster_setup=true

And then repeat with v1.19.7 as kube_version:

ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.19.7 -e upgrade_cluster_setup=true

The var -e upgrade_cluster_setup=true is needed to be set in order to migrate the deploys of e.g kube-apiserver inside the cluster immediately which is usually only done in the graceful upgrade. (Refer to #4139 and #4736)

Graceful upgrade

Kubespray also supports cordon, drain and uncordoning of nodes when performing a cluster upgrade. There is a separate playbook used for this purpose. It is important to note that upgrade-cluster.yml can only be used for upgrading an existing cluster. That means there must be at least 1 kube_control_plane already deployed.

ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.19.7

After a successful upgrade, the Server Version should be updated:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Pausing the upgrade

If you want to manually control the upgrade procedure, you can set some variables to pause the upgrade playbook. Pausing before upgrading each upgrade may be useful for inspecting pods running on that node, or performing manual actions on the node:

• upgrade_node_confirm: true - This will pause the playbook execution prior to upgrading each node. The play will resume when manually approved by typing "yes" at the terminal.
• upgrade_node_pause_seconds: 60 - This will pause the playbook execution for 60 seconds prior to upgrading each node. The play will resume automatically after 60 seconds.

Pausing after upgrading each node may be useful for rebooting the node to apply kernel updates, or testing the still-cordoned node:

• upgrade_node_post_upgrade_confirm: true - This will pause the playbook execution after upgrading each node, but before the node is uncordoned. The play will resume when manually approved by typing "yes" at the terminal.
• upgrade_node_post_upgrade_pause_seconds: 60 - This will pause the playbook execution for 60 seconds after upgrading each node, but before the node is uncordoned. The play will resume automatically after 60 seconds.

Node-based upgrade

If you don't want to upgrade all nodes in one run, you can use --limit patterns.

Before using --limit run playbook facts.yml without the limit to refresh facts cache for all nodes:

ansible-playbook facts.yml -b -i inventory/sample/hosts.ini

After this upgrade control plane and etcd groups #5147:

ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "kube_control_plane:etcd"

Now you can upgrade other nodes in any order and quantity:

ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node4:node6:node7:node12"
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node5*"