{"id":1873,"date":"2022-01-05T07:05:30","date_gmt":"2022-01-05T07:05:30","guid":{"rendered":"https:\/\/blog.samarthya.me\/wps\/?p=1873"},"modified":"2022-01-05T07:05:32","modified_gmt":"2022-01-05T07:05:32","slug":"upgrading-k8s-cluster","status":"publish","type":"post","link":"https:\/\/blog.samarthya.me\/wps\/2022\/01\/05\/upgrading-k8s-cluster\/","title":{"rendered":"Upgrading K8S cluster"},"content":{"rendered":"\n<p>In this blog I will talk about how you can upgrade a node and the kubernetes cluster version to a specific version.<\/p>\n\n\n\n<p>I have a 6 node local Kubernetes cluster like below<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>k get nodes\nNAME                           STATUS   ROLES                  AGE    VERSION\nmachinedev002687.samarthya.me  Ready    &lt;none>                 41d    v1.22.4\nmachinedev003277.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachinedev003278.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachinedev003968.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachineqa003969.samarthya.me   Ready    control-plane,master   130d   v1.23.1\nmachineqa003970.samarthya.me   Ready    &lt;none>                 130d   v1.23.1<\/code><\/pre>\n\n\n\n<p>and as evident one of the node is having a different version than the others <code>1.22.4<\/code> which gave me the opportunity to apply what I learned for upgrading clusters using <code>kubeadm<\/code>. <\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>Refer to the official documentation for finding detailed steps<\/p><cite>https:\/\/kubernetes.io\/docs\/tasks\/administer-cluster\/kubeadm\/kubeadm-upgrade\/<\/cite><\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">Step 1: <code>kubeadm upgrade plan<\/code><\/h2>\n\n\n\n<p>First thing I tried as the documentation mandated was to plan for the upgrade<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>kubeadm upgrade plan<\/code><\/pre>\n\n\n\n<p>The output was loads of useful information but in particular<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;upgrade\/config] Making sure the configuration is correct:\n&#91;upgrade\/config] Reading configuration from the cluster...\n&#91;upgrade\/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'\n&#91;preflight] Running pre-flight checks.\n&#91;upgrade] Running cluster health checks\n&#91;upgrade] Fetching available versions to upgrade to\n&#91;upgrade\/versions] Cluster version: v1.22.1\n&#91;upgrade\/versions] kubeadm version: v1.23.1\n&#91;upgrade\/versions] Target version: v1.23.1\n&#91;upgrade\/versions] Latest version in the v1.22 series: v1.22.5\nComponents that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':\nCOMPONENT   CURRENT       TARGET\nkubelet     1 x v1.22.4   v1.22.5\n            5 x v1.23.1   v1.22.5\n\nUpgrade to the latest version in the v1.22 series:\n\nCOMPONENT                 CURRENT   TARGET\nkube-apiserver            v1.22.1   v1.22.5\nkube-controller-manager   v1.22.1   v1.22.5\nkube-scheduler            v1.22.1   v1.22.5\nkube-proxy                v1.22.1   v1.22.5\nCoreDNS                   v1.8.4    v1.8.6\netcd                      3.5.0-0   3.5.1-0\n\nYou can now apply the upgrade by executing the following command:\n\n        kubeadm upgrade apply v1.22.5\n\n_____________________________________________________________________\n\nComponents that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':\nCOMPONENT   CURRENT       TARGET\nkubelet     1 x v1.22.4   v1.23.1\n            5 x v1.23.1   v1.23.1\n\nUpgrade to the latest stable version:\n\nCOMPONENT                 CURRENT   TARGET\nkube-apiserver            v1.22.1   v1.23.1\nkube-controller-manager   v1.22.1   v1.23.1\nkube-scheduler            v1.22.1   v1.23.1\nkube-proxy                v1.22.1   v1.23.1\nCoreDNS                   v1.8.4    v1.8.6\netcd                      3.5.0-0   3.5.1-0\n\nYou can now apply the upgrade by executing the following command:\n\n        kubeadm upgrade apply v1.23.1\n\n_____________________________________________________________________\n\n\nThe table below shows the current state of component configs as understood by this version of kubeadm.\nConfigs that have a \"yes\" mark in the \"MANUAL UPGRADE REQUIRED\" column require manual config upgrade or\nresetting to kubeadm defaults before a successful upgrade can be performed. The version to manually\nupgrade to is denoted in the \"PREFERRED VERSION\" column.\n\nAPI GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED\nkubeproxy.config.k8s.io   v1alpha1          v1alpha1            no\nkubelet.config.k8s.io     v1beta1           v1beta1             no\n_____________________________________________________________________<\/code><\/pre>\n\n\n\n<p>As the output suggested I had two options to choose from and I went with <code>v1.23.1<\/code> and application was pretty simple<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master> kubeadm upgrade apply v1.23.1<\/code><\/pre>\n\n\n\n<p>The output was very verbose and it helped me understand what all it was doing. It is plain &amp; simple steps that it was following. (If you are trying it out in your cluster the output might be little different)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;upgrade\/config] Making sure the configuration is correct:\n&#91;upgrade\/config] Reading configuration from the cluster...\n&#91;upgrade\/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'\n&#91;preflight] Running pre-flight checks.\n&#91;upgrade] Running cluster health checks\n&#91;upgrade\/version] You have chosen to change the cluster version to \"v1.23.1\"\n&#91;upgrade\/versions] Cluster version: v1.22.1\n&#91;upgrade\/versions] kubeadm version: v1.23.1\n&#91;upgrade\/confirm] Are you sure you want to proceed with the upgrade? &#91;y\/N]: y\n&#91;upgrade\/prepull] Pulling images required for setting up a Kubernetes cluster\n&#91;upgrade\/prepull] This might take a minute or two, depending on the speed of your internet connection\n&#91;upgrade\/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'\n&#91;upgrade\/apply] Upgrading your Static Pod-hosted control plane to version \"v1.23.1\"...\nStatic pod: kube-apiserver-machineqa003969.samarthya.me hash: d4b824139c7c46f146c89d485f786b82\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\n&#91;upgrade\/etcd] Upgrading to TLS for etcd\nStatic pod: etcd-machineqa003969.samarthya.me hash: b6988bee4d945836d19e723256d23d35\n&#91;upgrade\/staticpods] Preparing for \"etcd\" upgrade\n&#91;upgrade\/staticpods] Renewing etcd-server certificate\n&#91;upgrade\/staticpods] Renewing etcd-peer certificate\n&#91;upgrade\/staticpods] Renewing etcd-healthcheck-client certificate\n&#91;upgrade\/staticpods] Moved new manifest to \"\/etc\/kubernetes\/manifests\/etcd.yaml\" and backed up old manifest to \"\/etc\/kubernetes\/tmp\/kubeadm-backup-manifests-2022-01-05-06-32-29\/etcd.yaml\"\n&#91;upgrade\/staticpods] Waiting for the kubelet to restart the component\n&#91;upgrade\/staticpods] This might take a minute or longer depending on the component\/version gap (timeout 5m0s)\nStatic pod: etcd-machineqa003969.samarthya.me hash: b6988bee4d945836d19e723256d23d35\nStatic pod: etcd-machineqa003969.samarthya.me hash: b6988bee4d945836d19e723256d23d35\nStatic pod: etcd-machineqa003969.samarthya.me hash: 0112495d26c7c568eeca87d0bd40d695\n&#91;apiclient] Found 1 Pods for label selector component=etcd\n&#91;upgrade\/staticpods] Component \"etcd\" upgraded successfully!\n&#91;upgrade\/etcd] Waiting for etcd to become available\n&#91;upgrade\/staticpods] Writing new Static Pod manifests to \"\/etc\/kubernetes\/tmp\/kubeadm-upgraded-manifests3925365806\"\n&#91;upgrade\/staticpods] Preparing for \"kube-apiserver\" upgrade\n&#91;upgrade\/staticpods] Renewing apiserver certificate\n&#91;upgrade\/staticpods] Renewing apiserver-kubelet-client certificate\n&#91;upgrade\/staticpods] Renewing front-proxy-client certificate\n&#91;upgrade\/staticpods] Renewing apiserver-etcd-client certificate\n&#91;upgrade\/staticpods] Moved new manifest to \"\/etc\/kubernetes\/manifests\/kube-apiserver.yaml\" and backed up old manifest to \"\/etc\/kubernetes\/tmp\/kubeadm-backup-manifests-2022-01-05-06-32-29\/kube-apiserver.yaml\"\n&#91;upgrade\/staticpods] Waiting for the kubelet to restart the component\n&#91;upgrade\/staticpods] This might take a minute or longer depending on the component\/version gap (timeout 5m0s)\nStatic pod: kube-apiserver-machineqa003969.samarthya.me hash: d4b824139c7c46f146c89d485f786b82\nStatic pod: kube-apiserver-machineqa003969.samarthya.me hash: d4b824139c7c46f146c89d485f786b82\nStatic pod: kube-apiserver-machineqa003969.samarthya.me hash: d4b824139c7c46f146c89d485f786b82\nStatic pod: kube-apiserver-machineqa003969.samarthya.me hash: 0cda1c6b70fd6f80c894dde50147f924\n&#91;apiclient] Found 1 Pods for label selector component=kube-apiserver\n&#91;upgrade\/staticpods] Component \"kube-apiserver\" upgraded successfully!\n&#91;upgrade\/staticpods] Preparing for \"kube-controller-manager\" upgrade\n&#91;upgrade\/staticpods] Renewing controller-manager.conf certificate\n&#91;upgrade\/staticpods] Moved new manifest to \"\/etc\/kubernetes\/manifests\/kube-controller-manager.yaml\" and backed up old manifest to \"\/etc\/kubernetes\/tmp\/kubeadm-backup-manifests-2022-01-05-06-32-29\/kube-controller-manager.yaml\"\n&#91;upgrade\/staticpods] Waiting for the kubelet to restart the component\n&#91;upgrade\/staticpods] This might take a minute or longer depending on the component\/version gap (timeout 5m0s)\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: 7b04fc1bdfe67b074090ed91e434e5e8\nStatic pod: kube-controller-manager-machineqa003969.samarthya.me hash: f2e4c6777f4c0675b3987cadd9e82c16\n&#91;apiclient] Found 1 Pods for label selector component=kube-controller-manager\n&#91;upgrade\/staticpods] Component \"kube-controller-manager\" upgraded successfully!\n&#91;upgrade\/staticpods] Preparing for \"kube-scheduler\" upgrade\n&#91;upgrade\/staticpods] Renewing scheduler.conf certificate\n&#91;upgrade\/staticpods] Moved new manifest to \"\/etc\/kubernetes\/manifests\/kube-scheduler.yaml\" and backed up old manifest to \"\/etc\/kubernetes\/tmp\/kubeadm-backup-manifests-2022-01-05-06-32-29\/kube-scheduler.yaml\"\n&#91;upgrade\/staticpods] Waiting for the kubelet to restart the component\n&#91;upgrade\/staticpods] This might take a minute or longer depending on the component\/version gap (timeout 5m0s)\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 0d03ef8ddb9b9c950d546a2993ef7aa0\nStatic pod: kube-scheduler-machineqa003969.samarthya.me hash: 15a674e5b7a122f1e55f639ad88f1cbe\n&#91;apiclient] Found 1 Pods for label selector component=kube-scheduler\n&#91;upgrade\/staticpods] Component \"kube-scheduler\" upgraded successfully!\n&#91;upgrade\/postupgrade] Applying label node-role.kubernetes.io\/control-plane='' to Nodes with label node-role.kubernetes.io\/master='' (deprecated)\n&#91;upload-config] Storing the configuration used in ConfigMap \"kubeadm-config\" in the \"kube-system\" Namespace\n&#91;kubelet] Creating a ConfigMap \"kubelet-config-1.23\" in namespace kube-system with the configuration for the kubelets in the cluster\nNOTE: The \"kubelet-config-1.23\" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just \"kubelet-config\". Kubeadm upgrade will handle this transition transparently.\n&#91;kubelet-start] Writing kubelet configuration to file \"\/var\/lib\/kubelet\/config.yaml\"\n&#91;bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes\n&#91;bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials\n&#91;bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token\n&#91;bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster\n&#91;addons] Applied essential addon: CoreDNS\n&#91;addons] Applied essential addon: kube-proxy\n\n&#91;upgrade\/successful] SUCCESS! Your cluster was upgraded to \"v1.23.1\". Enjoy!\n\n&#91;upgrade\/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.<\/code><\/pre>\n\n\n\n<p>It took me around 2 minutes for the process to complete, but eventually it was success <\/p>\n\n\n\n<p><code>SUCCESS! Your cluster was upgraded to \"v1.23.1\". Enjoy!<\/code><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Step 2: Check the nodes<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>k get nodes -o wide\nNAME                            STATUS   ROLES                  AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME\nmachinedev002687.samarthya.me   Ready    &lt;none>                 41d    v1.22.4   10.80.120.188   &lt;none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker:\/\/20.10.12\nmachinedev003277.samarthya.me   Ready    &lt;none>                 130d   v1.23.1   10.80.120.148   &lt;none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker:\/\/20.10.12\nmachinedev003278.samarthya.me   Ready    &lt;none>                 130d   v1.23.1   10.80.120.149   &lt;none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker:\/\/20.10.12\nmachinedev003968.samarthya.me   Ready    &lt;none>                 130d   v1.23.1   10.80.241.70    &lt;none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker:\/\/20.10.12\nmachineqa003969.samarthya.me    Ready    control-plane,master   130d   v1.23.1   10.80.241.78    &lt;none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker:\/\/20.10.12\nmachineqa003970.samarthya.me    Ready    &lt;none>                 130d   v1.23.1   10.80.241.80    &lt;none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker:\/\/20.10.12<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 3: Upgrade the node<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3.1 Drain the node <code>machinedev002687.samarthya.me<\/code> for upgrade <\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>k drain machinedev002687.samarthya.me --ignore-daemonsets\nnode\/machinedev002687.samarthya.me cordoned\nWARNING: ignoring DaemonSet-managed Pods: default\/kubernetes-ingress-z6m8x, ingress\/haproxy-kubernetes-ingress-6jfnn, kube-system\/calico-node-6nx7r, kube-system\/kube-proxy-n6mgh\nevicting pod kube-system\/coredns-64897985d-tvr5s\nevicting pod cert-manager\/cert-manager-webhook-9cb88bd6d-lsh8f\nevicting pod cert-manager\/cert-manager-57d89b9548-shqzc\nevicting pod ingress\/haproxy-kubernetes-ingress-default-backend-7c55f74d7f-5pcwj\nevicting pod ingress\/haproxy-kubernetes-ingress-default-backend-7c55f74d7f-7vgr8\npod\/haproxy-kubernetes-ingress-default-backend-7c55f74d7f-5pcwj evicted\npod\/cert-manager-webhook-9cb88bd6d-lsh8f evicted\npod\/cert-manager-57d89b9548-shqzc evicted\npod\/haproxy-kubernetes-ingress-default-backend-7c55f74d7f-7vgr8 evicted\npod\/coredns-64897985d-tvr5s evicted\nnode\/machinedev002687.samarthya.me drained<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3.2 Upgrade the <code>kubeadm<\/code> on the node<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@node5 ~]# yum install -y kubeadm-1.23.1 --disableexcludes=kubernetes\nLoaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager\n\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nLoading mirror speeds from cached hostfile\n * base: centos.excellmedia.net\n * epel: mirror.datto.com\n * extras: centos.excellmedia.net\n * updates: centos.excellmedia.net\nResolving Dependencies\n--> Running transaction check\n---> Package kubeadm.x86_64 0:1.22.4-0 will be updated\n---> Package kubeadm.x86_64 0:1.23.1-0 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n========================================================================================================================================================================================================================================\n Package                                                Arch                                                  Version                                                   Repository                                                 Size\n========================================================================================================================================================================================================================================\nUpdating:\n kubeadm                                                x86_64                                                1.23.1-0                                                  kubernetes                                                9.0 M\n\nTransaction Summary\n========================================================================================================================================================================================================================================\nUpgrade  1 Package\n\nTotal download size: 9.0 M\nDownloading packages:\nNo Presto metadata available for kubernetes\n0ec1322286c077c3dd975de1098d4c938b359fb59d961f0c7ce1b35bdc98a96c-kubeadm-1.23.1-0.x86_64.rpm                                                                                                                     | 9.0 MB  00:00:03     \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Updating   : kubeadm-1.23.1-0.x86_64                                                                                                                                                                                              1\/2 \n  Cleanup    : kubeadm-1.22.4-0.x86_64                                                                                                                                                                                              2\/2 \n  Verifying  : kubeadm-1.23.1-0.x86_64                                                                                                                                                                                              1\/2 \n  Verifying  : kubeadm-1.22.4-0.x86_64                                                                                                                                                                                              2\/2 \n\nUpdated:\n  kubeadm.x86_64 0:1.23.1-0                                                                                                                                                                                                             \n\nComplete!<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3.3 Upgrade <code>kubelet<\/code> &amp; <code>kubectl<\/code><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@node5 ~]# yum install -y kubelet-1.23.1 kubectl-1.23.1 --disableexcludes=kubernetes\nLoaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager\n\nThis system is not registered with an entitlement server. You can use subscription-manager to register.\n\nLoading mirror speeds from cached hostfile\n * base: centos.excellmedia.net\n * epel: mirror.datto.com\n * extras: centos.excellmedia.net\n * updates: centos.excellmedia.net\nResolving Dependencies\n--> Running transaction check\n---> Package kubectl.x86_64 0:1.22.4-0 will be updated\n---> Package kubectl.x86_64 0:1.23.1-0 will be an update\n---> Package kubelet.x86_64 0:1.22.4-0 will be updated\n---> Package kubelet.x86_64 0:1.23.1-0 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n========================================================================================================================================================================================================================================\n Package                                                Arch                                                  Version                                                   Repository                                                 Size\n========================================================================================================================================================================================================================================\nUpdating:\n kubectl                                                x86_64                                                1.23.1-0                                                  kubernetes                                                9.5 M\n kubelet                                                x86_64                                                1.23.1-0                                                  kubernetes                                                 21 M\n\nTransaction Summary\n========================================================================================================================================================================================================================================\nUpgrade  2 Packages\n\nTotal download size: 30 M\nDownloading packages:\nNo Presto metadata available for kubernetes\n(1\/2): 8d4a11b0303bf2844b69fc4740c2e2f3b14571c0965534d76589a4940b6fafb6-kubectl-1.23.1-0.x86_64.rpm                                                                                                              | 9.5 MB  00:00:03     \n(2\/2): 7a203c8509258e0c79c8c704406b2d8f7d1af8ff93eadaa76b44bb8e9f9cbabd-kubelet-1.23.1-0.x86_64.rpm                                                                                                              |  21 MB  00:00:05     \n----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nTotal                                                                                                                                                                                                   5.8 MB\/s |  30 MB  00:00:05     \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n  Updating   : kubectl-1.23.1-0.x86_64                                                                                                                                                                                              1\/4 \n  Updating   : kubelet-1.23.1-0.x86_64                                                                                                                                                                                              2\/4 \n  Cleanup    : kubectl-1.22.4-0.x86_64                                                                                                                                                                                              3\/4 \n  Cleanup    : kubelet-1.22.4-0.x86_64                                                                                                                                                                                              4\/4 \n  Verifying  : kubelet-1.23.1-0.x86_64                                                                                                                                                                                              1\/4 \n  Verifying  : kubectl-1.23.1-0.x86_64                                                                                                                                                                                              2\/4 \n  Verifying  : kubelet-1.22.4-0.x86_64                                                                                                                                                                                              3\/4 \n  Verifying  : kubectl-1.22.4-0.x86_64                                                                                                                                                                                              4\/4 \n\nUpdated:\n  kubectl.x86_64 0:1.23.1-0                                                                                          kubelet.x86_64 0:1.23.1-0                                                                                         \n\nComplete!<\/code><\/pre>\n\n\n\n<p>reload &amp; restart kubelet.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;root@node5 ~]# sudo systemctl daemon-reload\n&#91;root@node5 ~]# sudo systemctl restart kubelet<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3.4 Bring the node back online<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>kubectl uncordon machinedev002687.samarthya.me\nnode\/machinedev002687.samarthya.me uncordoned<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Step 4: Check the nodes <code>k get nodes -owide<\/code><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>root@master>k get nodes\nNAME                           STATUS   ROLES                  AGE    VERSION\nmachinedev002687.samarthya.me  Ready    &lt;none>                 41d    v1.23.1\nmachinedev003277.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachinedev003278.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachinedev003968.samarthya.me  Ready    &lt;none>                 130d   v1.23.1\nmachineqa003969.samarthya.me   Ready    control-plane,master   130d   v1.23.1\nmachineqa003970.samarthya.me   Ready    &lt;none>                 130d   v1.23.1<\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog I will talk about how you can upgrade a node and the kubernetes cluster version to a specific version. I have a 6 node local Kubernetes cluster like below and as evident one of the node is having a different version than the others 1.22.4 which gave me the opportunity to apply [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":1874,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"image","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[195,34],"tags":[18,135,220],"class_list":["post-1873","post","type-post","status-publish","format-image","has-post-thumbnail","hentry","category-kubernetes","category-technical","tag-k8s","tag-kubeadm","tag-upgrade","post_format-post-format-image"],"_links":{"self":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1873","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/comments?post=1873"}],"version-history":[{"count":0,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/posts\/1873\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media\/1874"}],"wp:attachment":[{"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/media?parent=1873"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/categories?post=1873"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.samarthya.me\/wps\/wp-json\/wp\/v2\/tags?post=1873"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}