Cluster Upgrades#

The multi-node notes ended with a one-liner: “upgrade the control plane first, one minor version at a time”. That’s the policy - this is the practice.

Upgrading a cluster is the most common maintenance task you will ever run, and the most common way to break one. The rules that keep it boring:

  • One minor version at a time. 1.30 → 1.31, never 1.30 → 1.32. The version skew policy allows the apiserver to lead kubelets by one minor; anything more is unsupported and the upgrade will refuse
  • Control plane first, workers after. The new apiserver must understand old kubelets, not the reverse
  • One node at a time. Drain → upgrade → uncordon → next
  • Read the output. kubeadm upgrade plan exists precisely because upgrades are not a single command

The apt packages are version-pinned with apt-mark hold from the install notes. Every upgrade starts with apt-mark unhold and ends with apt-mark hold again - forgetting the re-hold is how a future apt upgrade silently moves your cluster a minor version.

etcd Backup and Restore#

Before any upgrade - before any anything risky - take a backup. The multi-node notes covered snapshot save; this is the full cycle, because a backup you cannot restore is a hope, not a plan.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save /var/backups/etcd-$(date +%F).db

Verify the snapshot is usable now, not on the day you need it:

etcdctl snapshot status /var/backups/etcd-2025-01-01.db -w table

Dikkat: snapshot restore, çalışan etcd’nin mevcut verisinin üstüne backup’ı kopyalamak değildir.

Restore işlemi snapshot’tan yeni bir etcd data dir oluşturur; ardından etcd bu yeni dizin kullanılarak ayağa kaldırılır.

snapshot.db
     ↓
restore
     ↓
yeni data dir
     ↓
etcd bu dizinden başlar

Yani mevcut /var/lib/etcd dizininin içine restore yapmaya çalışma. Önce eski data dir’i güvenli şekilde kenara al, restore’u ayrı ve temiz bir dizine yap, ardından etcd’yi restore edilen dizine yönlendir.

Kısacası:

snapshot restore != mevcut etcd'nin üstüne yazmak
snapshot restore  = snapshot'tan yeni bir etcd data dir üretmek

Eski data dir’i korumadan doğrudan restore işlemine girişmek, recovery yapmak yerine recovery senaryosunun kendisine dönüşebilir.

Certificate Management#

kubeadm clusters issue their own certificates with a one-year expiry. Nobody remembers this until the cluster stops serving traffic at 9am on a random Tuesday.

kubeadm certs check-expiration
kubeadm certs renew all

Renewing writes new certs to /etc/kubernetes/pki - the static pods must then be restarted to pick them up. The fastest way is to touch the manifest files and let the kubelet re-create the pods:

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/ && mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

Worker kubelet certs also expire. kubeadm certs renew kubelet-client-current on the node, or configure kubelet certificate rotation and never think about it again.

How to Upgrade the Control Plane#

  1. Take a backup and make sure you could restore it:
etcdctl ... snapshot save /var/backups/etcd-$(date +%F).db
etcdctl snapshot status /var/backups/etcd-$(date +%F).db -w table
  1. See what you’re allowed to move to:
apt-mark unhold kubeadm
apt-get update && apt-get install -y kubeadm=1.31.*-1.1
kubeadm upgrade plan

upgrade plan prints the current and target versions and warns about anything unsupported. Read it.

  1. Apply the control-plane upgrade:
kubeadm upgrade apply v1.31.0

On additional control-plane nodes (see ha-control-plane), use kubeadm upgrade node instead - only the first node runs apply.

  1. Upgrade kubelet and kubectl, then restart kubelet:
apt-mark unhold kubelet kubectl
apt-get install -y kubelet=1.31.*-1.1 kubectl=1.31.*-1.1
apt-mark hold kubelet kubectl kubeadm
systemctl daemon-reload && systemctl restart kubelet
  1. Verify before touching workers:
kubectl get nodes
kubectl get pods -n kube-system
kubectl version

STATUS = Ready, all kube-system pods running, no version skew surprises.

How to Upgrade Worker Nodes#

  1. Cordon and drain - the pod-priority-disruption notes explain why this can stall:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
  1. On the node, same dance as the control plane, minus upgrade apply:
apt-mark unhold kubeadm
apt-get update && apt-get install -y kubeadm=1.31.*-1.1
kubeadm upgrade node
apt-mark unhold kubelet
apt-get install -y kubelet=1.31.*-1.1
apt-mark hold kubelet kubeadm
systemctl daemon-reload && systemctl restart kubelet
  1. Back on the control plane, uncordon and move to the next node:
kubectl uncordon <node-name>
kubectl get nodes
  1. Repeat one node at a time. Never drain two at once unless your PDB and capacity say you can.

How to Restore etcd from a Snapshot#

The day the backup earns its keep:

  1. Stop the static pod from fighting you - move the apiserver and etcd manifests aside:
mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-etcd.yaml /root/
  1. Move the old data dir out of the way (do not delete it until the restore is verified):
mv /var/lib/etcd /var/lib/etcd.broken
  1. Restore the snapshot into a fresh data dir:
ETCDCTL_API=3 etcdctl snapshot restore /var/backups/etcd-2025-01-01.db \
  --data-dir=/var/lib/etcd \
  --name=<node-name> \
  --initial-cluster=<node-name>=https://<node-ip>:2380 \
  --initial-advertise-peer-urls=https://<node-ip>:2380

The --name and cluster URLs must match the member’s identity in the snapshot - restoring with the wrong name is how you end up with an etcd that answers but has no leader.

  1. Put the manifests back and watch the cluster return:
mv /root/kube-apiserver.yaml /root/kube-etcd.yaml /etc/kubernetes/manifests/
crictl ps
kubectl get nodes
  1. Verify the data is the data you expected - a specific namespace, a specific deployment. Then and only then, remove /var/lib/etcd.broken.

On a multi-member etcd cluster (HA notes), restoring one member from snapshot is a membership operation: the restored member rejoins the cluster and syncs, or you rebuild the member entirely. etcdctl member list before you touch anything.

How to Renew Certificates#

  1. See what’s near expiry:
kubeadm certs check-expiration
  1. Renew everything at once:
kubeadm certs renew all
  1. Restart the static pods so they load the new certs - either touch the manifests as shown above, or reboot the node. Then confirm:
kubectl get --raw=/healthz
kubeadm certs check-expiration
  1. Distribute the updated admin.conf to anyone holding an old kubeconfig, and update workers’ kubelet certs if you use the client cert flow. A renewed CA cert (only on kubeadm certs renew ca, which you should almost never do) invalidates every cert in the cluster - treat that as a disaster recovery scenario, not maintenance.

özetle - sıralama: backup → upgrade plan → upgrade apply (CP) → node’ları tek tek drain+upgrade+uncordon → certs check-expiration takvime not et. Bu sırayı bozmadığın sürece upgrade bir bakım işlemidir, zar atmak değil.

for more ha-control-plane

Backward multi-node-kubeadm ha-control-plane Forward