Releases & version skew

What a release contains and how far components may drift.

A node went NotReady an hour after a routine upgrade, and the pods sitting on it never came back cleanly. (A node is one of the worker machines in the cluster. A pod is the little wrapper Kubernetes runs your container inside.) The app itself was fine. The real trouble was that the machine's own agent had drifted too far from the control plane, the handful of managers that steer the whole cluster, and Kubernetes quietly stopped promising the two would get along. That drift has a name. It's version skew, and it comes with a short list of rules. Learn them before you touch an upgrade, because they're the reason the actual upgrade in the next lesson goes smoothly instead of sideways.

What a release actually contains

Kubernetes versions read as 1.MINOR.PATCH, like 1.30.4. A new minor version lands about every four months, so three a year. Between those, the project ships patch releases that fix bugs and security holes without changing how anything behaves. The part that catches people out is the support window. Only the three newest minor versions get patches, which works out to roughly fourteen months of support for any one release. Fall off the back of that and the security fixes stop arriving. That's not something you get to postpone to a calmer quarter. An unpatched control plane is a live way in for an attacker, so the calendar ends up making the upgrade decision for you.

Every release ships notes, and the line that matters most sits under removed APIs. An API here is just the agreed format the cluster uses to talk about a thing, like a Deployment or a policy. Think of an API version as a government form. When a newer form replaces the old one, there's a grace period where the counter still accepts both. Then comes a date after which the old form gets rejected outright. A beta API you've quietly been relying on gets exactly that treatment. So read the target release's notes for removals before you go anywhere near the upgrade button.

# Client (your laptop) + server (control plane) versions
kubectl version

Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.4

Client Version is the kubectl on your laptop (kubectl is the command-line tool you use to talk to the cluster). Server Version is the API server running inside the cluster. Here they sit one minor apart, 1.31 talking to 1.30.4, which is inside the supported one-minor range for kubectl. One thing you might notice: there's no --short flag to reach for anymore. It got deprecated back in 1.28, when this compact output became the default, and it's gone now.

The skew rules, and why the manager upgrades first

A busy restaurant kitchen during the dinner rush. The expediter at the pass calls out the tickets, and that's your API server. The line cooks work those tickets, and they're your kubelets, the small agent program that runs on every node and actually starts your containers. A cook can be a little behind on this week's new menu and still do the job. What a cook can never be is ahead of the expediter, plating a dish the pass hasn't called, because now there's food nobody ordered jamming up the line. Version skew has the same shape.

Here are the numbers for 1.31 and up. The kubelet can run up to three minor versions older than the API server, and never newer. kubectl should stay within one minor of the API server in either direction. The two other control-plane pieces, the scheduler and the controller-manager, can trail by up to one minor but must never lead. And the big one: you move the control plane up one minor at a time. From 1.28 you go to 1.29, then 1.30, then 1.31. You never jump 1.28 straight to 1.31. Because the kubelet is allowed to trail but never lead, the order writes itself. Control plane first, nodes after.

# The VERSION column is each node's kubelet version
kubectl get nodes

NAME     STATUS   ROLES           AGE    VERSION
cp-1     Ready    control-plane   210d   v1.30.4
node-1   Ready    <none>          210d   v1.30.4
node-2   Ready    <none>          88d    v1.27.9

Read that last line carefully. node-2 is on 1.27 while the control plane is on 1.30. That's exactly three minors back, right on the edge of what's supported. It works today. But bump the control plane to 1.31 and node-2 is suddenly four minors behind, out of skew, and now you're in the exact situation that produced the dead node at the top of this lesson. Fix the lagging node before the upgrade, not after.

Here's the uncomfortable part: at runtime, almost nothing enforces this. There's no hard gate that rejects an old kubelet the way a firewall drops a packet. Skew is a tested-and-supported promise, not a lock, which is why a violation feels completely fine right up until it doesn't. The one-minor-at-a-time rule does have a solid reason under it, though. The API server keeps every object in etcd, the cluster's key-value database, written at one particular storage version. During an upgrade the server reads the old stored shape and converts it to the new one on the way through. That conversion only gets written and tested for a single-minor hop. Skip a version and you're asking it to translate across a gap nobody ever tested.

Removed APIs are what break the upgrade

Skew rules break nodes. Removed APIs break your manifests, and that break is a lot louder. (A manifest is just the text file, usually YAML, that tells Kubernetes what you want running.) When a beta API graduates to stable, its old version gets a removal date stapled on. PodSecurityPolicy vanished in 1.25, and the modern replacement is Pod Security Admission, built right into the API server. Some older flowcontrol and autoscaling beta versions have been dropped in the releases since. If a Deployment, a Helm chart (a packaged, reusable app template), or an operator (a controller that runs and manages an app for you) still points at a version that's been removed, kubectl apply starts throwing errors. And any GitOps controller, the kind of robot that constantly re-syncs the cluster to match files kept in Git, will wedge itself in a retry loop trying to apply a manifest the server keeps refusing. You do not want to meet this problem for the first time during a failed rollout at 2am. Scan ahead of time. Two tools, kubent (short for kube-no-trouble) and Pluto, read your live cluster and flag anything using an API that's on the chopping block.

# Scan the live cluster for deprecated / removed APIs
kubent

9:12AM INF >>> Kube No Trouble `kubent` <<<
9:12AM INF version 0.7.3
9:12AM INF Initializing collectors and retrieving data
9:12AM INF Target K8s version is 1.24.17
9:12AM INF Retrieved 214 resources from collector name=Cluster
__________________________________________________________________________________
>>> Deprecated APIs removed in 1.25 <<<
----------------------------------------------------------------------------------
KIND               NAMESPACE     NAME         API_VERSION      REPLACE_WITH (SINCE)
PodSecurityPolicy  <undefined>   restricted   policy/v1beta1   <removed> (1.21.0)

Here kubent is scanning a 1.24 control plane, one hop away from the 1.25 that deletes PodSecurityPolicy for good. It turned up a single object still on policy/v1beta1: a PodSecurityPolicy named restricted that would stop working the instant you crossed into 1.25. The fix is to migrate it, or delete it, before the upgrade, then rerun the scan until it comes back clean. Do this at every minor step, not just once at the start, because each hop carries its own removal list.

kubectl too old or too new hides deprecations and breaks apply. Keep CI images current.

Addons like CoreDNS and CNI have their own matrices. Read the release notes for your install tool.

Skew is how silent incompatibilities accumulate. Calendar upgrades; do not wait for end of life emails.

Try this

Run kubectl version and compare client, server, and node kubelet versions. Note any skew beyond one minor and plan upgrades.

# Client (your laptop) + server (control plane) versions
$ kubectl version
 
# The VERSION column is each node's kubelet version
$ kubectl get nodes

Takeaway

Version skew rules are load-bearing. Control plane first, then nodes, keeping kubelets within supported distance of the API.

Related