Restrict API access & upgrade often

NodeRestriction, version currency, exposure.

Plenty of clusters put the API server behind a public load balancer, hand out cluster-admin like party favors, and quietly run a Kubernetes version that shipped eighteen months ago. Each of those is a free win for whoever finds it, and each is cheap to close. This lesson is about two of the least glamorous, highest-payoff hardening moves you have: shrink who can reach the control plane, and stay current on versions.

Give each kubelet its own keycard

In a hotel where every housekeeper carries a master key, one bad hire exposes every room. A safer setup hands each housekeeper a keycard that opens only the rooms they're assigned to, and nothing else. Two API server settings do that job as a pair. The Node authorizer (turned on with authorization-mode=Node) is the keycard: it lets a kubelet read and write only the objects tied to pods scheduled on its own node. NodeRestriction is the rule that stops a housekeeper from cutting fresh keys for other rooms. It blocks a kubelet from editing any Node object except its own, and from touching pods that aren't bound to it.

Skip these and one compromised node becomes a whole-cluster problem. A kubelet identity that can patch any Node object can relabel a control-plane node to attract sensitive pods, or reach Secrets it never had a reason to see. The Node authorizer also scopes reads: a kubelet can only pull the Secrets, ConfigMaps, and volumes belonging to pods that actually landed on its node, so a nosy node can't enumerate every credential in the cluster. With Node authorization plus NodeRestriction in place, a node that gets popped can wreck itself and go no further. You switch both on in the API server's static pod manifest, alongside anonymous-auth=false so no unauthenticated caller gets past the door in the first place.

spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC              # Node authorizer gates kubelet identities
    - --enable-admission-plugins=NodeRestriction  # then admission locks each kubelet to its own node
    - --anonymous-auth=false                       # unauthenticated callers get nothing

Confirm the flags landed, then prove they bite. You don't need to break into a node to test this; impersonating its identity with --as is enough to watch the boundary hold. Act like a node, try to touch a different one, and it should be refused with NodeRestriction naming exactly why.

# confirm the flags are actually live on the running API server
$ kubectl -n kube-system get pod -l component=kube-apiserver \
    -o jsonpath='{.items[0].spec.containers[0].command}' | tr ' ' '\n' \
    | grep -E 'authorization-mode|admission-plugins|anonymous-auth'
--authorization-mode=Node,RBAC
--enable-admission-plugins=NodeRestriction
--anonymous-auth=false
 
# now impersonate node worker-1 and try to modify a different node
$ kubectl label node worker-2 pwned=true \
    --as=system:node:worker-1 --as-group=system:nodes
Error from server (Forbidden): nodes "worker-2" is forbidden: node "worker-1"
 is not allowed to modify node "worker-2"

The network decides who gets to knock

A nightclub has a bouncer at the door and a set of rules inside. The bouncer decides who even gets to walk up and show ID. The rules inside decide what you can do once you're through. RBAC (Role-Based Access Control) is the inside rules: it says what an authenticated caller is allowed to do. The network is the bouncer: it decides who can reach port 6443 to attempt authentication at all. Reachability on its own isn't a security control, but needless reachability is a standing liability. There's no reason to let the entire internet queue at your control-plane door, no matter how good your ID check is.

Put the API server behind a firewall or cloud security group that allows 6443 only from the networks that genuinely administer the cluster: your bastion, your CI runners, an operator VPN range. Drop the public load balancer sitting in front of it by default. One caution before you lock it down: make sure your own path in stays open, or you'll firewall yourself out of the cluster you're trying to protect. Then verify two things independently. From an untrusted network the port should be filtered and time out, not refuse. And with anonymous-auth off, a caller who does reach the port gets nothing without credentials.

# apply: allow 6443 only from the bastion /32, deny the rest (AWS security group)
$ aws ec2 authorize-security-group-ingress --group-id sg-0api \
    --protocol tcp --port 6443 --cidr 203.0.113.10/32
 
# verify from an off-net host: the port is filtered, the connection hangs then times out
$ nc -vz -w4 api.acme.internal 6443
nc: connect to api.acme.internal port 6443 (tcp) timed out
 
# verify anonymous auth is off: no credentials, no data, even when reachable
$ curl -sk https://api.acme.internal:6443/api/v1/namespaces
{
  "kind": "Status", "status": "Failure",
  "message": "Unauthorized", "reason": "Unauthorized", "code": 401
}

Run current, or run known-vulnerable

A published CVE (Common Vulnerabilities and Exposures) is a burglary guide with your address printed on it. Kubernetes ships security fixes in patch releases, and the project only supports the three most recent minor versions. Run months behind and you're serving documented holes in the most valuable process you own. The API server is the single richest target in the cluster, and an unpatched remote vulnerability there is a full compromise sitting idle until a scanner finds it. Patch releases rarely touch behavior you depend on; they close holes. The cost of staying current is almost always lower than the cost of the one CVE you skipped.

Upgrades follow a fixed order because of version skew rules, and the order is not a preference. The control plane goes first. The kubelet is allowed to trail the API server by up to three minor versions, but it's never allowed to lead it, so nodes always upgrade after the control plane, never ahead of it. The reason the kubelet must never lead is practical: a newer kubelet can send API fields an older API server doesn't understand and will silently drop, so you get subtle, hard-to-debug breakage instead of a clean failure. With kubeadm the sequence is plan, apply on the control plane, then drain, upgrade, and uncordon each node one at a time so workloads keep moving while you go.

# control plane first
$ sudo kubeadm upgrade plan
[upgrade] Latest stable version: v1.32.5
COMPONENT        CURRENT   TARGET
kube-apiserver   v1.31.4   v1.32.5
$ sudo kubeadm upgrade apply v1.32.5
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.32.5".
 
# then each worker, one at a time
$ kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
$ sudo kubeadm upgrade node && sudo systemctl restart kubelet
$ kubectl uncordon worker-1
node/worker-1 uncordoned

Then prove the version actually moved. kubectl get nodes reports each node's kubelet version, and a node still on the old minor after its restart is a node whose upgrade quietly didn't take. Here worker-2 hasn't been through the loop yet; it's the next one in line.

$ kubectl get nodes
NAME        STATUS   ROLES           AGE    VERSION
control-1   Ready    control-plane   210d   v1.32.5
worker-1    Ready    <none>          210d   v1.32.5
worker-2    Ready    <none>          210d   v1.31.4   # still pending, drain it next

Version skew rules exist because the API moves. Skipping minors is how you inherit broken aggregators and surprise deprecations mid-incident.

Network exposure of 6443 to the whole internet is a gift. Put the API behind a bastion, VPN, or private endpoint, and watch audit logs for probes.

Back up etcd before upgrades. A clean rollback story is part of hardening, not an afterthought for the change ticket.

Skew between kubectl and the server hides deprecations until apply time. Keep client and server on the same minor in CI images. For worker pools, surge upgrades with pod disruption budgets so hardening rollouts do not become accidental outages. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Confirm NodeRestriction is on, check the server version skew, and sketch the upgrade order before you touch production.

$ kubectl version --short
Client Version: v1.29.4
Server Version: v1.29.4
 
$ sudo grep NodeRestriction /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=...,NodeRestriction,...
 
$ kubectl get nodes -o wide
NAME       STATUS   VERSION   INTERNAL-IP
master-0   Ready    v1.29.4   10.0.1.10
worker-a   Ready    v1.29.4   10.0.1.21
 
$ # control plane first, then workers one by one — never skip a minor
$ echo "plan: 1.28 -> 1.29 only after etcd backup + drain rehearsal"

Takeaway

Shrink who can reach the control plane, give each kubelet NodeRestriction, and stay on a supported minor. Old clusters are known-vulnerable clusters.

Related