Minimize the host OS

Cut services, ports, and packages on the node.

An attacker who lands a shell on a worker node has already won more than most people expect. That box holds the kubelet's credential, a socket to the container runtime that is root-equivalent control of the machine, and the secrets of every pod scheduled there, sitting in memory. Every node also shares one Linux kernel with all the containers on it, so a kernel bug reachable from a pod is a bug in the very thing that's supposed to keep pods apart. And every NetworkPolicy, RBAC (Role-Based Access Control) rule, and Pod Security label you wrote quietly assumes the node underneath is trustworthy. So hardening starts at the node, not above it. The plan is footprint reduction, which is really attack-surface reduction: the fewer programs, ports, and packages a machine runs, the fewer ways in, and the fewer ways to pivot once someone is already inside. A locked building with three doors is easier to guard than one with thirty.

There are two roads to a small node. Start from something minimal, or trim down the general-purpose distro you inherited. Purpose-built node operating systems like Flatcar, Bottlerocket, and Talos take the first road: no package manager, sometimes no shell at all, a root filesystem mounted read-only. It's closer to a sealed kitchen appliance than a full kitchen. There are no loose knives to grab, and when an update lands you swap the whole unit instead of repairing it in place. That swap-don't-patch idea is what people mean by an immutable OS. An attacker who gets in finds nothing to install to and no writable system files to backdoor. Most teams, though, are running a stock Ubuntu or something like it, so the rest of this lesson is the trimming playbook for the node you actually have.

You can't shrink a footprint you haven't measured. Start by listing what actually runs and what actually listens, then close anything Kubernetes doesn't need. A print spooler, a mail transfer agent, a leftover desktop daemon: none of it belongs on a node, and each one is a live service someone can probe, crash, or exploit. This isn't a one-time chore, either. A base-image bump or a config-management run can re-add a package or flip a service back on, so the check belongs in your pipeline, not in your memory. Disabling a service stops it today. Masking is stronger. It points the unit at /dev/null so nothing can start it again, not a curious admin and not another unit that lists it as a dependency.

# What's running and what's listening right now?
$ systemctl list-units --type=service --state=running
$ ss -tulnp | grep :631            # CUPS print spooler on 631
tcp  LISTEN 0 128 0.0.0.0:631 0.0.0.0:* users:(("cupsd",pid=812,fd=6))
 
# Stop it now, keep it off at boot, then weld the door shut
$ sudo systemctl disable --now cups
$ sudo systemctl mask cups
Created symlink /etc/systemd/system/cups.service -> /dev/null.
 
# Prove it: masked, and no longer listening
$ systemctl is-enabled cups
masked
$ ss -tulnp | grep :631            # (no output = port 631 is gone)

Cut packages and close ports

Think about what you leave lying around. A production node with gcc, git, and a package manager installed hands an attacker a workshop: compile an exploit in place, pull down a second stage, build tooling right on the box instead of smuggling it past you. Security folks call this living off the land, using what's already installed so nothing you'd flag as malicious ever has to touch disk. The same goes for interpreters you aren't using and debug tools left over from provisioning. Take the workshop away. A node needs its kubelet, a container runtime, and very little else. When you genuinely need to build something, do it in CI (Continuous Integration) and ship an image, not on a live node.

# An attacker on the box should not find a compiler waiting
$ sudo apt-get purge -y gcc make git
Reading package lists... Done
Removing gcc (4:12.2.0-3) ...
Removing git (1:2.39.5-0+deb12u2) ...
 
# Prove the toolchain is gone
$ for b in gcc make git; do command -v "$b" || echo "$b: not found"; done
gcc: not found
make: not found
git: not found
$ dpkg -l | grep -E '^ii  (gcc|git) '   # (no output)

A host firewall works like a bouncer with a guest list. Nobody gets in unless they're on it. Set the default to deny, then allow only the ports Kubernetes actually speaks on: the API server and etcd on control-plane nodes, the kubelet and the NodePort range on workers. Everything that used to answer a stray connection now goes silent. The same logic says keep management access off the open internet: reach nodes through a bastion or a private network, so the only things the world can touch are ports a cluster genuinely has to serve. Fewer open ports means fewer front doors to pick, and the CIS (Center for Internet Security) Benchmark expects exactly this default-deny posture on a node.

$ sudo ufw default deny incoming
$ sudo ufw allow 6443/tcp          # kube-apiserver (control plane)
$ sudo ufw allow 2379:2380/tcp     # etcd client + peer (control plane)
$ sudo ufw allow 10250/tcp         # kubelet API (all nodes)
$ sudo ufw allow 30000:32767/tcp   # NodePort range, if you use it
$ sudo ufw enable
 
# Prove the policy is live and the default really is deny
$ sudo ufw status verbose
Status: active
Default: deny (incoming), allow (outgoing), deny (routed)
 
To                Action    From
--                ------    ----
6443/tcp          ALLOW IN  Anywhere
2379:2380/tcp     ALLOW IN  Anywhere
10250/tcp         ALLOW IN  Anywhere
30000:32767/tcp   ALLOW IN  Anywhere

Shrink the kernel and lock down access

The kernel loads modules on demand, which means a workload can trigger code for hardware and protocols you will never use. Every module that can load is code that can carry a bug, and code that never loads can't be exploited. Treat module loading like a phone's contact whitelist: if a number isn't saved, the call doesn't connect. Block the obscure network protocols (DCCP, SCTP, RDS) so a malicious socket call can't pull them in and reach a rarely-tested corner of the kernel. While you're in there, close the small settings that let a compromised node act as a pivot against its neighbors. Ignore ICMP (Internet Control Message Protocol) redirects and source-routed packets so the box can't be tricked into rerouting or spoofing traffic for the machines around it. Then handle access. Ban shared root login over SSH: logins should trace back to named humans before anyone escalates, because a shared root account erases your audit trail before an incident even starts.

# Close pivot tricks: drop ICMP redirects and source routing, hide kernel pointers
$ cat <<'EOF' | sudo tee /etc/sysctl.d/90-hardening.conf
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.kptr_restrict = 2
EOF
$ sudo sysctl --system
 
# Refuse to load protocols no normal workload needs
$ for m in dccp sctp rds; do echo "install $m /bin/true"; done | sudo tee /etc/modprobe.d/blocklist.conf
install dccp /bin/true
install sctp /bin/true
install rds /bin/true
 
# Prove it: the knob is set, and the module can't load
$ sysctl net.ipv4.conf.all.accept_redirects
net.ipv4.conf.all.accept_redirects = 0
$ sudo modprobe -n -v dccp
install /bin/true                  # loading dccp runs a no-op instead

# Named humans only: no shared root login over SSH
$ sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
$ sudo systemctl reload ssh
 
# Prove it against the running config, not just the file
$ sudo sshd -T | grep -i permitrootlogin
permitrootlogin no

hostPath mounts and privileged pods turn a soft node into a shared root shell. Node hardening without Pod Security is incomplete.

SSH access should be break-glass with MFA and short certs, not a shared password for the whole platform team.

Kernel modules and sysctls you do not need are attack surface. Prefer CIS node images or a documented baseline over pet snowflakes.

Container runtimes and CNI plugins add sockets and config files on the node. Inventory them. World-readable CNI configs and docker.sock mounts are classic lateral paths that never show up in a pod YAML review if you only look at Deployments. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

On a worker lab node, list listening ports and unnecessary packages, then confirm kubelet permissions on its config files.

$ sudo ss -lntup | grep -E '10250|10255|22|2379' | head
LISTEN 0 128 127.0.0.1:10248 0.0.0.0:* users:(("kubelet",pid=1234,fd=12))
LISTEN 0 128 0.0.0.0:10250     0.0.0.0:* users:(("kubelet",pid=1234,fd=15))
LISTEN 0 128 0.0.0.0:22        0.0.0.0:* users:(("sshd",pid=800,fd=3))
 
$ sudo ls -l /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf
-rw------- 1 root root 2801 Jul 12 09:11 /var/lib/kubelet/config.yaml
-rw------- 1 root root 1440 Jul 12 09:11 /etc/kubernetes/kubelet.conf
 
$ dpkg -l | grep -E 'telnet|ftp|rsh' || echo "no legacy remote tools"

Takeaway

Workers are production hosts. Cut packages, close ports, lock file modes, and never leave read-only kubelet ports open.

Related