Hardening the API server & kubelet

anonymous-auth, authorization-mode, kubelet webhook.

Every kubectl command, every controller, every kubelet, and yes, every attacker who gets a foothold, talks to the same endpoint: the API server on port 6443. It's the only component that reads and writes etcd, the database that holds the entire cluster state. So a request that slips past its checks isn't a small problem. It's the whole cluster: every secret, every workload, every node, the lot. A handful of command-line flags decide how thorough those checks really are, and a few of them ship, or used to ship, looser than most people assume. This lesson is about turning the two front doors, the API server and the kubelet on each node, from 'technically locked' into 'actually locked, and I can prove it.'

The API server works like the door staff at a members' club, and it runs three checks in order. First it checks your ID: authentication, which only answers 'are you who you say you are?' Then it checks the guest list for the exact room you asked for: authorization, which answers 'is this identity allowed to do this thing?' Even if you clear both, a manager standing inside can still send you back to fix your jacket or refuse you outright: admission, where a plugin can rewrite your request or reject it. Only after all three passes does anything get written to etcd. Every single request runs that same gauntlet, and the flags below decide whether each check does real work or just waves people through.

Four flags do most of the work. Turn anonymous auth off so an unauthenticated request gets a flat 401 instead of being quietly mapped to the built-in system:anonymous identity, which a couple of default bindings grant a small sliver of access to (health and version endpoints). Set authorization to Node,RBAC (RBAC = Role-Based Access Control, the system that maps an identity to a list of what it's allowed to touch) so the real authorizers run on every call. The value you never want to see is AlwaysAllow: it switches authorization off completely, so anyone who can authenticate, even with a low-privilege token scraped out of a pod, can do anything at all. Turn profiling off, because the /debug/pprof endpoints it exposes leak memory contents and internal state that help an attacker map the process. And switch audit logging on, so you have a record, the cluster's CCTV, of who asked for what when something goes sideways later.

Two more flags belong in that list, but not for the reason you might expect: both are already the default, and you write them down so nobody can change them without it showing up in a diff. service-account-lookup has defaulted to true for years, and it makes the API server confirm a token's backing service account still exists before honoring it, so deleting a service account really does kill its tokens. CIS asks you to assert it in writing because someone may have turned it off. tls-min-version=VersionTLS12 is the same move against ancient, breakable TLS (Transport Layer Security). All of this lives in the API server's static pod manifest, a plain YAML file on the control-plane node that the kubelet watches. Save the file and the kubelet restarts the API server within a few seconds, and on a kubeadm cluster that first restart is exactly where anonymous-auth=false bites, for a reason the crash-loop callout below explains.

spec:
  containers:
  - command:
    - kube-apiserver
    - --anonymous-auth=false
    - --authorization-mode=Node,RBAC
    - --enable-admission-plugins=NodeRestriction,PodSecurity
    - --profiling=false
    - --service-account-lookup=true        # already the default; assert it
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-log-maxage=30
    - --tls-min-version=VersionTLS12

Setting a flag in a file and having the running server enforce it are two different things. Maybe the static pod didn't restart cleanly. Maybe you edited the manifest on one control-plane node and the other two behind the load balancer are still running yesterday's arguments. Either way the live process can be up with stale arguments while the file in front of you looks perfect. So check the process from the outside, not the file. A home inspector doesn't trust the blueprint; they walk the house. kube-bench is that inspector: it runs the CIS Kubernetes Benchmark (CIS = Center for Internet Security, which publishes hardening checklists auditors treat as the baseline) against the live process and reports PASS, FAIL, or WARN per control. That's how a CKS exam assessor and a real-world auditor both grade a cluster.

# 1) the door is locked: an unauthenticated call gets 401, not a mapped identity
$ curl -sk https://localhost:6443/api/v1/namespaces/kube-system/secrets -o /dev/null -w '%{http_code}\n'
401
 
# 2) kube-bench audits the RUNNING flags, not the file (k8s 1.31 -> cis-1.11)
$ kube-bench run --targets=master | grep -E '1.2.(1|7|8|15|16) '
[WARN] 1.2.1  Ensure that the --anonymous-auth argument is set to false (Manual)
[PASS] 1.2.7  Ensure that the --authorization-mode argument includes Node (Automated)
[PASS] 1.2.8  Ensure that the --authorization-mode argument includes RBAC (Automated)
[PASS] 1.2.15 Ensure that the --profiling argument is set to false (Automated)
[PASS] 1.2.16 Ensure that the --audit-log-path argument is set (Automated)

One line there looks like a failure and isn't. kube-bench marks the anonymous-auth check Manual, so it prints WARN instead of PASS: the tool won't confirm that flag on its own and kicks it back to you. That's what the curl above is for. The 401 is your proof anonymous auth is really off, and it beats a green checkmark a tool guessed at. Here's the sharp part. If anonymous auth were still on, that same call would come back 403 Forbidden, authenticated as nobody-in-particular but blocked by RBAC. With it off, you get 401, not authenticated at all. 401 versus 403 is the whole tell.

The kubelet is the soft underbelly

Every node runs its own agent, the kubelet, and it exposes its own HTTPS API on port 10250. Think of it as the service entrance at the back of each building. You can guard the front door all you like, but if the loading dock is propped open, someone just walks in the back. The old kubelet defaults prop it open: anonymous access allowed, and authorization set to AlwaysAllow, which together mean anyone who can reach port 10250 can list the pods on that node and exec a shell straight into your containers. No credentials asked.

Lock it down in the kubelet's config file. Disable anonymous auth. Require the API server's certificate authority for client certificates, so the two ends show each other valid ID badges before trusting anything; that mutual check is mutual TLS, or mTLS (mTLS = both sides present certificates, not just the server). Switch authorization to Webhook, which means every incoming call gets radioed back to the API server to ask 'is this caller allowed to do this?' instead of the kubelet deciding on its own. And set the read-only port to 0 to close the old unauthenticated endpoint on :10255, which used to hand out the full pod list to anyone who asked.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true          # bearer tokens checked via the API server
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook            # not AlwaysAllow
readOnlyPort: 0            # close the unauthenticated :10255 port

$ sudo systemctl restart kubelet
 
# an anonymous request to the authenticated port is now refused
$ curl -sk https://localhost:10250/pods | head -c 12
Unauthorized
 
# the read-only port is gone (it used to dump the pod list with no auth at all)
$ curl -s http://localhost:10255/pods
curl: (7) Failed to connect to localhost port 10255: Connection refused
 
# and the CIS node checks pass
$ kube-bench run --targets=node | grep -E '4.2.(1|2) '
[PASS] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)

AlwaysAllow on the kubelet is the soft underbelly: anyone who can reach port 10250 can run exec and read secrets from pods on that node. Webhook mode asks the API server, which is what you want.

Back at the front door, the manager standing inside is the check we've only pointed at so far, and two admission plugins do most of the work there. NodeRestriction is the one that stops a bad node spreading: a kubelet can only edit its own Node object and the pods actually bound to it, so a compromised node-7 can't relabel node-3 or attach itself to somebody else's pod. PodSecurity is the built-in replacement for the old PodSecurityPolicy, and it has shipped on by default since 1.25, but read that carefully. Switching the plugin on blocks exactly nothing. It only acts where you've labelled a namespace, with pod-security.kubernetes.io/enforce=restricted or similar, or where you've set a cluster-wide default in an AdmissionConfiguration file. Listing PodSecurity in enable-admission-plugins and stopping there is a checkbox, not a control.

That's the habit worth taking from all of this: prove it, don't assume it. Every control here has a negative test, and the negative test is the one that counts. Curl the kubelet with no client certificate and expect Unauthorized. Ask the API server for a secret with no credentials and expect 401. A grep that finds the flag only tells you what the file says, and you already know the file can be out of date.

Flags drift, most often after a hand-rolled upgrade regenerates the manifest and quietly drops something. Keep the kube-apiserver manifest in a reviewed repository or a config-management system, so a change to anonymous-auth or authorization-mode arrives as a diff somebody has to approve instead of a surprise at 3am. Write the verification command next to the control in the same pull request and keep its sample output, because the day you need it is the day you won't have time to work out which flag mattered.

Try this

Inspect API server and kubelet auth settings on a lab control plane. You want anonymous auth off, a real authorization mode, and kubelet authn/authz that is not AlwaysAllow. Read the files first, then check the running process, because only the second one is evidence.

$ sudo grep -E 'anonymous-auth|authorization-mode|enable-admission' \
  /etc/kubernetes/manifests/kube-apiserver.yaml
    - --anonymous-auth=false
    - --authorization-mode=Node,RBAC
    - --enable-admission-plugins=NodeRestriction,PodSecurity
 
# the file is only a claim; this is the process actually serving 6443
$ ps -ef | grep '[k]ube-apiserver' | tr ' ' '\n' \
  | grep -E 'anonymous-auth|authorization-mode|enable-admission'
--anonymous-auth=false
--authorization-mode=Node,RBAC
--enable-admission-plugins=NodeRestriction,PodSecurity
 
$ sudo grep -A2 '^authentication:' /var/lib/kubelet/config.yaml
authentication:
  anonymous:
    enabled: false
 
$ sudo grep -A1 '^authorization:' /var/lib/kubelet/config.yaml
authorization:
  mode: Webhook
 
$ curl -k https://127.0.0.1:10250/pods
Unauthorized

Takeaway

The API server is the only etcd door. Turn off anonymous auth, keep Node+RBAC, and never leave the kubelet on AlwaysAllow.

Related