>_SecOpsLog_
CoursesKubernetes security & hardeningCIS benchmarks with kube-bench

CIS benchmarks with kube-bench

Score a node against the benchmark, then remediate.

Advanced12 min · lesson 3 of 24

A car can start, steer and brake perfectly and still fail its annual safety inspection, because "it drives" and "it is safe" are two different questions. Clusters work the same way. Yours can schedule pods, serve live traffic and pass every smoke test while the API server (application programming interface server), the single front door that every kubectl command and every internal controller talks to, still answers callers who never proved who they are. Functional tests never ask whether anonymous access is switched off, or whether the etcd data directory is readable by half the machine. etcd is the small database holding your cluster's entire state, so that second question matters rather a lot. The gap between "it works" and "it is safe" is what the CIS Benchmark, published by the Center for Internet Security, exists to cover.

The CIS Kubernetes Benchmark is the closest thing this community has to an agreed hardening checklist. It reads like a building inspector's clipboard: hundreds of specific, testable items covering control-plane flags, file ownership and permissions, and kubelet settings. The kubelet is the agent running on every node that actually starts your containers. Each item comes with a check and a fix printed right next to it. The document runs long and it is dry as sand, which is exactly why nobody works through it by hand.

Let kube-bench read it for you

kube-bench walks that whole checklist against one node and scores it, quoting the item number and the recommended fix beside every finding. Run it once per node role. Control-plane nodes and worker nodes are graded against different sections, so a worker is never asked about API server flags, and a control-plane node carries its own long list. Inside a running cluster you launch it as a Job, a one-shot pod that runs to completion, built from the aquasec/kube-bench image. On a node you are hardening by hand, or in the CKS (Certified Kubernetes Security Specialist) exam, the binary is usually already sitting on the box.

Four verdicts come back. PASS and FAIL mean what they say. WARN nearly always means "you check this one, I cannot see enough to judge it", which makes it a manual item rather than a free pass. The anonymous access check is the classic case. If the flag is not spelled out in the API server's argument list, kube-bench cannot prove what the default is doing, so it stamps the item WARN and hands the decision back to you. INFO is background. Read the FAILs first, because every one of them names the file and the exact value it wants.

terminal
$ kube-bench run --targets master
 
[INFO] 1 Control Plane Security Configuration
[PASS] 1.2.5  Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)
[FAIL] 1.2.15 Ensure that the --profiling argument is set to false (Automated)
[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
[WARN] 1.2.1  Ensure that the --anonymous-auth argument is set to false (Manual)
 
== Remediations master ==
1.2.15 Edit the API server pod spec /etc/kubernetes/manifests/kube-apiserver.yaml
       on the control plane node and set --profiling=false

Read the report, then fix it on the node

Control-plane flags live in static pod manifests under /etc/kubernetes/manifests. Those are plain YAML files (YAML is the whitespace-sensitive text format Kubernetes config is written in) that the kubelet runs directly, with no scheduler and no API server in the middle. There is no kubectl apply here. The kubelet watches that folder the way a printer watches a hot folder: drop a change in, and it kills the old container and starts a fresh one carrying your flag, usually within a few seconds.

Here is the part almost everyone misses on a first pass. The benchmark does not only grade the API server. It grades the controller-manager and the scheduler as well, as separate numbered items in their own sections. Set --profiling=false in kube-apiserver.yaml and you have cleared exactly one of three profiling findings. A clean sheet means editing all three manifests. Most people stop after the first.

/etc/kubernetes/manifests/kube-controller-manager.yaml
spec:
  containers:
  - command:
    - kube-controller-manager
    - --profiling=false                        # CIS 1.3.2
    - --use-service-account-credentials=true   # CIS 1.3.3

Now prove it. Saving the manifest should have triggered a restart, so confirm that a new container really came up before you trust anything. Check with crictl, the container runtime command line tool that talks straight to the node's runtime without going through the API server. Then re-run kube-bench scoped to the handful of items you touched. A rising ATTEMPT count tells you the kubelet reloaded the pod. The PASS lines tell you the fix landed.

terminal
$ crictl ps --name kube-controller-manager
CONTAINER      IMAGE       CREATED           STATE     NAME                      ATTEMPT   POD ID
9f3c1a7b2e4d   1d3c9f...   14 seconds ago    Running   kube-controller-manager   3         7a2f8b...
 
$ kube-bench run --targets master --check 1.2.15,1.3.2
[PASS] 1.2.15 Ensure that the --profiling argument is set to false (Automated)
[PASS] 1.3.2  Ensure that the --profiling argument is set to false (Automated)

Work in short loops. Fix a batch, re-run kube-bench scoped to only those item numbers, confirm every FAIL flipped to PASS, then take the next batch. Scoping keeps the output short enough to actually read, and it gives you a real feedback loop instead of a hundred lines to scroll past on every pass.

The kube-bench fix loop
1Scan the nodekube-bench run --targets…2Triage the FAILsEach FAIL names the CIS item,…3Fix it in placeA flag goes in a static pod…4Component reloadsSaving a manifest makes the…5Re-run, scopedkube-bench --check 1.2.15,1.3.2…
Run the loop once per node role: control-plane and worker nodes are graded against different sections.

Not every FAIL needs a flag

A big slice of the benchmark is unglamorous housekeeping. chmod 600 on the manifests and kubeconfig files, which is the Linux way of saying only the file's owner may read or write it. chown etcd:etcd on the etcd data directory, which hands ownership to the etcd user so a workload that breaks out of its container cannot stroll into a copy of your cluster's entire state. These fixes bite the moment you run them, and checking your work is equally fast: read the ownership back, then re-run that single item.

terminal
$ chown etcd:etcd /var/lib/etcd
 
$ stat -c '%U:%G' /var/lib/etcd
etcd:etcd
 
$ kube-bench run --targets master --check 1.1.12
[PASS] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)

Read the remediation text before you touch anything. Some items want a permission change rather than an edit, and adding a flag your Kubernetes version does not recognise will crash the component the second it restarts. When you deliberately leave something failing, whether it is a WARN you have judged acceptable or a flag you cannot set because it breaks a real workload, write down the reason. Auditors accept a justified exception. They do not accept a silent one.

One typo in a manifest can lock you out of the API server
Static pod manifests get no admission checks and no dry run. Fat-finger a flag or knock the indentation out of line in kube-apiserver.yaml, save the file, and the kubelet obediently restarts the API server straight into a crash loop. kubectl then goes silent, because the thing it talks to is gone. Do not panic, and do not keep editing blind. Drop down to the node and debug there: crictl ps -a shows the crashing container, crictl logs <id> tells you why it died, and journalctl -u kubelet gives you the kubelet's own account of the reload. Repair the manifest on disk and the kubelet brings the component back on its own.

Not every FAIL is a day-one production blocker. Some checks assume one distribution's file layout, or a control plane you do not actually own. That is the usual story on a managed service like EKS (Elastic Kubernetes Service) or GKE (Google Kubernetes Engine), where the vendor holds the API server manifest and you have no way to edit it. Note which lines those are and how the provider covers them. An unexplained gap in the report is what turns a ten-minute audit conversation into a two-hour one.

Permission items on manifests and etcd certificates turn up in every single report. They look dull and they matter. A world-readable API server pod spec hands anyone with a shell on that node the full flag list, the certificate paths and the etcd endpoints, which amounts to a map of exactly where to go next.

Re-run kube-bench after every cluster upgrade, and after anyone edits an API server or kubelet flag by hand at three in the morning. Config drifts quietly, and drift is how "we hardened this last year" becomes "anonymous auth is back on" without a single person noticing.

Findings with no owner rot. Put the CIS item number in the ticket title so 1.2.15 is still searchable a year from now, paste the kube-bench lines straight into the description, and close the ticket only when a scoped re-run prints PASS or a named person has signed off the exception.

Keep the proof next to the change. When the manifest edit goes up for review, put the exact kube-bench --check command and its output in the same pull request. The next person on call can then tell a pass from a fail without reconstructing your reasoning from scratch. Do that consistently and the second sweep, the one you run under real pressure during an incident, takes minutes instead of an afternoon.

Try this

Run kube-bench against a control-plane node, or against a lab cluster you spin up locally with kind or k3d (both run Kubernetes inside containers on your own machine) using the job manifest. Then fix one FAIL you can prove with either a flag or a file permission check.

terminal
$ kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-master.yaml
job.batch/kube-bench created
 
$ kubectl logs job/kube-bench | sed -n '/\[FAIL\]/,/\[PASS\]/p' | head -20
[FAIL] 1.2.1 Ensure that the --anonymous-auth argument is set to false
[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate
...
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive
 
$ # prove one finding on the node
$ sudo grep anonymous-auth /etc/kubernetes/manifests/kube-apiserver.yaml
    - --anonymous-auth=false

Takeaway

kube-bench grades a node's config against the CIS Kubernetes Benchmark and hands you the item number along with the fix. Treat a FAIL as a ticket with a citation attached, never as a feeling. Fix it, re-run with --check, keep the output.

Quick check
01kube-bench still reports FAIL for --profiling on both the controller-manager and the scheduler. You already set --profiling=false in kube-apiserver.yaml and the API server restarted cleanly. Why are those two findings still open?
Incorrect — There is no such pass-through. Each control-plane component reads its own flags out of its own static pod manifest, independently of the others.
Correct — Profiling is checked per component (1.3.2 for the controller-manager, and the scheduler in its own section), so kube-controller-manager.yaml and kube-scheduler.yaml both need the edit.
Incorrect — No. --profiling is an ordinary per-component command-line flag, not a feature gate, and you set it separately on each binary.
Incorrect — No. These are real, testable process flags. Ignoring them leaves the built-in profiling endpoint (pprof) live on both components for anyone who can reach it.
02kube-bench prints [WARN] against the --anonymous-auth check on the API server. What does that verdict actually tell you?
Incorrect — No. WARN is neither a pass nor a fail. It is a judgement kube-bench deferred, and an auditor still wants your proof.
Incorrect — No. WARN says nothing at all about risk. It says kube-bench could not judge the item on its own.
Correct — WARN means "you look at this one". Here the effective flag is invisible to the tool, so the call comes back to you.
Incorrect — No. Anonymous access is a live control on current clusters. WARN reflects a check the tool could not verify, not a retired one.
03You add a flag to /etc/kubernetes/manifests/kube-apiserver.yaml, save it, and now kubectl hangs with no answer at all. crictl shows the API server container crash-looping. What is the right next move?
Correct — A static pod has no API path into it. You debug on the node with crictl, fix the file, and the kubelet brings the pod back by itself.
Incorrect — No. The API server is a static pod rather than a Deployment, and kubectl is dead anyway because the server it talks to is down.
Incorrect — No. kubectl cannot reach a crashed API server, and static pod manifests are never managed through kubectl apply.
Incorrect — No. The kubelet faithfully restarts whatever the file says. It will keep crash-looping until you edit the manifest.
Network policies & default-denyIngress with TLS

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep