>_SecOpsLog_
CoursesKubernetes security & hardeningAudit logs

Audit logs

An audit policy that records what matters.

Advanced12 min · lesson 24 of 24

You can operate inside a cluster for weeks and leave nothing behind, if audit logging is off. On a fresh cluster it usually is off, or it's set to a starter policy that records almost nothing worth keeping. Falco watches the host and its syscalls: a shell spawns, a binary you didn't expect starts running. The Kubernetes API (Application Programming Interface) server audit log watches a different layer. It records who asked the control plane to do what, and when. Put the two side by side and you cover both halves of one investigation. Falco tells you a shell opened in payments-api. The audit log tells you which identity ran that exec at 15:02, and that the same identity listed three Secrets a minute later. It's a structured, time-ordered record of every request the API server handled. But you get it only if you write a policy first. With no policy loaded, the API server records nothing.

An audit policy is a list of rules the API server reads top to bottom, first match wins, the way a bouncer works down a guest list until a name matches. Each rule assigns one of four levels, and a level is just how much you write down about a request. Think of a front-desk visitor log. None means you don't record the visit at all. Metadata records who arrived, when, and who they came to see, but nothing they said. Request adds the body, what they actually asked for. RequestResponse keeps both sides of the exchange, the question and the answer. Watch out for one trap here: Request and RequestResponse both write the request body, and for a Secret the create request body is the Secret data itself. That's why sensitive resources belong at Metadata, not one notch higher. The whole skill is recording enough to reconstruct an incident later without drowning in noise, or worse, copying Secret contents onto disk where they don't belong. So sensitive resources sit at Metadata, the writes that carry a body worth keeping get RequestResponse, and routine background chatter gets None.

/etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: None                            # drop kube-proxy's constant endpoint/service watches
  users: ["system:kube-proxy"]
  verbs: ["watch"]
  resources: [{ group: "", resources: ["endpoints", "services"] }]
- level: None                            # drop kube-system service-account chatter
  userGroups: ["system:serviceaccounts:kube-system"]
- level: Metadata                        # secrets: record access, never contents
  resources: [{ group: "", resources: ["secrets", "configmaps"] }]
- level: Metadata                        # exec/attach: the command is in the URI, there is no body
  resources: [{ group: "", resources: ["pods/exec", "pods/attach"] }]
- level: RequestResponse                 # rbac writes: keep the exact object submitted
  verbs: ["create", "update", "patch", "delete"]
  resources: [{ group: "rbac.authorization.k8s.io", resources: ["rolebindings", "clusterrolebindings"] }]
- level: Metadata                        # everything else, minimally

Wire the policy into the API server and point it at a log file, with rotation caps so a busy day can't fill the disk and take the control plane down with it. For detection anyone will actually act on, stream the events off the box too. A webhook backend forwards each event to your SIEM (Security Information and Event Management platform, the tool your responders live in). Run it in batch mode so a slow SIEM can never stall an API request while the server waits for an acknowledgement. It's the same logic as a security camera that uploads its footage off-site instead of trusting the one tape in the lobby: a local audit log on a compromised node is one of the first files an intruder trims or wipes, so you want the record sitting somewhere they can't reach before they even think to look for it.

/etc/kubernetes/manifests/kube-apiserver.yaml
- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/var/log/kubernetes/audit.log
- --audit-log-maxage=30                 # days to keep
- --audit-log-maxbackup=10              # rotated files to keep
- --audit-log-maxsize=100               # MB per file before rotating
- --audit-webhook-config-file=/etc/kubernetes/audit/webhook.yaml
- --audit-webhook-mode=batch            # non-blocking: batch events to the SIEM

Restarting the API server with new flags proves nothing on its own. A flag can be present and the control still be useless, because the flag turns logging on but says nothing about whether your policy captures the right things at the right levels. So prove it twice. First, that the flags line up with the CIS (Center for Internet Security) Kubernetes Benchmark, the community baseline most auditors grade you against. Then, separately, that real events actually land at the levels you intended. Neither check alone is enough. Pass the benchmark with a policy that logs nothing useful and you've got a green report and a blind cluster.

terminal
$ kube-bench run --targets master | grep audit-log
[PASS] 1.2.22 Ensure that the --audit-log-path argument is set (Automated)
[PASS] 1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
[PASS] 1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
[PASS] 1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

kube-bench checks the flags against that benchmark and nothing more. It can't see whether your policy behaves the way you wrote it, so it will happily pass a cluster whose policy logs everything or nothing. That gap between a passing report and a policy that actually works is where most teams pick up a false sense of safety. For the real test you have to read the log. Trigger a Secret read and an exec by hand, then check what level each one landed at, and whether any request body got written to disk.

terminal
$ kubectl get secret db-creds -n payments >/dev/null        # generate a read
$ kubectl exec -it payments-api-7d9 -n payments -- sh        # generate an exec, then exit
$ tail -n 300 /var/log/kubernetes/audit.log \
    | jq -c 'select(.objectRef.resource=="secrets" and .verb=="get")
             | {level, user:.user.username, body:(has("responseObject"))}' | tail -1
{"level":"Metadata","user":"alice","body":false}
 
$ tail -n 300 /var/log/kubernetes/audit.log \
    | jq -c 'select(.objectRef.subresource=="exec")
             | {level, verb, uri:.requestURI}' | tail -1
{"level":"Metadata","verb":"create","uri":"/api/v1/namespaces/payments/pods/payments-api-7d9/exec?command=sh&stdin=true&tty=true"}

The Secret read shows body:false, so its contents never touched the log. You know alice read db-creds; you don't learn what it held. If that field came back true, you'd stop and fix the policy before shipping one more event, because every forwarded record would be carrying plaintext you'd then have to scrub out of the SIEM. The exec event puts the command right there in the URI, which is the part you actually need, and it is also all you can get. An exec upgrades into a streaming connection, so there is no request or response body for the API server to write down: RequestResponse on pods/exec hands you these same fields at several times the storage cost, because requestURI is already recorded at Metadata. What gets typed into that shell is Falco's job, not the audit log's. Both checks behaved, then. What none of this tells you is whether the events ever left the node, so read the backend's own counters next.

terminal
$ kubectl get --raw /metrics \
    | grep -E '^apiserver_audit_(event|error|requests_rejected)_total'
apiserver_audit_event_total 48213
apiserver_audit_error_total{plugin="buffered"} 0
apiserver_audit_requests_rejected_total 0

Read those three for what they are, not for what you hope. apiserver_audit_event_total counts the events your policy generated, not events any backend delivered. apiserver_audit_requests_rejected_total counts API requests the server refused because an audit backend errored, which is a different failure entirely. The one that moves when the buffered webhook throws events away, because the SIEM is slow or simply gone, is apiserver_audit_error_total, labeled by the plugin that lost them. That is the deal you made when you chose batch mode: a dead SIEM never slows a single kubectl command, so nothing looks wrong from the cluster side until you look here. And the only end-to-end proof is the round trip. Run one exec you'll recognize, then go and find it in the SIEM and time how long that takes. If the answer is hours, or it never arrives, the control won't help you during an incident even though every byte is sitting on the node.

How a request becomes an audit event
1API requestverb + resource + identity2match rulestop to bottom, first match wins3assign levelNone / Metadata / Request /…4emit eventto log file + webhook
None short-circuits: a request that matches a None rule is never written, and evaluation stops there. Order is everything. Put the drop rules for kube-proxy and kube-system chatter first. A broad Metadata catch-all placed above them swallows every request, and the tighter None rules below it never run.

From alert to answer

Here's where it pays off on a bad day. Falco fires: unexpected shell in payments-api. You pivot to the audit log, scope it to that namespace and the few minutes around the alert, and ask two questions. Which RBAC (Role-Based Access Control) identity ran the exec? And what did that same identity touch right after? Same identity, three namespaces, three seconds apart reads as automated credential sweeping, not a human. That's the jump from 'something happened somewhere' to a named subject and a blast radius you can actually measure: revoke that one service account and rotate exactly the Secrets it reached, instead of guessing.

terminal
$ jq -c 'select(.objectRef.namespace=="payments" and .verb=="create"
       and .objectRef.subresource=="exec")
       | {who:.user.username, pod:.objectRef.name, when:.requestReceivedTimestamp}' \
    /var/log/kubernetes/audit.log
{"who":"system:serviceaccount:ci:deployer","pod":"payments-api-7d9","when":"2026-07-16T15:02:11Z"}
 
$ jq -c 'select(.user.username=="system:serviceaccount:ci:deployer"
       and .objectRef.resource=="secrets" and .verb=="list")
       | {ns:.objectRef.namespace, when:.requestReceivedTimestamp}' \
    /var/log/kubernetes/audit.log
{"ns":"payments","when":"2026-07-16T15:02:40Z"}
{"ns":"billing","when":"2026-07-16T15:02:41Z"}
{"ns":"identity","when":"2026-07-16T15:02:43Z"}
A log you never read is cost, not security
RequestResponse on everything floods storage, drags API-server latency, and writes Secret values and bearer tokens straight into a file you then forward to your SIEM. Now that plaintext lives in two more places than it ever should. Reserve full capture for the handful of verbs that genuinely need it. Keep sensitive resources at Metadata so you never copy a Secret to disk, and let routine traffic stay at None. And remember the one blind spot: the audit log only sees the API server. A read straight from etcd, or a request to the kubelet on its own port, never shows up in it.

Storage is part of the control

Here is the part that gets skipped once the flags pass. The audit log is now a data store, and a data store needs an owner, a lifetime, and a lock. The file on the node is root-only for a reason, and the bucket or index it lands in deserves the same care, because an audit archive every engineer can read is a searchable map of which identity reached which Secret in which namespace, handed over to whoever phishes one laptop. So decide who can read it and how long you keep it at the same time you decide what to record, and keep the verification query and its expected output in the same pull request as the policy, so the next person on call can tell pass from fail without inventing a jq filter at three in the morning.

Try this

Enable a focused audit policy on a lab API server, exec into a pod, and find the matching audit event by user and resource. Watch which identity turns up: it's whoever your kubeconfig says you are, not the pod's service account. And if the grep comes back empty, that's the lesson rather than a broken cluster. A request that matches no rule at all is written nowhere, so a policy with a gap in it and a policy that was never loaded look identical from the outside.

terminal
$ sudo cat /etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources: [{ group: "", resources: ["secrets"] }]
- level: Metadata                       # your exec matches here, whoever you are
  resources: [{ group: "", resources: ["pods/exec"] }]
 
$ kubectl -n payments exec deploy/payments-api -- id
uid=10001 gid=10001 groups=10001
 
$ sudo grep '"subresource":"exec"' /var/log/kubernetes/audit.log \
    | tail -1 | jq -c '{level, who:.user.username, pod:.objectRef.name}'
{"level":"Metadata","who":"kubernetes-admin","pod":"payments-api-7d9f8c6b4-2xk9p"}

Takeaway

API audit logs answer who asked the control plane to do what. Pair them with runtime alerts so exec and secret reads become a timeline.

Quick check
01Your policy opens with a broad level: Metadata catch-all rule, and below it sits a level: None rule meant to drop kube-proxy's noisy watch traffic on endpoints and services. What actually happens to that watch traffic?
Incorrect — No. Rules are first-match-wins, top to bottom. The Metadata catch-all matches the watch request first, so the None rule below it is never reached.
Correct — First match wins, so a broad rule on top swallows everything and the specific drop rule under it never fires. Put your None rules first.
Incorrect — No. Evaluation stops at the first matching rule; a request is recorded at exactly one level.
Incorrect — No. The policy parses and loads fine. The ordering just produces a silently useless result.
02The lesson keeps Secrets and ConfigMaps at level: Metadata rather than Request or RequestResponse. Why?
Incorrect — every level above None records the user; the reason to cap Secrets is about the body, not the identity.
Correct — logging a Secret at Request or higher writes its contents into the audit file and then your SIEM, which is exactly what Metadata avoids.
Incorrect — they can be, and that is the danger; the policy deliberately chooses Metadata to avoid capturing the data.
Incorrect — the driver is avoiding plaintext Secret contents on disk, not throughput.
03kube-bench reports [PASS] on every --audit-log flag. A teammate concludes the cluster is now capturing the security-relevant events. Are they right?
Incorrect — the flags being set says nothing about what your policy captures or at which levels.
Incorrect — kube-bench checks only that the flags exist; it never reads or evaluates the policy's behavior.
Correct — a green benchmark can sit on a policy that logs nothing useful, so prove it twice: flags pass, then real events verify.
Incorrect — kube-bench does run against the control plane; the gap is that it grades flags, not policy behavior.
Container immutability at runtimeFinish · self-test

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep