BlogKubernetes

Turn on the Kubernetes audit log and actually read it

Write a Kubernetes audit policy that captures the right events at the right level, then query the log to find exactly who deleted that deployment at 2am.

Sep 24, 2024·11 min readAdvanced

When someone asks 'who deleted that deployment?', the Kubernetes audit log is the only thing that can answer — but it is off by default. You give the API server an audit policy (what to record, at what detail) and a backend (where to write it), then you can actually reconstruct what happened.

A focused audit policy

audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# full request/response for secrets + RBAC changes
- level: RequestResponse
resources:
- group: ""
resources: ["secrets"]
- group: "rbac.authorization.k8s.io"
# metadata for everything else that writes
- level: Metadata
verbs: ["create", "update", "patch", "delete"]
# drop the noise
- level: None
resources: [{ resources: ["events"] }]

Wire it up with --audit-policy-file and --audit-log-path on the API server.

Answer the 2am question

bash — who deleted itlive
cat audit.log | jq 'select(.verb=="delete" and (.objectRef.resource=="deployments"))'
user: jenkins-deployer objectRef: shop/api stage: ResponseComplete
name, time, and source IP, all recorded
RequestResponse is expensive
Logging full request bodies for everything will flood your storage. Reserve RequestResponse for sensitive resources and use Metadata for the rest.
Go deeper in a courseKubernetes security & hardeningAudit logging, RBAC, and the rest of CKS, hands-on.View course

Related posts