BlogKubernetes
Turn on the Kubernetes audit log and actually read it
Write a Kubernetes audit policy that captures the right events at the right level, then query the log to find exactly who deleted that deployment at 2am.
When someone asks 'who deleted that deployment?', the Kubernetes audit log is the only thing that can answer — but it is off by default. You give the API server an audit policy (what to record, at what detail) and a backend (where to write it), then you can actually reconstruct what happened.
A focused audit policy
audit-policy.yaml
apiVersion: audit.k8s.io/v1kind: Policyrules:# full request/response for secrets + RBAC changes- level: RequestResponseresources:- group: ""resources: ["secrets"]- group: "rbac.authorization.k8s.io"# metadata for everything else that writes- level: Metadataverbs: ["create", "update", "patch", "delete"]# drop the noise- level: Noneresources: [{ resources: ["events"] }]
Wire it up with --audit-policy-file and --audit-log-path on the API server.
Answer the 2am question
cat audit.log | jq 'select(.verb=="delete" and (.objectRef.resource=="deployments"))'user: jenkins-deployer objectRef: shop/api stage: ResponseCompletename, time, and source IP, all recordedRequestResponse is expensive
Logging full request bodies for everything will flood your storage. Reserve RequestResponse for sensitive resources and use Metadata for the rest.