Turn on the Kubernetes audit log and actually read it

Write a Kubernetes audit policy that captures the right events at the right level, then query the log to find exactly who deleted that deployment at 2am.

When someone asks who deleted that Deployment at 2am, etcd cannot help you — it only holds current state. The Kubernetes audit log is the API server's record of every request: who called it, from where, what object, what verb, and whether it succeeded. It is off by default because a naive policy generates terabytes of noise. The work is writing an audit policy that captures security-relevant events at the right detail level, wiring a backend that survives restarts, and knowing how to query the result.

Managed clusters (EKS, GKE, AKS) expose audit logging differently — often as a control-plane setting you toggle rather than flags on kube-apiserver. Self-managed clusters pass --audit-policy-file and --audit-log-path (or webhook backends) directly. Either way, the policy levels are the same: None, Metadata, Request, and RequestResponse. Reserve RequestResponse for Secrets and RBAC; use Metadata for everything else you care about. The Kubernetes security (CKS) track covers audit policy alongside RBAC and admission.

Write a focused audit policy

Rules are evaluated in order; the first match wins. Log full request and response bodies for Secret access and RBAC changes — that is where credential theft and privilege escalation show up. Log metadata (user, verb, object ref, source IP) for other mutating operations. Explicitly drop high-volume noise like Events and read-only list/watch on well-known resources.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
      - group: "rbac.authorization.k8s.io"
  - level: Metadata
    verbs: ["create", "update", "patch", "delete"]
  - level: None
    users: ["system:kube-proxy", "system:kubelet"]
  - level: None
    resources: [{ group: "", resources: ["events"] }]

Wire the policy on self-managed clusters via API server flags: --audit-policy-file=/etc/kubernetes/audit-policy.yaml and --audit-log-path=/var/log/kubernetes/audit.log. For production, prefer a webhook backend to a log shipper so audit events leave the control plane node immediately — local disk fills up faster than you expect during an incident.

Answer the 2am question

Audit events are JSON lines. Filter by verb, resource, namespace, and user. A complete delete shows stage: ResponseComplete with the user identity from the authentication layer — human, service account, or controller. Correlate sourceIPs with your VPN or bastion logs when the caller is unexpected.

Alert on the events that matter

Forensics after the fact is necessary but slow. Stream audit logs to your SIEM and alert on high-signal patterns: cluster-admin RoleBinding created, anonymous access succeeding, Secret enumeration from an unexpected namespace, or mass deletes in a short window. Tune alerts to service accounts and humans separately — CI deployers delete Deployments constantly; humans deleting ClusterRoles at 3am is different.

Where this goes next

Audit logs tell you what happened; RBAC and admission policy prevent it from happening again. Pair logging with least-privilege Roles for CI service accounts, Gatekeeper constraints that block risky bindings, and regular kubectl auth can-i audits. The Kubernetes security (CKS) path covers audit logging, RBAC, and runtime hardening together.

Related posts