Audit logs

An audit policy that records what matters.

Falco sees the host and the syscalls; the Kubernetes audit log sees the API server and who asked it to do what. Together they answer different halves of an investigation: Falco says “a shell opened in payments-api,” the audit log says “this identity exec’d into that pod at 15:02 and also read three secrets.” The audit log is a structured, chronological record of every request the API server handled — but only if you configure a policy, because with no policy nothing is logged.

An audit policy is a list of rules evaluated top-down, first match wins, each assigning a level: None (drop it), Metadata (who, what, when, from where — no bodies), Request (adds the request body), or RequestResponse (adds both bodies). The art is logging enough to reconstruct an incident without drowning in noise or writing secret contents to disk — so sensitive resources get Metadata, high-value verbs like exec get full detail, and routine chatter gets None.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: None                            # drop kubelet & probe noise first
  users: ["system:kube-probe"]
- level: None                            # drop kube-system service-account chatter
  userGroups: ["system:serviceaccounts:kube-system"]
- level: Metadata                        # secrets: record access, not contents
  resources: [{ group: "", resources: ["secrets", "configmaps"] }]
- level: RequestResponse                 # full detail for exec / attach
  resources: [{ group: "", resources: ["pods/exec", "pods/attach"] }]
- level: Metadata                        # everything else, minimally

Wire the policy into the API server and point it at a log file, with rotation limits so it cannot fill the disk. For real detection you send it onward — a webhook backend streams events to a SIEM — and you always ship it off the node, because a compromised node’s local audit log is the first thing an attacker edits.

- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/var/log/kubernetes/audit.log
- --audit-log-maxage=30            # days to keep
- --audit-log-maxbackup=10         # rotated files to keep
- --audit-log-maxsize=100          # MB per file before rotating
# (mount the policy file + log dir into the static pod via volumes)

From alert to answer

The payoff is a tractable investigation. When Falco fires, you pivot to the audit log filtered to that namespace and pod to see which identity created or exec’d into it and what else that identity touched in the same window. That is the difference between “something happened” and “the ci:deployer token exec’d into payments-api and then listed secrets in three namespaces.”

$ jq 'select(.objectRef.namespace=="payments" and .verb=="create"
       and .objectRef.subresource=="exec")
       | {who:.user.username, what:.objectRef.name, when:.requestReceivedTimestamp}' \
    /var/log/kubernetes/audit.log