>_SecOpsLog_
CoursesKubernetes attack & defenseDetecting cluster attacks

Detecting cluster attacks

Audit log, runtime rules, honeytokens.

Advanced35 min · lesson 14 of 15

A motion sensor nobody wired to an alarm is just a plastic box on the wall. Kubernetes ships with two of them, and both are off by default. The API server, the front desk that every command in the cluster passes through, can record who did what. Runtime tools can watch what actually happens inside your running containers. Neither one saves you unless something is watching the feed and knows what a break-in looks like. Prevention locks the doors you know about. Detection is how you find out that someone walked through a door you forgot to lock.

Switch on the audit log

Think of the API server as a front desk that every request in the cluster has to walk up to. Create a Pod, read a Secret (a Secret is the Kubernetes object that stores passwords, tokens, and keys), bind a new role, open a shell in a container: all of it arrives as a request at that one desk. The audit log is the desk's logbook. It records who asked, what they wanted, the source address the request came from, and whether it was allowed. The catch is that the logbook starts empty, and it stays empty until you hand the API server a Policy that spells out what to write down. Each rule in that Policy picks a level. None throws the event away. Metadata logs the headline: who, what, when, from where. Request adds the object the user submitted. RequestResponse piles the server's full reply on top of that.

/etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  # Full request body for the attack-adjacent writes.
  - level: Request
    verbs: ["create"]
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["clusterrolebindings", "rolebindings"]
  # Exec and attach: capture who ran what, where.
  - level: Request
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach"]
  # Secrets at Metadata only (never log the payload).
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
  # Drop the high-volume noise so the log stays readable.
  - level: None
    resources:
      - group: ""
        resources: ["events", "endpoints"]

You wire the API server to that file with two flags, set in its own manifest at /etc/kubernetes/manifests/kube-apiserver.yaml: --audit-policy-file and --audit-log-path. Where that Policy lives matters more than it looks. It's a plain file on the control-plane node, the machine that runs the cluster's brain, not an object inside Kubernetes. So an attacker with ordinary API access can't run kubectl delete to switch your logging off and slip into the dark. Someone who gets a foothold on the control-plane machine itself can, which is the whole reason you copy every line off the box the moment it's written. The API server records each event late, once the request has finished and the outcome is known, so the log tells you not just what was attempted but whether it worked. Don't trust the config file to prove logging is on. Read the running process.

check the running API server
ps -ef | grep '[k]ube-apiserver' | grep -o -- '--audit-[^ ]*'
output
--audit-policy-file=/etc/kubernetes/audit/policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=30
--audit-log-maxbackup=10

Every entry is a single line of JSON (a plain-text format of key-value pairs). A few fields carry almost all the meaning. user.username tells you who. verb and objectRef.resource say what they touched. sourceIPs shows where the request came from. An exec shows up as the resource pods with the subresource exec, and the command the person ran rides along inside requestURI. Pull just the exec events out of the pile and the difference between routine and hostile jumps out.

read exec events from the audit log
jq -c 'select(.objectRef.subresource=="exec")
  | {t:.requestReceivedTimestamp, user:.user.username,
     ns:.objectRef.namespace, pod:.objectRef.name,
     ip:.sourceIPs[0], uri:.requestURI}' \
  /var/log/kubernetes/audit/audit.log
output
{"t":"2026-07-16T14:22:07Z","user":"[email protected]","ns":"staging","pod":"web-7c9d8","ip":"203.0.113.8","uri":"/api/v1/namespaces/staging/pods/web-7c9d8/exec?command=sh&container=web&stdin=true&tty=true"}
{"t":"2026-07-16T03:12:07Z","user":"system:serviceaccount:kube-system:coredns","ns":"payments","pod":"payments-db-0","ip":"10.8.1.44","uri":"/api/v1/namespaces/payments/pods/payments-db-0/exec?command=sh&stdin=true&tty=true"}

Read those two lines the way a night guard reads a logbook. The first is a named engineer, jane, opening a shell in staging in the middle of the afternoon from an office address. Boring. Exactly what you want. The second is a service account, and that phrase is worth a second: a service account is the identity Kubernetes hands to a program instead of a person. This one is coredns, the pod that runs the cluster's internal phone book, turning names like payments-db into addresses. It has one job, and opening a shell is not it. Yet here it is, dropping into the payments namespace at three in the morning from an address inside the pod network. Same verb, exec, and a completely different story. The tells stack up fast: the actor is a machine that should never do this, the target is where the money lives, the source is inside the cluster, and the hour is when nobody's watching.

How an exec becomes an alert
1kubectl execsomeone asks for a shell2Auth checkwho are you, are you allowed?3Admissionpolicies get a vote4Audit eventwritten once the request…5Backenda log file or a webhook6Central storeappend-only, off the cluster7Alerta rule matches, someone gets…

When the audit log goes blind

The audit log stops at the container's edge. It will faithfully tell you that jane ran exec with the command sh. It cannot see one keystroke she types after that shell opens. Worse, it never sees a process that a compromised container starts on its own, because nothing ever asked the API server for permission. The audit log is the badge reader on a door: it logs the swipe, and it has no clue what you do once you're inside the room. For that you need a camera in the room. Falco is that camera. It's an open-source tool that hooks into the Linux kernel through eBPF (extended Berkeley Packet Filter, a safe way to run small inspector programs down where the operating system itself lives) and matches live system calls against rules like this one.

/etc/falco/rules.d/shell.yaml
- rule: Shell In Container
  desc: A shell was spawned inside a running container
  condition: >
    spawned_process and container
    and proc.name in (shell_binaries)
    and not container.image.repository in (allowed_shell_images)
  output: >
    Shell in container (user=%user.name pod=%k8s.pod.name
    ns=%k8s.ns.name image=%container.image.repository
    proc=%proc.name parent=%proc.pname cmd=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution, T1059]

Load that rule and the instant a shell starts inside any container, Falco writes a line. No API call needed, because it isn't watching the API. It's watching the kernel. Here's what it caught at eleven minutes past three that morning, inside a payments pod.

Falco alert (stderr)
03:11:41.882 Warning Shell in container (user=root pod=payments-api-5f7c9
ns=payments image=registry.internal/payments-api proc=sh
parent=payments-server cmd=sh -i)

Look at the parent field: payments-server, which is the application itself. A normal shell in a container has a parent like the container runtime, the plumbing that runs when someone types kubectl exec. This shell's parent is the app process, which means the running program spawned its own shell and no human asked it to. That's the shape of a reverse shell: a compromised program opens an outbound connection back to the attacker and hands them a command line inside your container, like a burglar who props a back window open and phones a friend to climb through. The audit log shows nothing for this, because the app never made an API request. Only the camera in the room saw it. One more trap earns its keep here. A honeytoken is a decoy you would never touch on purpose, like a marked twenty left on the dresser to find out if the cleaner takes it. You create a Secret that nothing in the cluster ever reads, give it a name too tempting to skip, and raise an alarm on any read at all. Real workloads never look at it, so a single get is about as close to proof as detection gets: someone is going through your drawers for credentials. Wire it to the audit log and you also learn which identity and which address reached for the bait.

plant a canary Secret, then hunt reads of it
kubectl -n payments create secret generic aws-prod-creds \
  --from-literal=access_key_id=AKIA0000EXAMPLE \
  --from-literal=secret_access_key=wJalrEXAMPLEKEY
 
jq -c 'select(.objectRef.resource=="secrets"
  and .objectRef.name=="aws-prod-creds" and .verb=="get")
  | {t:.requestReceivedTimestamp, who:.user.username, ip:.sourceIPs[0]}' \
  /var/log/kubernetes/audit/audit.log
output
secret/aws-prod-creds created
{"t":"2026-07-16T03:12:58Z","who":"system:serviceaccount:kube-system:coredns","ip":"10.8.1.44"}

Now stand back and line the timestamps up. At 3:11:41 the payments-api app spawned a shell on its own, which Falco caught and the audit log could not. Cross-check the audit log for a matching pods/exec on that pod and you find nothing, which is the point: nobody opened that shell through the front desk. Seconds later the same pod's traffic, from 10.8.1.44, starts acting as the coredns service account. At 3:12:07 that identity execs into the database pod. At 3:12:58 it reads a canary Secret it has no reason to know exists. A coredns token showing up from a payments pod address is a token that has been stolen and replayed, because coredns doesn't run there and never will. Any one of these alerts on its own is a maybe. Stacked together across two sensors, they tell one story: a foothold in payments-api, a stolen token, and a hunt for cloud keys, all inside about a minute. That stacking is the entire reason you run both layers instead of picking one.

Logging Secrets at full detail turns your audit log into a Secret store
It's tempting to set the secrets rule to RequestResponse so you can see exactly what got read. Don't. At that level the API server writes the object's whole body into the log, which means every Secret read copies its plaintext keys and passwords straight into audit.log. Now anyone who can read that log can read your credentials, and that's a wide group once you ship the log to a SIEM (Security Information and Event Management platform, the central system that collects logs and fires alerts) half the company can query. Keep Secrets at Metadata level. You still learn who read which Secret and when, without cloning the payload into a second place an attacker would be thrilled to find.
Quick check
01Falco fires 'shell in container' for a pod in payments. You cross-check the API audit log. Which finding most strongly says intrusion rather than routine operations?
Incorrect — Your own image is supposed to be there. The image name says nothing about who started the shell inside it, so this is a non-signal.
Incorrect — That's the signature of a normal kubectl exec: a human opened the shell through the front desk, on the record. This is the least suspicious of the four.
Correct — A live shell with no matching API exec means nobody opened it through the API server. The app process spawned it itself, which is exactly what a reverse shell or code-execution exploit looks like.
Incorrect — Priority is a label you chose in the rule. It reflects your tuning, not how real the intrusion is, so it can't confirm anything.
02Your audit Policy logs Secrets at level Metadata. A colleague suggests bumping that rule to RequestResponse 'so we can see exactly what was read.' Why does the lesson tell you not to?
Correct — Metadata records who read which Secret and when without cloning the payload, whereas RequestResponse turns the audit log into a second copy of your credentials.
Incorrect — RequestResponse is a real level, and the objection is about what it records, not that it is invalid.
Incorrect — it records more, not less, and still captures the actor along with the sensitive payload you did not want logged.
Incorrect — the opposite holds, since RequestResponse is the less secure choice because it leaks the Secret contents into the log.
03You plant a canary Secret named aws-prod-creds that no workload in the cluster is ever supposed to read, and wire an audit filter for get requests against it. Overnight the filter returns exactly one hit: system:serviceaccount:kube-system:coredns from an in-cluster address. How should you read this?
Incorrect — the point of a decoy nothing legitimately reads is that a single get is near-proof, not noise.
Incorrect — the values are fake and unused, and the signal is the read of the decoy itself, not any use of its contents.
Correct — coredns runs the cluster's DNS and never reads Secrets, so its identity touching the bait means the token was stolen and replayed.
Incorrect — coredns has one job and reading Secrets isn't it, so a system service account acting outside its role is exactly the anomaly you're hunting.

When one of these fires for real, the time for tuning rules is over and a different clock starts. You have to freeze the pod without tipping off whoever's inside it, grab the token they used, and map everything that token can still reach before they get there first. That's a different job from watching the feed, and it's exactly where the next lesson picks up.

Try this

Work through “When the audit log goes blind” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: logging Secrets at full detail turns your audit log into a Secret store. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Lateral movementKubernetes incident response

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningKubernetes interview questionsInterview prep