>_SecOpsLog_
CoursesKubernetes attack & defenseKubernetes incident response

Kubernetes incident response

Cut identity, preserve, hunt persistence, trust.

Expert35 min · lesson 15 of 15

A Falco alert lands at 2 a.m. Falco is the tool that watches for weird behavior inside running containers, and right now it's telling you a shell just spawned inside a pod named payments-api-7c9. A shell means someone, or something, has a live command prompt running inside your production workload. Seconds later that same pod started looking up a web address your DNS logs (the record of every domain your systems try to reach) have never seen before. That's a live compromise. What you do in the next ten minutes decides whether you learn how they got in, or whether you delete the only copy of the evidence by accident.

Incident response on a cluster pulls you two ways at once. You want to stop the attacker this second, and you also want to understand what they did, and those two urges fight each other. Kill the pod and the network traffic stops, sure, but so does the process memory, the open connections, and whatever the attacker staged in /tmp. All gone. So work like a careful investigator, not a bouncer. Seal the room so nobody gets in or out. Photograph the scene before you touch a thing. Then change the locks. In Kubernetes that maps to four moves, and the order matters: isolate the pod, freeze the node, snapshot for forensics, and revoke the identity.

Seal the room without killing the witness

A NetworkPolicy is a firewall you write as a Kubernetes object. It picks pods by their labels and spells out what traffic is allowed in and what's allowed out. Most clusters ship wide open, allow-all by default, so a policy that selects your bad pod and permits nothing at all turns into a hard quarantine. No more talking to the database. No more beaconing home, which is the malware term for quietly calling back to the attacker's server. No reaching the pod running right next to it. And the pod keeps running, which is the whole point. It's boxed in but still alive, so its memory and its file system sit there waiting for you to inspect them. Leave out the ingress and egress rules entirely and you've said deny everything.

quarantine.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine-payments-7c9
  namespace: prod
spec:
  podSelector:
    matchLabels:
      quarantine: "isolate"
  policyTypes:
    - Ingress
    - Egress
terminal
kubectl label pod payments-api-7c9 quarantine=isolate -n prod --overwrite
kubectl apply -f quarantine.yaml
output
pod/payments-api-7c9 labeled
networkpolicy.networking.k8s.io/quarantine-payments-7c9 created

Now freeze the node. kubectl cordon marks a node unschedulable, so the scheduler quits placing new pods on it while you work. Resist the urge to reach for kubectl drain. Drain evicts every pod on the node, the compromised one included, and eviction deletes it. That's bulldozing the crime scene with the evidence still inside. Cordon holds the node still and touches nothing. Drain comes later, after you've captured everything worth keeping.

terminal
kubectl cordon node-17
kubectl get node node-17
output
node/node-17 cordoned
NAME      STATUS                     ROLES    AGE   VERSION
node-17   Ready,SchedulingDisabled   <none>   88d   v1.31.4
A NetworkPolicy is only as real as your CNI
Writing a deny-all NetworkPolicy does nothing at all unless your cluster runs a Container Network Interface (CNI) plugin that actually enforces it, like Calico or Cilium. Flannel and several managed defaults ignore NetworkPolicy completely, so the object gets created, kubectl cheerfully says success, and the pod keeps beaconing out the whole time. Test enforcement on a quiet afternoon, long before you need it. On a cluster that doesn't enforce, your fallback is to break the pod off its Service: change the label the Service selects on so the pod drops out of the endpoint list (the set of pod addresses the Service forwards traffic to), then lean on cordon plus a node-level firewall.

Photograph the scene

With the pod boxed in, grab its state before any of it changes. Three things matter here. First, the node's disk, so you hang on to the container file system and anything the attacker wrote outside it. That snapshot is a cloud operation on the underlying volume, an Elastic Block Store (EBS) snapshot on AWS or a Persistent Disk snapshot on Google Cloud, and you take it from the provider, not from kubectl. Second, the pod's logs, and pull the crashed prior instance too with --previous. Third, the audit log, which is how you piece together what the stolen identity actually did.

terminal
kubectl logs payments-api-7c9 -n prod --previous > payments-7c9.log
aws ec2 create-snapshot --volume-id vol-0ab1234cd5678ef90 --description "IR payments-7c9 node-17"
output
{
    "SnapshotId": "snap-0f3e9c7b1d2a4e5f6",
    "State": "pending",
    "VolumeId": "vol-0ab1234cd5678ef90",
    "StartTime": "2026-07-16T02:14:07+00:00"
}

The API audit log is the running record of every request the API server handled, who made it and what they asked for. If you don't have audit logging switched on, that's the finding to fix the moment this incident is over, because without it you're just guessing. Pull every call the compromised service account made and read it back as a timeline. jq lets you filter that stream of JSON down to one identity and just the fields you care about.

terminal
jq -c 'select(.user.username=="system:serviceaccount:prod:payments") | {t:.requestReceivedTimestamp, verb, res:.objectRef.resource, name:.objectRef.name}' /var/log/kubernetes/audit.log
output
{"t":"2026-07-16T01:52:03Z","verb":"list","res":"secrets","name":null}
{"t":"2026-07-16T01:52:44Z","verb":"create","res":"pods","name":"cryptominer-x"}
{"t":"2026-07-16T01:53:10Z","verb":"create","res":"clusterrolebindings","name":"cluster-admin-backdoor"}

There's the entire story in three lines. The token listed every Secret in the namespace, launched a miner, then wrote itself a ClusterRoleBinding for persistence so it holds cluster-admin even after you kill the pod. A ClusterRoleBinding is the object that grants a role to an identity across the whole cluster, and cluster-admin is the top one, full control over everything. That binding just went on your eradication list.

Change the locks

Now you revoke the identity, and this is the part people get wrong. A service account (SA) is the identity a pod uses to talk to the Kubernetes API. Modern clusters hand each pod a bound token: a signed credential, like a temporary keycard, that expires on a timer and is stapled to that one specific pod and SA. The API server honors it only while the pod still exists and the SA's internal ID (its UID, a unique fingerprint) still matches. Older clusters also carried long-lived Secret tokens that never expire at all. The two kinds need different handling.

Deleting the pod invalidates a bound token, because the token is stapled to that pod. Deleting the service account hits harder: it kills every token that SA ever handed out, because the API server checks that the SA still exists with a matching UID, and a deleted or recreated SA no longer matches. For a legacy Secret token, you delete the Secret it lives in. And that cluster-admin-backdoor binding you spotted in the audit log gets deleted now, before the attacker uses it again.

terminal
kubectl delete clusterrolebinding cluster-admin-backdoor
kubectl delete serviceaccount payments -n prod
kubectl delete secret payments-legacy-token -n prod
output
clusterrolebinding.rbac.authorization.k8s.io "cluster-admin-backdoor" deleted
serviceaccount "payments" deleted
secret "payments-legacy-token" deleted

Only now, with the evidence captured and the identity dead, do you drain the node, delete the pod, and rebuild the workload from an image you trust. Then you ask the hardest question last: how deep did they actually get? If the attacker only ever held this one pod's token, you're mopping up a contained mess and you can breathe. If they reached the control plane, or etcd (the cluster's key-value database, where every object and every Secret is stored), or the cluster's certificate authority (CA, the root of trust that signs every identity), then they can forge brand-new identities that survive everything you just did. Deleting bindings won't stop someone who can sign with your CA. That case means rotating the CA, rotating every credential, and rebuilding the cluster from the ground up. Painful, and sometimes it's the only honest call. So rehearse this whole sequence as a drill on a calm day, and the real 2 a.m. version becomes a runbook you follow instead of something you invent while your heart pounds.

Diagram
1Isolatedeny-all NetworkPolicy, label…2Freezecordon the node; do not drain…3Snapshotnode disk, pod logs (--previous…4Reconstructaudit log timeline filtered by…5Revokedelete the SA, attacker's…6Eradicatedrain, rebuild from clean images7Decide trustcontrol plane / etcd / CA…
Quick check
01A pod is compromised and its bound service-account token was stolen. Your reflex is kubectl delete pod. Why hold off?
Incorrect — It does invalidate the token, but that's a reason it feels tempting, not a reason it's safe. You still lose the evidence.
Correct — Eviction or deletion removes the container and everything live inside it. Quarantine with a NetworkPolicy, snapshot, then delete.
Incorrect — A NetworkPolicy matches a running pod by label selector and applies immediately. You never recreate the pod to quarantine it.
Incorrect — A node or volume snapshot is a separate cloud resource and isn't touched by deleting a pod.
02During the 'change the locks' step, why does deleting the compromised service account revoke access more completely than deleting the pod?
Incorrect — deleting the pod does invalidate that pod's bound token; it simply doesn't reach the other tokens the account issued.
Incorrect — that conflates two things, since account deletion never touches the CA, which is the separate and heavier trust-rebuild case.
Correct — a bound token is tied to one pod, but the account's unique ID (UID) backs all its tokens, so removing it breaks every one at once, and a legacy Secret token additionally needs its Secret deleted.
Incorrect — they differ in blast radius, which is exactly why the lesson distinguishes them.
03Your audit timeline shows the stolen token ran: list secrets, then create a pod, then create a ClusterRoleBinding granting cluster-admin. After you quarantine and delete the pod and delete its service account, why must you still explicitly delete that ClusterRoleBinding?
Correct — the binding outlives the identity that made it and may grant top-level access to something the attacker still controls, so it stays a live backdoor until deleted.
Incorrect — Kubernetes does not cascade-delete bindings when an account is removed, so the binding persists on its own.
Incorrect — the binding is not tied to the pod's lifecycle and remains fully effective after the pod is gone.
Incorrect — a NetworkPolicy governs traffic, not RBAC grants, so it does nothing to a cluster-admin binding.

Try this

Run kubectl label pod payments-api-7c9 quarantine=isolate -n prod --overwrite on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a NetworkPolicy is only as real as your CNI. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Detecting cluster attacksFinish · self-test

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningKubernetes interview questionsInterview prep