Runtime incident response

Isolate, preserve, hunt persistence, recover.

A burglar alarm goes off at two in the morning and the instinct is to run in and start tidying up. That is how fingerprints get wiped. When a runtime alert turns out to be a real compromise, you are racing someone whose foothold sits inside a pod that Kubernetes may reschedule out from under you at any second. What wins is a runbook you have already rehearsed: contain the attacker without shredding the evidence, work out how far they got, pull out every foothold, then bring the service back on purpose rather than by reflex.

A few terms, in plain words. IR is incident response, the sequence you follow once something bad is confirmed rather than suspected. On one physical server the compromised machine is the crime scene. In Kubernetes the scene is smeared across the cluster. Persistence, meaning the arrangements an attacker makes so they are still there tomorrow, can hide in a DaemonSet (a Kubernetes object that runs one copy of a pod on every node), in RBAC (role-based access control, the rules that decide who may do what), in an admission webhook (a service the API server calls to approve or rewrite every new object), or in files written straight onto a node's disk. Not only in the pod that tripped the alert. The order never changes: contain, preserve evidence, investigate, eradicate, recover.

Contain first, preserve second

Before any of that, write down who gets called. Platform team, the team that owns the application, cloud security, and legal. A runtime page lands on the platform on-call first almost every time, and that person can tell you which node is unhappy but usually cannot decide whether the company can absorb ten minutes of downtime on payments. Put the phone numbers in the runbook, not in one person's head.

When it is over, do not close the ticket until two pieces of work are filed: one detection that would have caught this sooner, and one control that would have stopped it outright. Response that only restores service teaches your team to sit through the same film again next quarter.

Run the commands on a staging cluster until they feel boring. Boring is what survives adrenaline.

Now the actual move. Deleting the pod feels like doing something, and it destroys the running process list and the memory you needed to read. Isolate instead. Put a quarantine label on the pod and let a deny-all NetworkPolicy (the Kubernetes rule that decides which pods may talk to which) match that label, so the workload can neither phone home nor take orders while it stays alive for you to study. If there is any sign the attacker got out of the container and onto the machine, cordon the node as well, which marks it so the scheduler stops placing new pods there. Then collect: container logs, the record of who ran an exec session against it, the Falco and Tetragon events filtered to that pod's UID (unique identifier), and a disk snapshot of the node taken through your cloud provider's API. Eviction happens after the evidence is somewhere safe.

kubectl logs payments-api-7c9 -n prod --previous > ir-8891-pod.log
# export Tetragon events for pod UID
# snapshot node volume via cloud API
kubectl delete pod payments-api-7c9 -n prod --grace-period=0

Follow the trail the kernel left

Build a timeline. Falco and Tetragon both emit JSON events, one line per interesting thing the kernel saw, so you can sort by timestamp and read the story straight through: the first shell that appeared inside the container, the files it opened, the outbound connections it tried. Lay the Kubernetes audit log beside it, because that answers what the kernel cannot. Who changed the Deployment's image? Which ServiceAccount tokens were minted? Who opened an exec session, and at what minute? All of this only holds up if the logs left the node the instant they were written. Evidence stored on a box the attacker owns is not evidence.

Hunt the spare keys

A burglar with any sense does not leave through the door they came in. They leave a window off the latch on the way out. Expect several: a new DaemonSet running quietly on every node, a binding that hands cluster-admin to an account nobody watches, a mutating webhook that re-adds the attacker's container after you delete it, a hostPath mount (a volume that maps a directory of the node's own filesystem into a pod) sitting in a namespace unrelated to the one that alerted, a cron entry on the node itself, a tampered image parked in your registry waiting for the next deploy. Search the whole cluster, not the one namespace where the alert fired.

kubectl get clusterrolebinding -o json | jq -r '.items[] | select(.subjects[]?.name|test("payments")) | .metadata.name'
 
# example output:
payments-admin-binding
suspicious-exec-binding

Pull the footholds, rotate the keys

Delete the workloads and bindings that have no business existing. Then rotate every credential that pod could reach: database passwords, the cloud role it could assume through the instance metadata service (the address a virtual machine queries to collect temporary cloud credentials), and any Kubernetes Secret mounted into it. If there is a believable story where the attacker touched the host, rebuild the node from a known-good image instead of cleaning it. Close the hole they came through too, whether that was a misconfiguration or an unpatched flaw, because rotating secrets around an open door buys you about a week.

Put the node back to work

kubectl uncordon node-17
 
# example output:
node/node-17 uncordoned

Chain of custody and the clocks you cannot pause

Evidence is worth something only if you can show nobody edited it. Hash each exported Falco JSON file and each disk image at the moment you export it, record those hashes somewhere separate, and keep the copies in a WORM bucket (write once, read many, meaning the storage itself refuses to overwrite or delete for a fixed period). Watch the calendar as well. Breach notification deadlines in many regimes start counting from discovery, and if your alert pipeline is wired properly the runtime alert's timestamp is your discovery time. Hold the post-incident review inside five business days, while people still remember what they typed, and let it change the runbook, the noisy rules and the admission policy, so the same escape route is shut the next time somebody tries it.

sha256sum ir-8891-pod.log
 
# example output:
a3f8c2...  ir-8891-pod.log

Rehearse a region failover as part of the drill. There is a nasty pattern where the attacker sets off something loud in one region, everyone piles into containment there, and the foothold waiting in your DR cluster (disaster recovery, the standby copy of production) wakes up while nobody is looking at it.

Field notes from real clusters

Evidence in a container evaporates faster than most people expect. The instant a pod dies its writable layer goes with it, the process table is gone, and anything the attacker held in memory is unrecoverable. If the Deployment restarts on failure, Kubernetes may do that for you while you are still typing. Treat the first two minutes as collection time rather than cleanup time.

Split the work between two people. One handles containment, the label and the policy and the cordon, while the other pulls evidence in parallel: describe and logs output for the pod, the node agent's own logs, Hubble flows if you run Cilium (Hubble is Cilium's flow viewer, the record of which pod talked to which), the Falco and Tetragon event stream, and a forensic disk snapshot if your team owns that tooling. Collecting one thing at a time is how you lose the pod halfway through.

The runtime trail answers four questions. Which binary ran? Which files did it open? Where did it connect? Which ServiceAccount token was used, and by what process? Put the deploy timeline next to those answers, because a compromise that begins ninety seconds after a new image rolled has told you where to look. Resist writing "Kubernetes was breached" in the incident doc when the trail points at one poisoned dependency in one image.

The footholds people miss are the mundane ones. A cron entry inside the container. A sidecar nobody on the team added. A rootkit on the node, if there is reason to believe the attacker got out of the container. Credentials belonging to CI (continuous integration, the pipeline that builds and ships your images), which is an elegant way to reintroduce malware through the front door on the next merge. The cloud token the pod could fetch from the metadata service. Response that ends at "pod deleted" left a key under the mat.

Rebuild from an image digest you can name out loud, rotate every secret the workload could read, patch the misconfiguration or the CVE (Common Vulnerabilities and Exposures, the public catalog number for a known flaw) that let them in, and invalidate live sessions. Write the rotation down as you go. The sequel incident, three weeks later, is almost always one shared credential nobody ticked off the list.

Recovery does not end the watching. Keep the rule that caught this pinned to a dashboard for a few days, and if you trust your enforcement path, turn it up for a while. Tell the service owners what you know and what you do not, in plain sentences, with no theater. Book the review before everyone scatters.

Notification clocks may already be running while you are still reading logs. Learn which thresholds start them for your business before an incident, not during one. When counsel asks what happened and when, your retention window and the integrity of those logs will matter more than how clever the detection rule was.

The worst mistake in Kubernetes response is running kubectl delete pod as step one because it feels decisive. One keystroke removed the attacker's process table and your own memory forensics together. Quarantine with a network policy or a deny-all identity, cordon if there is any chance the node is in play, then capture logs and snapshots. Delete belongs to the eradication phase, once the evidence is sitting in durable storage.

Reading Falco and Tetragon timelines next to Hubble flows, you are assembling a story: exec → file read → connect → possible token use. A missing link means either the traffic was encrypted somewhere you cannot see it, or a node was running with no agent on it. Write those gaps into the incident document in as many words. A trail presented as complete when two nodes had no probes will push the whole team toward the wrong containment call.

Two popular ways to get the cleanup wrong. First, rotating the database password and forgetting the refreshable cloud role, the CI deploy key, and the credentials of the sealed-secret controller that the pod could also read. Build a rotation checklist per service tier while everything is calm. Second, redeploying the same mutable :latest tag and calling the workload clean. Pin a digest that came out of a build you trust.

There is a genuine trade-off here. Deep forensic holds cost cloud storage and can keep personal data alive inside captured command lines, which your privacy people will have opinions about. Fast rebuilds restore service and destroy the record. Agree with legal and platform in advance on which severity levels trigger a snapshot hold. Settling that argument at 3 a.m. with an attacker in the cluster goes badly for everyone.

Keep a container response kit in one place: the quarantine label command, the Hubble filters you actually use, two or three Falco queries, the cordon check, and the contact list for secret rotation. Store it where on-call already looks, because searching the wiki while someone pivots through your cluster is not a plan. Update it after every real incident so it stays honest.

Keep the customer story separate from the technical one. Command lines, internal hostnames and node names have no place on a status page. Decide who writes the sanitized version and who reviews it before it ships. Response is half engineering and half communication, and dropping the second half hands you a second incident with your users or your regulator.

Write the collection commands so a tired person can paste them. One line each, nothing left to work out at 3 a.m. beyond the pod name and the namespace, and every one of them tried against a fake suspect workload in staging first.

Try this

Run this on a lab cluster or a single staging node. Read what comes back, and resist the urge to rewrite production policy off one look.

$ kubectl get pod suspect -o yaml | grep -E 'image:|privileged:' | head -4
    image: evil.example/c2:latest
    privileged: true
$ kubectl cordon worker-3 && echo CORDON_OK
node/worker-3 cordoned
CORDON_OK
$ kubectl get node worker-3 | awk 'NR==1 || /worker-3/'
NAME       STATUS                     ROLES    AGE
worker-3   Ready,SchedulingDisabled   worker   40d

Takeaway

If one thing sticks, make it the order. Quarantine and snapshot are what you do first; delete is what you do once the evidence is in a bucket you trust.

Book an hour, hand your on-call a fake Falco alert on a lab cluster, and time them: quarantine label on, snapshot taken, rotation checklist filled in. Then fix the step where they hesitated, in the runbook, that same afternoon.

Related