Routing & automated response
Falcosidekick, Talon, isolate on critical.
A smoke detector beeping in an empty house has done its job and changed nothing. Somebody still has to hear it, decide whether it is real, and either open a window or call the fire brigade. Falco is the detector. Left on its own it writes lines into a pod log that nobody reads at 3am. The wiring you build around it, the routing, the paging, the isolation, is what turns a detection into a contained incident instead of an archaeology dig three weeks later.
Response, in this lesson, means three things. Routing alerts by priority so the right ones land in the right place. Runbooks a tired human can follow without thinking hard. And a clear-eyed decision about when a machine is allowed to evict or quarantine a pod by itself. Speed counts, because an attacker with a shell is not waiting for your standup. Being wrong counts more. A false positive (an alert that fires on perfectly normal behaviour) that deletes a production pod hurts far more than a missed Notice, which is one of Falco's low-urgency priority levels.
Route it before you automate it
Put a speed limit on anything that acts on its own. A rule that flaps, firing and clearing and firing again, wired to a delete action will chew through your pods in a loop, and you have built your own outage with your own tooling. Give every destructive response two guardrails: a dedupe window, meaning repeats of the same alert are ignored for some number of minutes, and a hard cap on actions per hour. If that cap trips, treat the trip itself as worth a page.
Keep a second exit. The incidents that stress your cluster tend to be the same ones stressing your chat tool, and "we paged in Slack" is worthless when Slack is the thing that is down. Have a path that reaches a human through your SIEM (security information and event management, the central system where security logs land and get searched) or through a plain ticket, with no chat in the middle. Rehearse it on staging until running those commands feels boring. Boring is the goal. Boring is what people can still do at 4am with a headache.
Falcosidekick is the mailroom for all of this. Falco produces one stream of alert events and stops there. Falcosidekick picks each event up and decides which copies go where, so you configure destinations once in one place instead of teaching every rule how to find your pager.
It forwards to Slack, PagerDuty (a service that phones whoever is on call), Elasticsearch, Loki and plain webhooks. Setting minimumpriority per destination is the trick that keeps chat readable while the SIEM swallows everything: chat gets warning and above, Elasticsearch takes the lot, the automation webhook only sees critical. Configure retries too. Alerts lost during an outage do not bounce back to you. They are invisible until the day you go looking and find nothing there.
falcosidekick:config:slack: { webhookurl: $SLACK, minimumpriority: warning }elasticsearch: { hostport: "https://es:9200" }webhook: { address: "http://falco-talon:8080/events", minimumpriority: critical }
Correlate before you act
One frame from a security camera rarely tells you what happened. You want the minutes either side. Alerts work the same way. In your SIEM, join the Falco field k8s.pod.name against the Kubernetes audit log's user field across the same few minutes, so you can see who touched that pod right before the alert fired. Then build the page template so every critical alert arrives already carrying the pod, the node, the image digest (the exact content hash of the container image that is running) and any deploy that landed nearby. A responder who has to hunt down those four things has lost ten minutes before starting.
Automated response patterns
Falco Talon is the piece that does something rather than telling someone. Its gentlest useful action is a label: it sets quarantine=true on the offending pod, and a deny-all NetworkPolicy that selects on that label cuts the pod off from the network without killing it. The harsh action is eviction on Critical rules. The label is moving a sick patient into a side room. Eviction is sending them home mid-diagnosis. Run alert-only first, for weeks, and switch the acting part on only once the numbers show that rule almost never cries wolf.
kubectl get networkpolicy -n prod quarantine-isolate -o yaml | grep -A2 podSelector# example output:podSelector:matchLabels:quarantine: "true"
Runbooks and rehearsal
Write the response down per rule, the way a kitchen writes down the recipe instead of trusting the chef's memory on a busy night. Who gets paged. What you capture before anything is evicted, because eviction destroys the evidence you will want tomorrow. Whether this one calls for cordoning the whole node or only isolating the pod. Then run the drill every quarter: inject a Critical event on purpose and put a stopwatch on how long containment actually takes.
# On Critical shell in prod:# 1. label quarantine=true# 2. capture logs# 3. evict if policy requires
Feedback loop to tuning
Any time automation evicts a pod that turns out to be innocent, that rule goes back on the bench within 24 hours. Not "when someone has time". Detection, routing, response and tuning only work as a loop, and tuning is the end teams quietly drop first. Drop it and your automation drifts into being the thing everybody works around.
Legal and privacy in automated response
Automation that kills a pod can also destroy data somebody is legally required to keep. Talk to your legal or compliance people before pointing it at any tier handling PII (personally identifiable information, meaning names, addresses, card numbers and the rest). Log everything the response ServiceAccount (the identity Kubernetes hands to software so it can talk to the cluster) does, into storage nobody can edit afterwards, because "the robot did it" is not an answer an auditor accepts. Some jurisdictions go further and require a human to sign off before you disconnect a production payment workload, whatever the alert says.
kubectl auth can-i delete pods --as=system:serviceaccount:falco:falco-talon -n prod# example output:yes
Scope the automation's RBAC (role-based access control, the Kubernetes system for who is allowed to do what) tightly. Labelling and isolating in prod, yes. Deleting cluster-scoped objects or reading secrets, no. A response bot is a webhook receiver sitting inside your cluster with permission to change things, which makes it worth attacking. Small permissions mean that if somebody takes it over, the damage stops at the edge of what you granted.
Field notes from real clusters
The gap between a detection and an outcome is where most programmes actually fail. Falco writes the line. Everything after that, whether a phone rings, whether a ticket opens, whether a network policy tightens, whether the pod gets boxed in, is wiring you built or wiring you never built. Skip it and you have paid for a very sophisticated logging system.
Falcosidekick, or whatever equivalent your stack runs, fans one event out to Slack, Teams, Elasticsearch, PagerDuty, SNS (Amazon's Simple Notification Service, a message fan-out queue) and a long list of others. Start with two destinations: chat and SIEM. Add anything that acts only once false positives have dropped to a level you can live with. An auto-delete hanging off a noisy rule teaches developers to hate the security team, then to route around it, and that costs you more than the rule ever saved.
Correlation is the difference between a word and a sentence. One shell alert on its own might be a broken liveness probe (the health check Kubernetes runs against a container) shelling out because somebody wrote it badly. Three shells in a minute, plus an outbound connection nobody expected, plus a new cron entry (a scheduled job) inside the container, is a story with a shape. Pull in recent deploys, the identity behind the action, and Hubble flows (Hubble is Cilium's view of which pod talked to what) so the responder opens a scene rather than a single syllable.
Actions that hold up in practice: put the quarantine label on the pod so the NetworkPolicy isolates it, snapshot the logs somewhere durable before anything restarts, notify the owning team read straight off the pod's Kubernetes annotations, open a ticket that carries the evidence with it. Actions that will burn you: cluster-wide kills fired by a low-confidence rule, and credential rotation bots that spin without a human saying yes on shared accounts.
Heroics do not scale, and the hero is usually on holiday. For each family of high-severity rules, write four short answers: what this alert means in one sentence, what you check in the first five minutes, the point at which you isolate, and the name of the person you call. Then rehearse it as a game day once a quarter. Reading a runbook for the first time during a real breach is how a twenty-minute incident becomes a two-day one.
Every false positive that woke somebody up earns either a tuning ticket or a written decision to accept the noise. Both are fine. Silence is not. Every true positive earns a different question: could admission control (the check that runs before a pod is ever allowed to start) or MAC (mandatory access control, the kernel-level rules AppArmor or SELinux enforce) have stopped this earlier and cheaper? Response that teaches you nothing is expensive adrenaline.
The alerts themselves carry sensitive cargo. A process command line can hold an API token somebody passed as an argument, or a customer identifier, and Falco will forward the whole string without blinking. Decide what is allowed to leave the cluster, redact the rest, and keep retention on your alert store aligned with the policy that governs your logs. Piping full command lines into a wide open Slack channel has turned a detection into a disclosure more than once.
One team wired automated quarantine to a clumsy rule and it deleted a payment pod every single time a liveness probe ran its kill -0 check, because the patterns overlapped. That outage bought the organisation a year of refusing to discuss response automation at all. The lesson was never "do not automate". It was two things: measure a rule's false-positive rate before anything acts on it, and make network isolation the first generation of action instead of deletion. Containment that keeps the pod alive beats eradication that wipes out your evidence and your uptime in one move.
Read a delivery failure exactly as seriously as a missed detection, because they have the same effect. If Slack answers 401 (the web status code meaning your credentials were rejected) or your SIEM pipeline starts refusing the JSON (JavaScript Object Notation, the text format the alerts ship in), your detection is running blind while looking perfectly healthy from the inside. Alert on the delivery errors themselves. Keep a fallback route, even a second webhook, for anything high severity. The delivery pipe belongs inside the control's availability SLO (service level objective, the reliability target you hold yourself to), not outside it.
The most common way teams ruin this is wiring every severity to PagerDuty on day one. Fatigue shows up inside a week, and after that people swipe the page away before reading it. Map severities to actions instead: low opens a ticket, medium goes to chat, high pages a human, critical pages and isolates. Revisit that map every quarter against real volume numbers rather than what you guessed in the design meeting. A severity model nobody recalibrates turns into fiction.
You are trading speed against certainty, and no setting gives you both. A kill enforced in the kernel lands in milliseconds and will occasionally take out a legitimate tool that looked wrong. A human in the loop is careful and slow, and slow loses to anything that spreads by itself. Decide per rule which side you want and write it down, with a calm head. A vendor demo in front of an excited executive is the worst possible moment to pick.
When you run that game day, plant a deliberate false positive in it. If the automation is too eager you find out in a rehearsal instead of in production. If people freeze, you learn that too, which is more useful than it feels at the time. Write down the wall-clock minutes from alert to isolation. That one number tells you more about your programme than any slide listing the tools you own.
Keep the credentials for the sidekick and for any SOAR (security orchestration, automation and response, the tooling that runs your playbooks) in a secrets manager with rotation switched on. An expired webhook is one of the most common reasons a team believes it has response and does not. Fire a synthetic test alert once a week so the pipe has to prove out loud that it still delivers.
The failures here are dull rather than dramatic. A webhook URL that got rotated and never updated. A NetworkPolicy so tight it blocks falcosidekick from reaching anything. An action with a selector one character wrong that isolates the pod next door. None of that shows up in a design review. It shows up the first time somebody walks the whole path end to end under pressure, which is exactly why you walk it before the real page arrives.
Try this
Run this on a lab cluster or a spare staging node, never on anything serving traffic, since the last command force-deletes a pod. Read what comes back and sit with it. A first look is not a reason to change production policy.
$ kubectl -n falco get pods -l app=falcosidekickNAME READY STATUS RESTARTS AGEfalcosidekick-0 1/1 Running 0 2d$ curl -sS -o /dev/null -w "%{http_code}\n" http://falcosidekick:2801/healthz200$ kubectl delete pod offender --force --grace-period=0pod "offender" deleted from default
Takeaway
Falco's output is worth whatever your routing does with it. Give each alert a destination that matches its priority, enough surrounding context to be read in one glance, and a runbook somebody has actually rehearsed. Let a machine act on its own only where you measured the rule first and found it dull.
Next: take one Critical rule you already run and prove end to end that it reaches a human channel carrying the pod, the namespace and the owning team. One rule, all the way through. After that, look at Tetragon, which can stop an action inside the kernel rather than reporting it after the fact.