Detect vs enforce, safely

The spectrum and a waved rollout.

A speed camera takes your photograph. A concrete bollard stops your car. Both sit by the road, both get called road safety, and only one of them keeps a truck out of the shop window. Runtime security splits the same way. Detection watches and records. Enforcement steps in and stops the action while it is still happening. Real tooling covers the whole range between those two ends, from a Falco alert that writes a log line to Tetragon killing a process inside the kernel and Cilium refusing a network connection outright. Teams who run this well use both on purpose, and can tell you which one is switched on in which namespace.

Detection is the easy half to describe. Enforcement takes more care, because the tools do not all stop the same thing. Cilium refuses the connection at the network layer, so the packet never lands. Tetragon's Sigkill action kills the process at the kernel hook, which stops whatever that process was going to do next but does not always unwind the operation already in flight: a read of the shadow password file can copy bytes back to the caller before the signal arrives. The action that genuinely refuses that read is Override, which hands the caller a permission denied error instead of killing anything. Falco is the detection tool most estates start with, watching syscalls (the requests a program makes to the kernel, things like open this file or start this process). Tetragon and Cilium sit on the enforcement side, both built on eBPF (extended Berkeley Packet Filter, small programs the kernel runs safely on your behalf). The working rule is short. Alert on broad signals everywhere. Block only on narrow conditions you are certain about. Turn blocking on in waves, so production stays up and you always keep a way back.

From watching to blocking

Fail-open or fail-closed is a business call, not an engineering preference. Fail-closed means that when the control breaks, traffic stops. Fail-open means that when the control breaks, traffic keeps flowing. A payments segment may happily take an outage rather than leave a data path wide open. A marketing microsite almost certainly will not. Write the answer down for every namespace wave, and write it before the wave ships rather than during the incident.

Practise the rollback, not only the rollout. Run a game day where the whole exercise is turning enforcement back off: revert the policy, confirm the workload recovers, note how long it took. A control nobody knows how to undo gets ripped out in a panic during the first storm of false positives, and it usually gets ripped out badly. Keep rehearsing on staging until the commands feel dull. Dull is what you want when it is real.

Detection on its own leaves a gap between the moment something bad happens and the moment anyone does something about it. That gap is measured in minutes on a good day. In-kernel enforcement closes it to microseconds, and pays for that speed with a real risk: a rule written slightly wrong kills a healthy process instantly, with no appeal and no retry. So most teams split the difference. Broad detection covers everything. Enforcement covers a short list of conditions with almost no innocent explanation, like a shell starting inside a locked-down tier, a read of the host's shadow password file, or outbound traffic to a CIDR (Classless Inter-Domain Routing block, a range of IP addresses written like 203.0.113.0/24) that nobody approved.

# Phase 1 — Tetragon Post only
matchActions: [{ action: Post }]
 
# Phase 2 — after validation
matchActions: [{ action: Sigkill }]

Roll it out one namespace at a time

Never enforce cluster-wide on day one. Start observe-only in staging, and put the system namespaces and the workloads with known-broken profiles in that first pass, because that is where you learn what your rules really match. Then take one production namespace per wave, in an order you chose deliberately: payments before internal tooling, if protecting the money path is the point of the programme. Give each wave a week of watching your error budget before the next one starts. Build the escape hatch before you need it, and remember that a label on a namespace changes nothing by itself: Tetragon policies select pods, so the hatch is a pair of policies. One only Posts, and selects pods labelled runtime.enforce=observe. The kill policy selects everything else in the wave. Dropping a workload back to log-only is then a label edit on its deployment rather than an emergency rewrite of a live kill rule. Teams that skip waves to have something to show an auditor tend to cause the outage that gets the whole control switched off permanently.

Every block still needs an alert

A blocked action is not a non-event. Somebody tried to read the host password file from inside a payments pod, and the fact that Tetragon stopped them makes the attempt no less interesting. The alert needs three facts: what was blocked, which policy blocked it, and which pod it happened in. Send Falco events and Tetragon Post events to your SIEM (security information and event management, the central place where logs get searched and alerts get raised) even when Sigkill fires, so hunters can see intent, repetition and timing. Silence after a kill is how you miss the attacker's second and third attempt.

# events are JSON on the export-stdout container; the agent container's own
# logs are startup and health messages, not events
# -l reads every agent pod, so a kill on any node shows up; ds/tetragon reads one
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon \
  -c export-stdout --max-log-requests=20 --since=1h \
  | jq -r 'select(.process_kprobe.action == "KPROBE_ACTION_SIGKILL")
           | "\(.process_kprobe.policy_name)  \(.process_kprobe.process.pod.name)  \(.process_kprobe.process.binary)"'
 
# example output:
block-shadow-read  payments-api-7c9  /bin/sh

Admission control and runtime enforcement do different jobs

Admission control is the bouncer at the door. It inspects a pod before it is scheduled and turns away the shapes you have banned, like privileged: true or a hostPath mount of the node filesystem. Runtime enforcement is the guard inside the building. It watches what the pod does once it is running: shells, file reads, outbound connections, escape attempts. Turning a privileged pod away with Pod Security Admission (the built-in Kubernetes gate that rejects pods breaking a security profile) or with Kyverno costs you a rejected deploy and an irritated developer. Killing that same container mid-breakout with Tetragon costs you an incident. You need both layers, and neither one covers for the other. Admission cannot see behaviour that only appears at runtime. Runtime enforcement cannot un-schedule a privileged pod after the fact.

The numbers worth reporting

Four numbers carry the story. Mean time to contain, from the first alert to the threat actually being stopped. The false positive rate on your enforcing rules. The share of namespaces in enforce mode against the share still in observe. And the overlap between Falco Critical alerts and Tetragon Sigkill events, because without deduplication a single attempted action shows up as two separate incidents in the SIEM and inflates every count above it. Keep one coverage figure beside those four: the share of nodes running a healthy agent right now, because a rule with no live agent to run it is not a control at all. Leadership reads the first number and the trend line. The engineers on the rota read the rollback count, which is the honest measure of how well the rules were written.

kubectl get ns -L runtime.enforce
 
# example output:
NAME       STATUS   ENFORCE
payments   Active   wave2
tools      Active   observe
core       Active   wave1

Making mixed mode legible

kubectl get tracingpolicy -A --show-labels | grep enforce-wave
 
# example output:
block-shell-prod   enforce-wave=2
block-shadow-read  enforce-wave=3

Line the wave labels up across Tetragon and Cilium so one query answers one question. Two keys carry the whole scheme. runtime.enforce says where something stands: on a namespace it is the wave that namespace has reached, and on a pod it is the escape hatch holding one workload at observe. enforce-wave on a policy says which wave switches that policy on. When an operator sees the same wave number on a process policy and on a network policy, they know that namespace is under strict process and network enforcement at the same time. That combination is exactly the one that produces a baffling outage when nobody realised both were live.

Notes from clusters that learned this the hard way

The choice was never binary. Ranked from gentlest to bluntest, your options run: Falco alerting and nothing else, a userspace responder that deletes the offending pod a second or two later, Tetragon sending SIGKILL (the signal that ends a process immediately, with no chance to clean up), a BPF-LSM deny that refuses the operation at the kernel's own security hook, and a network policy that drops the packet. You pick per threat and per maturity, and then you revisit the pick. This is not something you decide once and carve into a runbook.

Expect to live in mixed mode for a long time. Some teams stay observe-only because their applications are fragile and a stray kill would take out a checkout flow. That can be a perfectly good answer, provided a named person accepted the risk and put a date on it. Mixed mode that nobody tracks is how a temporary exception quietly becomes a permanent hole, discovered during an incident review.

Write the narrative before the rollout, not after the first page. What do you enforce? What do you only detect? A fail-closed network policy and a fail-open kill hook behave in opposite ways when the same node has a bad night, so say which is which, per component, in language your on-call engineer can read while half asleep. Plenty of stacks fail open on purpose, because availability wins that argument in most businesses. If yours does, your detection has to page loudly the moment an enforcer disappears, because a silent fail-open is a policy hole nobody can see.

The outage everybody remembers is enforce-all-on-Monday. One team shipped a broad Tetragon kill policy, a brand new Falcosidekick responder that deleted pods, and a Cilium default-deny, all inside the same change window. Rolling it back took three Git pull requests while production sat frozen. Wave the controls instead. Count false positives per wave. Keep a documented break-glass that switches off one layer without blinding you completely. Enforcement programmes die when the only undo anyone knows is turn security off.

The common misuse is treating we have runtime security as though it meant we enforce. A board hears the first sentence and files away the second. Publish a plain matrix instead: control, mode (observe or enforce), coverage percentage, owner. Update it whenever a mode changes. That honesty costs you an awkward conversation now and saves you a blame session later, when an alert-only tool gets accused of letting an attack finish.

The trade-off is speed against blast radius. A synchronous deny stops the damage inside the kernel and makes every false positive expensive. A userspace response is slower, and far easier to stage and reverse. For most organisations the sensible sequence is network controls and admission early, process kills late, once the rules have earned some trust. Order matters more than ambition here.

Write break-glass as a real procedure with names in it. Who is allowed to disable a kill policy? For how long? Against which ticket? What monitoring stays on while it is off? Break-glass that lives as a message in a chat channel becomes a permanent exception within a month. Time-box every disable and open the re-enable ticket automatically. Half of enforcement maturity is governance, and only the other half is kernel work.

Match the action to the blast radius. A network deny affects connections. A process kill affects one PID (process ID, a single running program). Cordoning a node affects scheduling for everything on it. A cluster-wide delete is almost never the right opening move. Pick the smallest action that genuinely breaks the attack chain you care about. Bigger is not safer if it teaches the rest of the company to route around you.

Try this

Run this on a lab cluster or a single staging node. Save the policy below as deny-shell.yaml, and give yourself something to shell into first: kubectl run attacker --image=busybox -- sleep 1d. In the session that follows, the second command is the point of the first: a server dry run checks the policy against the API server and then throws the object away, so the policy does not exist until you apply it for real. Read what comes back before you touch a production policy, because the first look at an event stream is usually the least informative one you will get.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: deny-shell
spec:
  podSelector:
    matchLabels:
      run: attacker          # the lab pod only, never the whole cluster
  kprobes:
  - call: security_bprm_check
    args:
    - index: 0
      type: linux_binprm
    selectors:
    - matchArgs:
      - index: 0
        operator: Postfix    # /bin/sh in busybox is a link to /bin/busybox
        values: ["/sh", "/busybox"]
      matchActions:
      - action: Sigkill

$ kubectl apply --dry-run=server -f deny-shell.yaml
tracingpolicy.cilium.io/deny-shell created (server dry run)
$ kubectl get tracingpolicy deny-shell
Error from server (NotFound): tracingpolicies.cilium.io "deny-shell" not found
$ kubectl apply -f deny-shell.yaml
tracingpolicy.cilium.io/deny-shell created
$ kubectl get tracingpolicy deny-shell
NAME         AGE
deny-shell   12s
 
# leave the watch below running, then from a second terminal:
#   kubectl exec -it attacker -- /bin/sh
# one node in the lab, so ds/tetragon is the agent that sees it; on a real
# cluster, exec into the agent running on the attacker pod's node
$ kubectl exec -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact | grep -i sigkill
💥 exit    default/attacker /bin/sh SIGKILL

Takeaway

Enforcement is a dial, not a switch, and the dial carries a setting for every namespace you own. If you cannot say today whether payments is on Post or on Sigkill, and what happens to that namespace the moment the Tetragon daemonset falls over, that gap matters more than the next rule you were about to write.

Next: put one namespace on your scorecard with an honest mode beside it, observe or enforce, then move on to container escapes so you know what your Sigkill rules are racing to beat.

Related