Falco runtime detection

Driver, engine, rules; shells in containers.

A smoke alarm wired to nothing is a plastic disc on the ceiling. It beeps, the house burns, nobody comes. Falco is the smoke alarm for the Linux kernel underneath your cluster. It watches syscalls (system calls, the requests every program makes when it wants the kernel to open a file, start a process or make a network connection), stamps each one with Kubernetes context, and shouts when a program does something you never said was normal. An image scanner tells you what could go wrong someday. Falco tells you someone is walking around inside the house right now.

The vocabulary first. Falco ships as a DaemonSet, a Kubernetes object that guarantees one copy of a pod on every node in the cluster. Each copy has three parts: an eBPF driver (extended Berkeley Packet Filter, a way to run small, safety-checked programs inside the Linux kernel) that taps syscalls, a rule engine that decides what matters, and a pile of YAML rules (plain-text configuration files) that you own and maintain. An alert is not a block. Falco tells you a thing happened. Turning that into a response is separate wiring you have to build yourself.

The driver, the engine, and your rules

Priorities and tags are how you keep the pager honest. A chatty NOTICE rule should not ride the same PagerDuty route as a CRITICAL "someone opened a shell in production" rule. Filter your outputs so that when a phone buzzes at 3am, the person holding it believes it.

When Falco falls behind under load it drops events, and the drop itself needs to alert someone. A dropped event looks exactly like a quiet night. Put drop counts on the same board as your API error rates, next to the numbers people already stare at. Practise the commands on staging until they bore you. Boring is the goal, because boring is reliable.

Here is the flow inside one pod. The eBPF driver captures syscalls and enriches them with pod, namespace and image data. The engine tests each event against your YAML rules, which are conditions over fields such as proc.name, fd.name and k8s.ns.name. Output templates attach a priority and one line of text a human can read at speed. New installs use the CO-RE eBPF probe (Compile Once, Run Everywhere, a build that works across kernel versions without recompiling on each node). The legacy kernel module still exists and is discouraged. Run one Falco per node. A single central copy sees its own kernel and nothing else, so every other node stays dark.

The rule language: lists, macros, rules

Lists are reusable sets of values, written once and referenced everywhere, like shell_binaries or sensitive_files. Macros are named conditions you compose, so container becomes shorthand for container.id != host. Rules fire when their condition matches and emit output at a priority somewhere between NOTICE and CRITICAL. Keep your own rules in a ConfigMap (a Kubernetes object that holds configuration files) or a separate sidecar file loaded after the defaults, so the next upgrade cannot flatten your tuning.

- list: shell_binaries
  items: [bash, sh, zsh, dash, ash]
- macro: container
  condition: container.id != host
- rule: Terminal shell in container
  condition: spawned_process and container and proc.name in (shell_binaries)
  output: "Shell (ns=%k8s.ns.name pod=%k8s.pod.name cmd=%proc.cmdline)"
  priority: WARNING

What Falco is genuinely good at

Shells and interpreters starting where none belong. Writes into /etc and credential paths. Reads of secret files. Privilege escalation patterns. Outbound connections to destinations you rarely talk to. Kubernetes audit events, once the audit plugin is enabled. Plugins widen the input beyond syscalls. None of this blocks anything on its own, though. Falco stays a detector until you pair it with automation that acts.

kubectl -n falco logs ds/falco --tail=2
 
# example output:
04:41:12 Warning Shell (ns=prod pod=payments-api-7c9 cmd=bash -i)
04:41:09 Notice Unexpected outbound (pod=payments-api-7c9 connection=203.0.113.44:443)

Deploy it, then prove it works on every node

Installing Falco is not the same as knowing it fires. After the install, go to staging and commit the offence on purpose. Run bash inside a locked-down deployment that has no business running bash, then confirm the alert lands with the right Kubernetes fields filled in. Watch the DaemonSet rollout while the cluster scales, too. A pending Falco pod is a node with no detection at all.

falco --list=rules | wc -l
 
# example output:
318

Wire the response path on day one

Turn on JSON output (JavaScript Object Notation, the machine-readable format your log tooling expects) or gRPC (a fast machine-to-machine messaging protocol), and stand up Falcosidekick the same day you install Falco. Alerts routed to nowhere train nobody. Set priorities honestly: NOTICE for threat hunting, WARNING for the ticket queue, CRITICAL for waking a person up.

falco:
  json_output: true
falcosidekick:
  enabled: true

Falco is a platform dependency, so track it like one. Follow the CNCF (Cloud Native Computing Foundation) release cadence, the driver security advisories and the community rule updates. Never merge upstream default rule changes blind. Diff them first, or you will reintroduce noise you spent a month tuning out, or quietly wipe your local append blocks. The workflow that survives contact with reality: import upstream defaults, overlay your changes in local.d, never edit vendor files in place. Run falco validate in CI (continuous integration, the automated checks on every pull request) whenever a rule file changes.

Health checks for the detection plane

kubectl -n falco get ds falco -o jsonpath="{.status.numberReady}/{.status.desiredNumberScheduled}"
 
# example output:
12/12

Pin the Helm chart (the packaged install template for a Kubernetes app) and the driver image together as one version. Upgrading Falco while the driver tag drifts is a classic post-incident finding: exec events stopped appearing after a routine chart bump, and nobody noticed for weeks.

Field notes from real clusters

The enrichment is what makes Falco usable by a tired human. A bare kernel event says a process named bash started. Falco's version says a shell started in the payments namespace, in the checkout pod, on image X. One of those you can act on. Before any of it helps, the detector has to be alive and healthy on every node.

Modern Falco prefers the eBPF probe. The engine loads your rules, the driver attaches its hooks in the kernel, and outputs ship JSON to stdout, to gRPC, or to sidekick. If the driver fails to load, your rules sit there unused, like a fire code binder locked in a cabinet. Check the logs for a successful probe open after every node image change.

Rules are YAML built from lists, macros and conditions on fields like proc.name, fd.name and k8s.ns.name. Start with the stable upstream ruleset. Then add one thin custom file covering your crown jewels. Editing dozens of default rules in place turns every upgrade into a merge fight, so overlay instead.

Falco shines on interactive shells in server pods, package managers running where nothing should ever be installed, sensitive file reads, and some crypto-miner patterns. It is weaker as a pure network IDS (intrusion detection system, a tool that watches traffic between machines for attack patterns). Pair it with Cilium and Hubble for east-west traffic. Know the lane, so you never expect one agent to be the whole SOC (security operations centre, the team and tooling that triages security alerts).

Deploy it as a DaemonSet with careful privileges and honest resource requests. Pin chart versions. Scrape metrics: event rate, drop counts, rule matches. A DaemonSet that reports Running while quietly dropping events is worse than no detector, because it buys you confidence you did not earn. Alert on a non-Ready Falco agent exactly the way you alert on kube-proxy.

Connect the output early, even if version one only posts to Slack. Teams wait for the perfect SOAR platform (security orchestration, automation and response, the tooling that runs playbooks for you) and spend months producing detections nobody ever sees. Falcosidekick fans out to chat, to your SIEM (security information and event management, the searchable log warehouse your security team lives in), and later to automation. Keep a raw log path for forensics, because chat clients truncate long JSON.

Detection health belongs on the same dashboard as cluster health, not a separate page nobody opens. Show four things: which probe type loaded, how many rules loaded, the timestamp of the last event, and any node missing its Falco pod. When a node pool scales up, confirm Falco scheduled there before you call the pool ready for sensitive workloads.

Falco Running but silent? Do not open the rule files. Plant a known-bad canary instead: a shell in a server pod that should trip an alert, and watch for the JSON event. No event anywhere points at the driver, the enrichment, or the output path. An event that lands somewhere nobody reads is a routing problem. An event that fires on half your nodes is DaemonSet skew. That triage order saves you from tuning a detector that never saw the syscall in the first place.

The common failure is not technical. A team installs the default ruleset, ticks the compliance box, and never opens a single alert. Six months later the channel is muted, the sidekick token has expired, and the board slide still reads "runtime detection: yes". Book a monthly fire drill. Plant the canary, confirm the page arrives, confirm the SIEM field mapping still matches. A detector nobody exercises is decoration.

Priority alone does not carry meaning. A NOTICE shell in a distroless namespace, where there is no shell binary to run at all, is a very different night from a NOTICE in a build namespace. Train responders on your top ten rules using screenshots of real JSON. Which field names the owning team. Which field justifies quarantining the pod. Which field is false often enough to discount. Skip that and every alert becomes a shrug or a panic, rarely a calibrated action.

Set against Tetragon and plain auditd (the Linux audit daemon that logs kernel events to disk), Falco is the approachable option, with a big community and an alert-centric design. It is not your in-kernel enforcer. If leadership asked for "block the shell", you need a different mode or a different tool, and you should say so out loud before the budget moves. Keep the pitch honest so you are never blamed for a product you never deployed.

Pin chart versions and record the hash of the ruleset you deployed. A surprise rule upgrade in the middle of an incident turns your timeline into fiction. When the project ships new defaults, stage them like an application release: diff, canary namespace, then fleet. Runtime detection is production software, and it earns the same change discipline you demand from the payments team.

Treat a silent Falco DaemonSet after a kernel upgrade as a detection outage, not a green dashboard. Your pipeline should assert READY replicas and a non-zero rules-loaded log line with the same seriousness it asserts application health.

Try this

Run this on a lab cluster or a single staging node. Read what comes back slowly. Do not rewrite production policy off a first look.

$ kubectl -n falco get ds falco
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
falco  3         3         3       3            3
$ kubectl -n falco logs ds/falco --tail=2
INFO falco: Loaded 312 rules
04:41:12 Warning Read sensitive file (file=/etc/shadow)

Takeaway

Falco is eBPF-backed syscall detection with Kubernetes context attached to every event. A healthy probe on every node and an output path that reaches a human matter at least as much as clever rules.

Next: generate one deliberate lab alert and follow it all the way to the place a responder would actually see it, then move on to tuning so you never mute the one signal that matters.

Related