Falco on bare hosts

The k8s runtime tool, minus the k8s.

A building gets watched two ways. A night watchman walks the halls once an hour and notes anything that looks off. Or a guard sits at a wall of live camera feeds and hits the alarm the second a door gets kicked in. Most tools in this course take rounds: a scheduled scan, a nightly diff, a file check on a timer. Falco is the guard at the cameras.

Falco is a runtime security tool from the CNCF (Cloud Native Computing Foundation, the same group that stewards Kubernetes). You may have met it in the container courses, bolted onto a cluster. Take the cluster away and the same engine runs fine on a plain Linux server. It listens to the constant stream of system calls (the requests every program makes to the kernel, the core part of the operating system that talks to the hardware, whenever it wants to open a file, start a process, or make a network connection), checks each one against a set of rules, and alerts the instant one matches.

On a host with no Kubernetes anywhere, that means catching the exact behaviors from the earlier lessons the moment they happen instead of on the next scan: a shell spawned by a web server that should only ever answer requests, a write landing under a cron directory, a process reading its way through everyone's SSH (Secure Shell) keys, an outbound connection from a daemon that is supposed to only listen.

How Falco hears the kernel

Every one of those system calls passes through one place, the kernel, the way every call in an old office passed through one switchboard. Falco needs a tap on that switchboard, and it has three ways to get one.

The oldest is a kernel module (kmod), a piece of code loaded into the kernel itself. It works almost everywhere, but it is code running in the most privileged part of the system, and it has to be built to match your exact kernel. The second is a classic eBPF (extended Berkeley Packet Filter, a way to run small, sandboxed programs inside the kernel without the danger of a full module) probe. The third, and the one to reach for on anything current, is the modern eBPF probe. It is compiled once into the Falco binary using CO-RE (Compile Once, Run Everywhere), so it needs no kernel headers and no build step on the host. It wants a Linux kernel of version 5.8 or newer.

Getting the sensor onto the host

Install the package and the Debian and Red Hat builds drop in three systemd services, one per driver (systemd is the program that starts and supervises services on modern Linux). During install a text menu asks which driver you want. Pick modern eBPF and it enables the matching service. Then check what you got.

Two lines tell you it worked, one near the top and one at the bottom. Up top: Active: active (running). At the bottom: Falco opening the 'syscall' source, with no error line after it. If the kernel is older than 5.8 the modern probe cannot load, the service dies during start, and that same status shows failed in place of running. Read it before you walk away and assume you are covered.

What it watches, and where you say so

Falco's behavior lives in two kinds of file: its main config, and its rules. The config, falco.yaml, tells Falco which driver to run and, more usefully to you, which rule files to read and in what order.

# excerpt
 
rules_files:
  - /etc/falco/falco_rules.yaml         # shipped, general ruleset (loads first)
  - /etc/falco/falco_rules.local.yaml   # your overrides (loads last, wins)
  - /etc/falco/rules.d                  # a directory of extra rule files
 
engine:
  kind: modern_ebpf
  modern_ebpf:
    cpus_for_each_buffer: 2
    buf_size_preset: 4
 
stdout_output:
  enabled: true
syslog_output:
  enabled: true
json_output: false
priority: notice        # ignore anything quieter than NOTICE

Order matters. The shipped falco_rules.yaml loads first and carries hundreds of general rules. Your falco_rules.local.yaml loads after it, so anything you define there wins ties. Files dropped into rules.d load as well. There is one rule about the rules: never touch falco_rules.yaml itself.

Rules for the behaviors you already studied

A Falco rule has three parts you care about: a condition (the thing that must be true), an output (the alert line it prints), and a priority (how loud it is, on a scale of eight steps from DEBUG at the quiet end up to EMERGENCY at the top, with CRITICAL sitting near the loud end). Conditions are built from fields like proc.pname (the parent process's name) and fd.name (the file or socket being touched). To keep those conditions readable, Falco lets you name a common phrase once and reuse it everywhere. That named phrase is a macro. spawned_process, shell_procs, and open_write are macros the shipped rules already define, so you can build straight on top of them.

Here are two rules keyed to techniques from earlier lessons, plus a macro of your own naming the accounts allowed to write to sensitive spots.

# Local overrides. The shipped falco_rules.yaml is replaced on every
# update, so nothing you want to keep lives there.
 
- macro: trusted_writers
  condition: proc.name in (apt, apt-get, dpkg, unattended-upgr, cron, run-parts, systemd)
 
- rule: Persistence location written
  desc: A write landed where attackers hide to survive a reboot or a re-login
  condition: >
    open_write and
    (fd.name startswith /etc/cron or
     fd.name endswith authorized_keys or
     fd.name = /etc/ld.so.preload or
     fd.directory = /etc/sudoers.d)
    and not trusted_writers
  output: >
    Persistence location written
    (file=%fd.name proc=%proc.cmdline parent=%proc.pname user=%user.name)
  priority: CRITICAL
  tags: [host, persistence]
 
- rule: Shell spawned by a service account
  desc: A long-running network service launched a shell, which it has no reason to do
  condition: >
    spawned_process and shell_procs and
    proc.pname in (nginx, httpd, apache2, postgres, mysqld, redis-server)
  output: >
    Service spawned a shell
    (parent=%proc.pname shell=%proc.name cmd=%proc.cmdline user=%user.name)
  priority: CRITICAL
  tags: [host, execution]

Read the second rule out loud: a process was spawned, it is a shell, and its parent is one of these long-running services. A healthy nginx never runs bash. When it does, someone is standing on its neck. That shell is the doorway to the rest of the post-break-in work from the earlier lessons: forcing a program to load a hostile library first with LD_PRELOAD (an environment variable that changes which code loads before a program's own), hunting for a SUID (set-user-ID) binary to climb to root, reading its way through everyone's credentials. Falco does not name the technique. It catches the moment the service hands someone a prompt, and that is the moment all of it becomes possible.

Prove it fires

Two checks before you trust a rule: confirm it parses, then confirm Falco actually loaded it. Validate the file first. One catch: your local rules lean on macros like open_write and spawned_process that live in the shipped falco_rules.yaml, so hand the validator both files, the shipped one first, or it rejects your rules for leaning on names it has never been told about.

Now set the trap off yourself. Follow Falco's alerts in one shell, and in another drop a fake persistence job into cron the way an attacker would after landing a foothold.

That alert reached journald (the systemd logging service) within the same second the file was written, carrying the file, the full command, the parent process, and the user. The write and the page happened together. A nightly scan would have found that cron file the next morning, twelve hours after the attacker was already back in through it.

Keep the tripwire armed

A detector that cries wolf gets silenced, and a silenced detector is worse than none, because now you think you are covered. Falco's default rules on a busy host are loud: package managers write system files, cron runs on its timer, admins open shells all day. The move is to carve out the known-good paths and leave the real detections paging at full volume.

Three habits do most of the work. Key rules on behaviors an attacker cannot cheaply avoid, like spawning a shell or writing a persistence file, rather than on specific tool names they can rename in a second. Add narrow exceptions for your own noisy-but-benign processes, the way the trusted_writers macro above excuses apt and cron. And split rules by loudness: send the chatty, low-signal ones to a dashboard you review on your own schedule, and page a human only on the short, rehearsed list of things that mean a genuinely bad day.

Where Falco sits next to the rest

Falco is your live layer, and it works best flanked. osquery answers point-in-time, fleet-wide questions (what is true across all 400 boxes right now). The audit pipeline keeps the durable, off-host record you read after the fact. Falco fires the instant a watched behavior occurs and hands its alert to that same pipeline, shipped off the host, correlated, and triaged. State, history, and live events, covered together. Its rules are where 'we know how attackers behave' turns into 'we get paged when they do.'

One check you will run constantly lives above: falco -L lists every rule Falco has actually loaded. After any edit, run it and look for your rule by name. If it is missing, Falco is not watching for that behavior, whatever the file on disk says.

Try this

Work through “Where Falco sits next to the rest” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: your edits to falco_rules.yaml quietly vanish. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related