Falco on a host

Rules, drivers and tuning alerts.

Falco is a runtime security sensor from the CNCF (Cloud Native Computing Foundation). It watches system calls on a host as they happen, evaluates them against rules, and raises an alert the moment one matches: a sensitive file read, a write into a persistence location, an unexpected shell. It is often met inside Kubernetes, but it runs just as well on a plain server. In this lesson you install Falco 0.45.0 from the vendor's repository on Ubuntu 26.04, check which driver it uses and what it opened, read the rules it ships, trigger a default rule with an ordinary admin action, write a local rule, tune noise with an exception, and remove Falco cleanly afterwards.

How Falco sees the host

Falco needs a tap on the kernel's system-call stream, and current releases offer two. The default is the modern eBPF probe: eBPF programs compiled into Falco itself and adapted to the running kernel through BTF (linux-det/ebpf explains both), so nothing is built on the host and no kernel headers are needed. It requires a kernel with BTF and the BPF ring buffer, which in practice means 5.8 or newer; both lab kernels qualify. The alternative is a kernel module, for older kernels, which must match the exact kernel and is loaded into the most privileged part of the system. The older "legacy" eBPF probe was removed in Falco 0.44.0.

Install from the vendor's repository

Falco's packages come from download.falco.org, signed with the project's key. Store the key in its own keyring, check the fingerprint against the one on falco.org's installation page, and name the keyring with signed-by so it vouches for this repository only (the pattern from "Packages and updates" in Linux essentials).

The fingerprint 478B2FBBC75F4237B731DA4365106822B35B1B1F matches the one Falco publishes, and the key expires in December 2028, so plan to refresh it. The candidate is 0.45.0, released on 2026-09-21. The package normally asks in a text menu which driver to use; three documented variables answer instead: FALCO_FRONTEND=noninteractive skips the menu, FALCO_DRIVER_CHOICE=modern_ebpf picks the driver, and FALCOCTL_ENABLED=no turns off automatic rule updates, which the lab does so the rules you read below cannot change mid-lesson.

The package is the only new one; the dkms, headers and dialog packages that Falco's install page also lists are needed for the kernel module and the menu, not for the modern probe. Its post-install script configured the modern_ebpf driver, masked falcoctl-artifact-follow.service (the rule updater, linked to /dev/null), enabled falco-modern-bpf.service with falco.service as an alias, and started it. The kmod and custom units stay disabled; exactly one Falco service should run.

Next, check the version, the driver and what the service opened.

Falco 0.45.0 with libs 0.26.0 and driver API 11.0.0, and the configuration files it read: the installer's config.d/engine-kind-falcoctl.yaml (the driver choice), a container plugin file, and falco.yaml. A version line proves the package is installed, not that the sensor is working. The service's own log for its current run does.

"Opening 'syscall' source with modern BPF probe" is the line that confirms capture started with the intended driver. Matching _SYSTEMD_INVOCATION_ID to the unit's current InvocationID limits the journal to this run of the service, so messages from an earlier start cannot mislead you. The two TOCTOU (time-of-check, time-of-use) notices are specific to aarch64: that architecture has no open or creat system calls, only openat, so the extra programs Falco attaches to cross-check those calls have nothing to attach to, and detection continues. Last, installing a sensor added a network listener: Falco's health web server on 0.0.0.0:8765, reachable from any network the host is on. If nothing remote needs it, set webserver.listen_address to 127.0.0.1 or disable the web server in a file under /etc/falco/config.d.

The rules and the configuration that loads them

falco.yaml loads falco_rules.yaml (the vendor's rules), then falco_rules.local.yaml, then every file in rules.d, in that order. priority: debug means rules of every priority are active. watch_config_files: true makes Falco reload itself when a configuration or rules file changes. With stdout_output and syslog_output enabled, alerts land in the journal of falco-modern-bpf.service. The package ships 25 rules, the project's stable set; more experimental rule sets are published separately. Here is the condition of one of them.

A condition is a boolean expression over event fields. open_read, sensitive_files and proc_name_exists are macros, named conditions defined elsewhere in the file (sensitive_files includes /etc/shadow), and names like user_mgmt_binaries are lists of programs trusted to read such files. Every and not ... line is an exception someone needed. Never edit falco_rules.yaml to add your own: the package upgrade, and the rule updater if you enable it, replace that file. Your rules and exceptions go in rules.d or falco_rules.local.yaml. By default Falco stops at the first rule that matches an event (rule_matching: first), so load order also decides which alert you see.

An alert from ordinary admin work, and the noise around it

Before triggering anything, look at what the default rules already report on this host.

Every command in this lab reaches the host through runuser -l deploy -c ..., a tool that runs a command as another account. Its PAM stack reads /etc/shadow, and runuser is not on the rule's trusted lists (the runuser_reading_pam macro above only excuses its reads of /etc/pam.d), so every lab step raised this alert. user=<NA> and user_uid=4294967295 (the unsigned form of -1) mean Falco had no user information for that process; the login identity it did have. Working over an ordinary SSH session you will probably not see this alert at all; automation that uses runuser produces it on real servers, and the Try this below gives you noise of your own to tune. The fix is an exception for that exact behaviour, not a quieter rule. The same local file can add a rule of your own, here for files appearing in the cron directories, a persistence location.

# Local additions. Files in rules.d load after falco_rules.yaml and survive package upgrades.
 
# runuser (used by automation to run commands as another account) reads /etc/shadow through PAM.
- rule: Read sensitive file untrusted
  exceptions:
    - name: runuser_pam_shadow
      fields: [proc.exepath, fd.name]
      comps: [=, =]
      values:
        - [/usr/sbin/runuser, /etc/shadow]
  override:
    exceptions: append
 
# A file written, moved or linked into a cron location is a persistence location changing.
- list: secopslog_cron_paths
  items: [/etc/crontab, /etc/cron.d, /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly, /var/spool/cron]
 
- rule: Cron file written
  desc: A process wrote, renamed or linked a file into a cron location
  condition: >
    (open_write and fd.name pmatch (secopslog_cron_paths))
    or (evt.type in (rename, renameat, renameat2, link, linkat)
        and fs.path.target pmatch (secopslog_cron_paths))
  output: Cron file written (evt=%evt.type file=%fd.name target=%fs.path.target process=%proc.name parent=%proc.pname user=%user.name loginuid=%user.loginuid command=%proc.cmdline)
  priority: WARNING
  tags: [host, persistence]

The first entry names an existing rule and uses override: exceptions: append to add one exception to it; the older append: true form is deprecated. The exception matches only when both fields match: the program's full path (proc.exepath, which an attacker cannot fake without writing to /usr/sbin, unlike a process name) and the file. The rest is a new rule. The list names every cron location, and pmatch matches a path equal to or below any of them. The first clause, built on the shipped open_write macro, catches a file opened for writing there. That alone is easy to walk around: write the file anywhere else on the same filesystem and mv it in, and the kernel performs a rename, which opens nothing, so an open-based rule never fires (a hard link with ln is the same). The second clause covers those system calls, using fs.path.target, the resolved destination path. Validate against the shipped rules, since the local file uses their macros, then restart and check what loaded.

Both files validate and the running Falco now reads the local file. (Falco had in fact already reloaded when the file appeared, because of watch_config_files; the restart makes the moment explicit.) Now do ordinary admin things: read /etc/shadow as root, write a placeholder file into /etc/cron.d, and move a second one in from /var/tmp.

Three alerts from this run of the service, and the runuser noise is gone. The first is the default rule doing its job: process=cat with parent=sudo, user=root, and user_loginuid=1001, the login identity (auid, linux-det/auditpipe) of deploy behind the sudo. proc_exepath is uutils' cat, and the gparent and ggparent shells belong to the lab's harness. The other two are the local rule. evt=openat is tee, run through sudo by login 1001, writing /etc/cron.d/fa-demo. evt=renameat2 is the mv: no file was opened, so file=<NA>, and the destination is in target=. Without the second clause that move would have passed in silence. Each alert carries the fields you need to decide whether it was expected. A root read of /etc/shadow by an administrator is often legitimate; the rule cannot know that, and your triage can.

When Falco loses events or goes quiet

A sensor that is running is not necessarily a sensor that sees everything. The probe hands events to Falco through ring buffers in memory; when Falco's user-space side cannot keep up (a burst of system calls, a starved CPU), the buffer fills and new events are dropped, and no rule can match an event it never received. Falco's CPU cost grows with the host's system-call rate, so the busiest hosts are the likeliest to drop. Two settings govern this.

Falco checks its drop counters once a second. When dropped events exceed threshold (a fraction: .1 is 10% of that second's events), it takes the listed actions: log writes a debug-level log line, alert raises an alert named "Falco internal: syscall event drop" through the normal outputs, and rate with max_burst allow one such message every 30 seconds. So by default a host losing 5% of its events every second never says so; set threshold: 0 if you want to hear about any loss. The modern_ebpf block sizes the ring buffers: one buffer shared by every two CPUs, at buf_size_preset: 4; a larger preset trades memory for fewer drops. Whether Falco is still running at all is a separate question.

systemd restarts the service if it exits with an error (Restart=on-failure), and NRestarts counts how often that has happened, which is worth collecting: a probe that fails to load after a kernel update, for example, makes Falco exit. /healthz on the web server answers ok while Falco runs. To see what a drop alert looks like without overloading a host, Falco can pretend: simulate_drops adds one dropped event every second. Put it in a file under config.d with threshold: 0, since one simulated drop is far below 10%, and with the action list spelled out: in the lab, a file that set only those two keys produced no alert.

The alert arrives at Debug priority, through the same outputs as every other alert, with the counters that explain it: n_drops="1" out of n_evts="1449" in that second, and the n_drops_buffer_* fields that say which kind of event was lost (all zero here, because the drop was simulated). A real one on a busy host is the cue to raise buf_size_preset, give Falco more CPU, or narrow the rules. Remove the file and restart once you have seen it, or the alert repeats every 30 seconds.

Stop and remove Falco

A sensor you are only evaluating should not keep running, or keep its listener open. Stop it, purge the package, and remove what the package does not own.

The purge stopped the services, removed the unit links, and warned that /etc/falco/rules.d and /etc/falco/config.d were not empty: dpkg never deletes files it did not install, such as your local rule and the installer's driver choice. Removing /etc/falco, the repository line and the keyring finishes the job, and no Falco unit is left (systemctl exits 1 because nothing matched).

Try this

Install Falco on a lab host as above, add the local rules file, and wait for "Opening 'syscall' source" in its journal. Write a two-line script /usr/local/sbin/shadow-hash (#!/bin/sh and sha256sum /etc/shadow >/dev/null), make it executable, run it with sudo, and find the alert with sudo journalctl -u falco-modern-bpf -o cat --since -1min | grep process=sha256sum: parent=shadow-hash, and on Ubuntu 26.04 proc_exepath=/usr/lib/cargo/bin/coreutils/sha256sum, because uutils provides the command. Add a second exception to rules.d with fields [proc.exepath, proc.pname, fd.name] and the values from that alert (/usr/lib/cargo/bin/coreutils/sha256sum, shadow-hash, /etc/shadow), validate it with falco -V, restart, and confirm the script no longer alerts while sudo cat /etc/shadow still does. Copy proc_exepath from the alert rather than guessing it: /usr/bin/sha256sum would not match. Stop and purge Falco when you are done.

Takeaway

Confirm from the service's own log that Falco opened the syscall source, keep local rules and the narrowest exceptions in rules.d, and test each rule against the obvious way around it, such as mv instead of a write. Then watch the sensor itself: drop alerts, restarts, and a host whose alerts or metrics stop arriving.

Related