Kernel events & enrichment

The syscalls that matter, plus container context.

A mall camera that records every footstep fills a drive with video nobody will ever watch. Tag each clip with the store, the badge that opened the door, and whether it happened after closing time, and the same footage becomes something you can investigate. Runtime visibility works the same way. Raw system calls are endless. Events with a name, an owner and a scope attached are signal.

A syscall (system call: a request a running program makes to the kernel, like start a program, open a file, or connect to a network address) means almost nothing on its own. Attach the pod name, the namespace and the container image, and it becomes a fact about a workload you deployed. Falco and Tetragon, the two runtime agents you will meet most often, both assume that enriched model. Before you write or tune a rule, you need to know what the engine can see and what it cannot.

Which kernel events are worth collecting

If volume forces you to sample, sample after the enrichment filters have run, never before you know which namespace an event came from. Sampling raw system calls is a coin flip, and the rare event from the one namespace you cared about is exactly the one you throw away. Scope first. Sample second.

Clocks matter more than people expect. When node clocks drift away from your SIEM (Security Information and Event Management system, the place logs land for search and alerting), every "who moved first" question turns into guesswork. Take timestamps from the node agent and keep NTP (Network Time Protocol, the service that keeps machine clocks in sync) healthy. Five minutes of skew during an incident can flip a timeline end to end and send you digging through the wrong deploy.

bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s -> %s\n", comm, str(args[0])); }'
 
# example output:
curl -> /usr/bin/curl
sh -> /bin/bash
python3 -> /usr/local/bin/gunicorn

From kernel bytes to Kubernetes nouns

The driver, the piece running eBPF (extended Berkeley Packet Filter, small sandboxed programs the Linux kernel runs on your behalf) or a kernel module, sees a raw event full of numbers. The agent does the translation. It reads the cgroup path to work out which container the process belongs to, asks the CRI socket (Container Runtime Interface, the API kubelet uses to talk to containerd or CRI-O) or its own cached metadata for pod name and namespace, then attaches the image digest and user namespace mappings. Your rules never touch raw kernel structs. They reference the finished fields: k8s.ns.name, container.image.repository, proc.ppid. Break enrichment during a runtime upgrade and the rules keep evaluating, only now against hollow context. That shows up two ways on the board: a sudden flood of benign alerts, or complete silence.

kubectl -n falco exec ds/falco -- falco --list=fields | grep k8s
 
# example output:
k8s.ns.name          Kubernetes namespace name
k8s.pod.name         Kubernetes pod name
k8s.pod.uid          Kubernetes pod UID
k8s.deployment.name  Kubernetes deployment name

The same exec, two different verdicts

- rule: Shell in production API tier
  condition: >
    spawned_process and container
    and proc.name in (shell_binaries)
    and k8s.ns.name = "prod"
    and k8s.deployment.name startswith "payments-"
  output: "Unexpected shell (pod=%k8s.pod.name image=%container.image.repository cmd=%proc.cmdline)"
  priority: WARNING

What syscall monitoring will never see

Plenty of attacker behavior never becomes a system call. Code that stays in memory, traffic encrypted end to end with no L7 (layer 7, the application layer where you can read the actual request) inspection, and anything that happens in the seconds before the agent starts can all slip past a syscall-only sensor. Other layers cover different gaps. Falco plugins pull in Kubernetes audit logs and cloud provider trails, Cilium Hubble shows network flows, and Tetragon hangs kprobes (probes attached to specific functions inside the kernel) on sensitive kernel functions. Write down what you actually care about protecting, then ask layer by layer who would notice. A database tier watched only by system calls can miss lateral movement that a network flow view would have shown you in seconds.

falco -o json_output=true 2>/dev/null | head -1 | jq -r ".output_fields | keys[]" | grep -E "container|k8s|proc" | head -8
 
# example output:
container.id
container.image.repository
k8s.ns.name
k8s.pod.name
proc.cmdline
proc.name
proc.ppid

Running the event stream day to day

Ship Falco or Tetragon output somewhere durable before the pod rotates its logs away. Before you turn verbose rules on in production, run them in staging against your busiest namespace and measure events per second, so the number never surprises you at 2am. Then correlate the two halves of the story. The Kubernetes audit log tells you who changed the Deployment. The runtime stream tells you what the resulting pod actually did. Visibility with no retention is forensics with amnesia.

kubectl -n falco logs ds/falco --since=1m | grep -c "Warning"
 
# example output:
14

Build your Falco and Tetragon JSON parsers with explicit field mappings and a regression test behind each one. Upstream adds a field, your dashboard quietly renders nothing, and nobody notices for a month. Keep the raw events immutable somewhere before normalization touches them, so a hunter can re-run a query across last quarter when a new IOC (indicator of compromise, a known-bad hash, path or address) turns up. Set hot retention long enough to cover how long your industry typically takes to spot an intrusion, and push the compliance years into warm storage. Skip that and "we monitor everything" falls apart the first time legal asks for proof of monitoring from six months ago.

One field dictionary everyone shares

kubectl -n falco exec ds/falco -- falco --list=fields | wc -l
 
# example output:
214

Two record sets, two different questions. Kubernetes audit logs tell you what happened to API objects. The runtime stream tells you how processes behaved. Someone holding stolen credentials can patch a Deployment without ever opening a shell, and audit is the only place that shows up, yet runtime still catches the shell once the new pod runs. Teach your SOC (Security Operations Center, the people watching the alert queue) to pivot from an audit user to a runtime pod timeline using the shared timestamp and the pod UID (the unique identifier Kubernetes stamps on every object).

Field notes from real clusters

The kernel is an honest witness with a terrible memory for names. It sees every system call on the box and can tell you that process 4421 opened a file at nanosecond precision. It has no idea that process belongs to your payments API. Enrichment is the whole difference between a line a responder acts on and a line they scroll past.

Start from a short list of events worth keeping: execve and its relatives (a program starting), openat on paths you treat as sensitive, connect and accept (a socket reaching out or answering), privilege changes, and mounts. You are not trying to store every read(2). You are trying to answer one question later. What did this workload do that a normal replica of it would never do? Hold every field you collect up against that question and the dictionary stays honest.

Enrichment is the bridge from kernel bytes to Kubernetes nouns. The probe captures a process ID and a cgroup. The agent turns that into pod name, namespace, deployment and image digest. When the map breaks (a stale CRI socket, missing pod metadata, the wrong container runtime assumed) your alerts read "pid 4421" and your responders shrug. Test that mapping after every runtime or agent upgrade, not once at install time.

Context flips the verdict. The same /bin/sh exec is boring inside a debug sidecar you deployed on purpose, and alarming inside a distroless payments API that ships with no shell at all. A rule that ignores namespace, image and capabilities will either drown you or miss the break-in. Allowlist the debuggers you know about by name instead of muting the rule for everyone.

Blind spots come with the design, so name them out loud. Encrypted node-local sockets, host processes running outside any container, and every minute the agent spends in CrashLoopBackOff are all holes in coverage. Treat agent health as a detection control with its own SLO (service level objective, the number you promise and get paged on). A green application dashboard sitting next to a red Falco DaemonSet is not a green security day.

Rate limits and sampling are survival gear in production, not cheating. One bursty CI (continuous integration, your build pipeline) namespace can melt a ring buffer on every node at once. Reach for scoped rules and namespace exclusions with an expiry date before you reach for a permanent global disable. Write down why the mute exists and who owns it. Future you will read that comment mid-incident and be grateful.

Hand the responders a field dictionary and keep it boring: process.cmdline, k8s.ns, k8s.pod, container.image.repository, evt.type, user.uid, fd.name. Consistent names beat clever ones every time. When every tool in the pipeline emits the same nouns, correlation stops being a craft project and turns into a query.

One team decided to stream every openat from every node into a cold bucket, "for forensics." Within a week the pipeline ran hours behind, ring buffers were dropping events, and the single real crypto-miner exec landed in a gap nobody was watching. Collection with no budget is a blindfold you paid for. Cap what you take, keep high-signal events hot, and make peace with the fact that most file reads will never be stored anywhere. The goal is reconstructing suspicious behavior, not keeping a perfect tape of the kernel.

Failed enrichment does not look like an error. It looks like noise. Events keep arriving, but k8s.ns is empty or the image digest is wrong, so every rule keyed on namespace stops firing and every rule that ignores namespace fires on everything. After a container runtime upgrade, exec a shell in a labeled canary pod on purpose and confirm the alert comes back with the right pod name and image. If all you get is a host process ID, stop the rollout. You have broken the nouns your responders trust.

The common mistake is reading "we run Falco" as "we see attacks." An agent crash-looping on two nodes, a Slack channel someone muted in March, and a SIEM parse error are all visibility outages, and none of them page anybody by default. Put agent Ready count and age of the last event on the same board as API server latency. Security telemetry that can go dark without waking someone will go dark in the exact week you needed it.

The trade-off is easy to explain upward. Richer fields cost CPU on every node and cardinality in the SIEM. Thinner fields cost hours of investigation later. A split that holds up in practice: full command line and file paths for exec, connect and privilege events, plain counts for bulk file noise. Write that split down with the reasoning attached, or a future engineer under pressure will turn everything back on.

One habit separates teams that trust their telemetry from teams that hope. After each cluster upgrade weekend, pick three nodes at random and check three things: agent Ready, a recent event timestamp, and a canary exec that arrives in the SIEM with the correct pod name. Fifteen minutes. It catches the half-finished upgrade where nine nodes are fine and three are dark, which is the failure a dashboard average hides beautifully.

Try this

Run this on a lab cluster or a single staging node. Read what comes back and sit with it for a minute. A first look is not a reason to change production policy.

$ kubectl -n falco get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE
falco-abc12   1/1     Running   0          3d
$ kubectl -n falco logs ds/falco --tail=3 | grep -i loaded
INFO falco: Loaded 312 rules
INFO falco: Opening 'ebpf' source with modern BPF probe

Takeaway

Visibility is a short list of high-signal events plus enrichment that holds up, not a policy of logging everything. Two things decide whether an alert is worth waking someone for: whether the agent is running, and whether k8s.ns.name has anything in it.

Next: take one namespace and write down which exec, open and connect events you expect on an ordinary Tuesday. That list becomes the baseline every tuning decision gets measured against.

Related