LSM & mandatory access control
Where SELinux/AppArmor and BPF-LSM hook in.
Your office badge gets you through the lobby. It does not get you into the server room, because a second guard stands at that door and answers to a different list. Linux works the same way. LSM (Linux Security Modules, the mandatory access layer built into the kernel) is that second guard. SELinux, AppArmor and BPF-LSM all get their say after the ordinary file permissions have already said yes, and any of them can tell root inside a container no.
Where LSM sits in the kernel
Two terms first, in plain words. DAC (discretionary access control) is the owner, group and other permission bits you already know: whoever owns a file decides who may read or write it, and root walks past all of it. LSM is the layer that runs afterwards and can still refuse. When a runtime tool advertises that it "denies in kernel", it almost always means it answered at an LSM hook. That hook is the whole difference between watching something happen and stopping it.
cat /sys/kernel/security/lsm# example output:lockdown,capability,yama,apparmor,bpf
Which modules appear, and in what order, depends on the distribution. capability and yama are the baseline pair nearly everyone has. apparmor or selinux is the one doing MAC (mandatory access control, rules that hold even when the process is root). bpf means the kernel will accept BPF-LSM programs. If your nodes ship SELinux in enforcing mode, the svirt container model stacks another set of labels on top of the permission bits.
SELinux and AppArmor, two ways of saying no
AppArmor, the default on Debian and Ubuntu, thinks in paths. A profile is a list: these files, these capabilities, these network families, nothing else. Kubernetes can attach one per container, and runtime/default is a sane curated starting point. SELinux, the equivalent on Red Hat and Fedora, thinks in labels instead. Every process and every file carries a security context, a kind of colour-coded badge, and the policy states which badge may touch which. Its container model, svirt, hands each container a unique MCS (multi-category security) category. Two containers both running as root still cannot read each other's files, even where the permission bits would have allowed it.
# AppArmor:aa-status 2>/dev/null | head -3# apparmor module is loaded.# 142 profiles are loaded.# SELinux (RHEL):getenforce# Enforcing
BPF-LSM: policy you load like a program
BPF-LSM lets you attach eBPF (extended Berkeley Packet Filter, the kernel's sandbox for small verified programs) code straight onto LSM hooks. Same hooks SELinux uses. Different way of writing the policy. Instead of compiling and shipping a distribution policy package, you load a program that can carry Kubernetes context with it, and you can swap it out while the node keeps serving traffic. Tetragon can return a SIGKILL from a hook. A custom BPF-LSM program can refuse a file open before a single byte is read. That is the bridge from Falco-style watching to real blocking, with no kernel patch involved.
kubectl -n kube-system logs ds/cilium-agent --tail=3 | grep -i lsm# example output:level=info msg="BPF LSM hooks enabled" subsys=datapathlevel=info msg="TracingPolicy controller started" subsys=tetragon
Read the denial, do not switch off the layer
A legitimate workload gets blocked, and the fastest-looking fix is to flip the whole layer to permissive. You have now traded a working control for five minutes of convenience. The real fix is much narrower: allow the one path, the one capability, or the one label transition the app actually needs. The logs tell you which one. aa-logprof, ausearch, and the AVC (access vector cache) lines in dmesg all name the exact rule that fired. A switched-off mandatory layer is precisely the gap an attacker hopes to find after landing in a container.
dmesg | tail -2 | grep -i apparmor# example output:[ 892341.221] audit: type=1400 audit(1721801234.221:441): apparmor="DENIED" operation="mknod" profile="k8s-restricted" name="/dev/tun" pid=44821 comm="vpn-init"
How LSM stacks with seccomp and capabilities
Three locks on the same door, each doing a different job. seccomp (secure computing mode, the syscall filter) decides which system calls a process may make at all. Capabilities decide which privileged powers it keeps. LSM decides whether a permitted syscall may touch a particular file, socket or device. None of the three notices behaviour that nobody wrote a rule for, which is why Falco and Tetragon still earn their keep. A well-run cluster does the lot: admission control keeps privileged pods out, seccomp RuntimeDefault is on cluster-wide, the mandatory layer stays enforcing, and runtime detection picks up what the static rules never planned for.
securityContext:appArmorProfile:type: RuntimeDefault # or Localhost + localhostProfile for custom
Managed Kubernetes does not rescue you here. Your worker nodes boot somebody else's node image, and whatever LSM posture that image ships with is the posture you have inherited. Check whether your provider lets you load custom SELinux policy or only runs its own baseline. If it is the baseline only, write that into the risk register rather than hoping. Coming from a Windows background, the shortest translation is this: an AppArmor profile reads a little like a firewall rule list where the addresses are file paths, and an SELinux context reads like a classification stamp on a document. Neither is optional on a node carrying regulated workloads.
Saying this in auditor language
An auditor asks whether mandatory access control is enforced on your container hosts. In operational terms they want four things: getenforce returning Enforcing or aa-status showing profiles in enforce mode, admission rejecting pods that ask for unconfined AppArmor, change control around profile edits, and denial logs shipped somewhere central. That last one does double duty. It proves the control was running during the audit window, and it hands a developer one specific line to fix instead of an excuse to request a blanket allow.
aa-status | head -6# example output:apparmor module is loaded.142 profiles are loaded.138 profiles are in enforce mode.4 profiles are in complain mode.
Complain mode is how you build a profile in the first place. It logs what would have been blocked and blocks nothing. On a production node that is the same posture as permissive SELinux wearing different vocabulary. Run a daily check that flags any production node sitting in complain or permissive, and treat every hit as a ticket rather than a note to yourself.
Field notes from real clusters
The line "root in a container is not root on the host" is doing a lot of quiet work, and it describes a stack rather than a single feature. User namespaces remap the UID. Mount namespaces hide most of the filesystem. Capabilities strip the privileged levers. LSM refuses the objects that survive all that. Pull the mandatory layer out and the sentence stops being true, even though the namespaces still look tidy on paper.
An attacker does not need a kernel exploit if an operator already handed them one. "Set SELinux permissive, only for this deploy" is a sentence that outlives the deploy by months. Nobody schedules the flip back, because nothing breaks when you forget, and the alert you would have gotten never fires. That is the quiet failure mode of every control you can turn off by hand.
AppArmor thinks in paths and profiles. SELinux thinks in labels and types. BPF-LSM thinks in programs you load and replace without shipping a policy package. All three answer at the same family of hooks. What differs is how you author policy and who has to reason about it at 3am. Start with whatever your node image already enforces, then add BPF-LSM only where your runtime tool genuinely needs live Kubernetes context to make the call.
A denial line is a gift. An AppArmor DENIED entry names the profile, the operation and the path, which is three quarters of a fix ticket already written for you. SELinux AVC messages do the same job with different field names. Get your on-call into the habit of pasting the denial into the ticket before anyone reaches for setenforce 0. That single habit prevents most of the multi-day outages that later get filed under "temporary workaround".
None of these layers is free, and none of them sees everything. seccomp costs close to nothing, because the filter is a tiny program on the syscall entry path. LSM hooks cost a bit more, since the check runs per object touched. A sloppy BPF-LSM program costs whatever you made it cost, on every file open, on every node. Measure on one node before you roll anything fleet-wide. Then remember the blind spot all three share: a process doing something ugly that no rule ever named. That is what Falco and Tetragon are for, and it is why observation and enforcement stay separate jobs.
A denial is not proof that somebody attacked you. Most of them are an installer writing where it has always written, or a sidecar reaching for a device nobody mentioned to the profile author. Treat the denial as a question rather than a verdict. Which process, which object, which permission, and does this app have any honest reason to want it? Sometimes the answer is a policy bug. Sometimes it is a foothold feeling for the edges of its box. You find out by reading the line, not by assuming either way.
Here is the failure that keeps repeating. An app cannot write to /tmp/cache. Someone runs setenforce 0 on the node so the release can ship. Two days later a container foothold reads host paths the policy would have refused, and the postmortem opens with a footnote about permissive SELinux. The fix was never a global flip. It was one typed allowance for that cache path, tested in complain mode on a single node, then rolled out to enforce. If setenforce 0 is still step one in your runbook, rewrite the runbook before the next rotation starts.
Reading a denial well takes practice. Three fields carry the meaning: the subject (the domain or profile the process runs under), the object (the path, socket or class it reached for), and the permission (write, exec, connect). Those three together tell you whether the app is wrong, the profile is wrong, or something is probing. Copy them into the ticket verbatim. A ticket saying "SELinux broke the app" teaches the next engineer to turn the control off. A ticket carrying the raw line teaches them to repair it.
The friction is real, so pick friction you can live with. Strict policy trips legacy installers that expect to rewrite system directories. Complain mode during a migration is an honest engineering choice. Complain mode eighteen months later is theatre. BPF-LSM buys you dynamic, Kubernetes-aware denials, and in exchange you now own a program lifecycle and a false-positive budget. Choose by what your team can operate on a bad night, not by which option lists the most acronyms in its README.
Keep the proof beside your other control evidence: a sampled getenforce or aa-status output per node, links to the tickets that denials turned into, and the date each profile moved from complain to enforce. When someone asks whether mandatory access control is on, you want to answer with files and timestamps rather than a memory of last quarter's design review.
Try this
Run this on a lab cluster or on one staging node. Read what comes back and sit with it for a minute. A first glance at a denial is nowhere near enough reason to touch production policy.
$ cat /sys/kernel/security/lsmlockdown,capability,yama,apparmor,bpf$ kubectl get nodes -o jsonpath='{.items[0].status.conditions[?(@.type=="Ready")].status}{"\n"}'True$ dmesg | grep -i apparmor | tail -2[ 412.101] audit: type=1400 apparmor="DENIED" operation="open" profile="k8s-nginx"
Takeaway
The thing to hold onto: LSM answers after the ordinary permissions have already said yes, and it can still say no. AppArmor, SELinux and BPF-LSM share those hooks and differ mainly in how you author policy. A denial is a line to read, never a reason to switch the layer off for the whole host.
Next, take one real denial on a lab node and pull it apart. Find the subject, the object and the permission inside it, and decide which of the three is wrong before you write a single line of policy. After that, move on to visibility, so you can see the syscalls that reach these hooks in the first place.