AppArmor & SELinux

Path-based and label-based container confinement.

Your office badge opens the lobby and your own floor. It does not open the server room, and your job title has nothing to do with it, because the reader on that door checks a policy list rather than your seniority. MAC (mandatory access control, policy that the person running a process cannot switch off, not even as root) works the same way inside Linux. A process that has become root inside a container can still be told no: this process type may not read that file label, end of conversation.

Two systems do this job on Linux, and they think about the world differently. AppArmor writes its rules about file paths. SELinux (Security-Enhanced Linux) writes its rules about labels attached to processes and files. Both run after seccomp (secure computing mode, the filter that decides which system calls a process may make at all): seccomp decides whether the call happens, MAC decides what that call is allowed to touch. Both exist to stop a compromised container from wandering around the host. The skill this lesson is about is fixing a denial with one small, specific rule instead of reaching for the permissive switch at 3 a.m.

AppArmor: rules written about paths

An AppArmor profile is a packing list taped to a process. These file paths, these capabilities (the individual root powers Linux hands out one at a time), these network families, nothing else. Anything you did not write down is denied, which keeps the profile readable at a glance and fits the way Kubernetes loads a Localhost profile from the node. On Ubuntu nodes the shipped runtime/default profile already blocks a pile of dangerous operations for ordinary containers. A custom profile goes further: no writes outside /tmp and /var/run, no mount, no raw sockets, no ptrace (attaching to another process to read its memory, the way a debugger does) across profile boundaries. Each of those lines is a corridor an attacker with a foothold no longer gets to walk down.

profile k8s-web flags=(attach_disconnected) {
  #include <abstractions/base>
  deny mount,
  deny network raw,
  /tmp/** rw,
  /var/run/** rw,
  /app/** r,
  /app/cache/** rw,
}

SELinux: rules written about labels

SELinux does not care what a file is called. It cares what the file is labelled. Every process and every object carries a security context made of a type, a role and a level, and the policy answers questions like "may container_t read var_log_t", never "does this path spell /var/log/app.log". Rename the file and the label rides along with it. svirt, the slice of policy that confines virtual machines and containers, gives each container its own MCS (Multi-Category Security) categories, so two containers both running as root cannot read each other's files even when the ordinary Unix owner and mode bits, DAC (discretionary access control), are set wrong. Red Hat platforms and OpenShift ship a targeted policy already tuned for containers. Permissive mode logs every denial without blocking anything, which is a gift while you debug and a quiet disaster if it is still set in production six months later.

kubectl exec -n prod deploy/payments-api -- cat /proc/1/attr/current 2>/dev/null || echo "no selinux attrs in container"
 
# example output:
system_u:system_r:container_t:s0:c512,c768

Turning MAC on in Kubernetes

Kubernetes 1.30 and later gives you appArmorProfile inside securityContext, set either to RuntimeDefault or to Localhost pointing at a profile already installed on the node. The SELinux type and level usually arrive from the container runtime, driven by the pod's scc (security context constraints, the OpenShift object that decides what a pod may ask for) or by securityContext.seLinuxOptions. Then check that the node actually enforces any of it. aa-status and getenforce each answer that in a line, and watching audit denials during a rollout tells you whether the profile matches how the app really behaves. Admission control can insist on RuntimeDefault AppArmor and reject any pod asking for an unconfined profile, with an exception only for a labelled break-glass namespace.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mac-demo
spec:
  securityContext:
    appArmorProfile:
      type: RuntimeDefault
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep","3600"]
EOF
 
# example output:
pod/mac-demo created

Reading a denial instead of switching MAC off

AppArmor writes its refusals to the kernel log and to audit. aa-logprof and aa-genprof read those and propose the rule you are missing, which beats guessing. SELinux refusals land in audit.log as AVC (Access Vector Cache) denials, and ausearch -m avc -ts recent prints the exact permission that was missing, so you are not reverse engineering it from an application stack trace. Write the smallest allow rule that unblocks the genuine need. Ship it through Git like any other change. Re-test. Then record why the rule exists, because a profile where every denial earned a blanket allow ends up full of holes shaped like last year's outages.

journalctl -k | grep -i "apparmor=DENIED" | tail -1
 
# example output:
Jul 24 04:31:02 node-12 kernel: apparmor="DENIED" operation="open" profile="k8s-web" name="/etc/shadow" pid=9912 comm="sh"

Where MAC sits in the hardening stack

Each control in the stack answers a different question. seccomp decides which system calls reach the kernel at all. Dropping capabilities takes away the privileged levers. MAC then limits what the calls that did get through are allowed to touch. Runtime tooling watches for behaviour nobody wrote policy about: a binary that appeared in /tmp ten minutes ago, an outbound connection to a C2 (command and control) address nobody has seen before. Keep MAC in enforce mode, tune it from audit evidence rather than from a hunch, and point a Falco rule at the paths your profile deliberately never allows. A denial and a detection firing on the same path is a much stronger lead than either alone.

securityContext:
  appArmorProfile: { type: RuntimeDefault }
  # container_t = confined workload (svirt). Never use spc_t here —
  # spc_t is Super Privileged Container (effectively unconfined).
  seLinuxOptions: { type: container_t, level: s0:c123,c456 }
  capabilities: { drop: ["ALL"] }
  readOnlyRootFilesystem: true

Keeping profiles current is an organisational problem wearing a technical hat. Every profile needs a named owner and an agreed response time for denial tickets, or the tickets rot. Your quarterly access review should cover who may edit profiles on the nodes, because that list grows quietly. When a team ships a new binary, the profile update belongs in the same change, ideally as a CI (continuous integration) check comparing the binary path against the AppArmor allow list or the SELinux file contexts. Skip that workflow and the profiles freeze while the applications keep moving. The ending is predictable: an outage, somebody with node access setting permissive mode to make it stop, and years of tuning undone in one panicked kubectl session.

Forensics on denials needs clocks that agree. If node time and SIEM (security information and event management, the platform your logs land in) time drift apart, your attack timeline comes out in the wrong order and you will argue about it for hours. During an incident, copy the audit logs off before you touch a profile, so you can still prove which denials happened before containment. Red teams earn their keep here. Let them try to escape AppArmor or SELinux through a badly written pod spec in the lab, then turn whatever worked into an admission constraint and a detection rule before somebody less friendly tries the same path in production.

Fleets that run more than one distro

A fleet with Ubuntu nodes and Red Hat nodes needs two playbooks, not one document that hand-waves at both. Your GitOps manifests, the Git repository that decides what runs, can branch securityContext by node selector, or you let admission inject the right RuntimeDefault for each operating system family. Either way, push profile changes to a canary node of each flavour first. A change that is harmless on one distro can be the reason nodes of the other kind start refusing writes.

kubectl get nodes -L kubernetes.io/os
 
# example output:
NAME     OS
d-node   linux
u-node   linux

Write the escalation path down before anybody needs it. A developer who wants a MAC exception files a ticket with the binary path, the business reason, and an expiry date on the approval. What you are heading off is the other version of that conversation: a direct message at 11 p.m. that ends with someone typing setenforce 0 on a production node.

Field notes from real clusters

One reframe makes the rest of this section click. Root inside a container is a statement about the container's own view of the world. The kernel is a separate audience, and MAC is the part of the kernel that was never impressed by uid 0 to begin with. Nearly every argument you will have about profiles traces back to that gap.

The two systems fail in different ways, and knowing which one you are holding saves an hour. AppArmor breaks when a path changes: a release moves a cache directory, the profile still names the old one, and the app returns an error that looks nothing like a permissions problem. SELinux breaks when a label is wrong: same path, same mode bits, denied anyway, because the context on the file does not match what the process type may touch. AppArmor rules travel with Debian and Ubuntu node images. SELinux rules travel with Red Hat style platforms, where svirt keeps two differently labelled containers out of each other's files even when both run as root.

Custom profiles belong in version control next to the application, reviewed like code, and rolled out complain first, enforce second. Complain mode logs what the profile would have blocked without blocking it, which is exactly how you find the three paths you forgot. Ship a brand new profile straight into enforce and you find those three paths at 2 a.m. with a pager in your hand.

Reading a denial is a skill, and it is worth practising on a calm day. Take the audit line, find the profile or the SELinux context it names, reproduce the failure, then make the smallest change that unblocks the real need and write down why you made it. "Make it unconfined" is not a fix, it is a regression with a bow on it. Keep a short runbook for aa-logprof and ausearch so application teams can do most of their own triage without asking anyone for node access.

Sooner or later a review asks why one workload needs seccomp and capability dropping and MAC all at once. Answer with failure modes, not with fashion. seccomp has nothing to say about a file the process is allowed to open. Capability dropping has nothing to say about a path the process was never meant to read. MAC has nothing to say about a system call it holds no opinion on. Each control covers a gap the others leave open, and runtime tooling watches the strange middle ground that none of them predicted.

Mixed fleets hurt in a specific way: half your incidents arrive in AppArmor vocabulary and half in SELinux vocabulary, and the engineer on call usually knows one of them well. Standardise node images where the business lets you. Where it does not, write the requirement as intent, "workloads run confined", and implement it twice. Make the admission policy fail closed, so a node that cannot apply the expected MAC type does not quietly keep taking pods.

Regulated workloads need proof rather than opinions. Export the count of nodes in enforce mode, the denial volume across the period, and the change tickets behind every profile edit. An auditor asking about confinement wants evidence that the control was switched on for the months under review. A well-argued design document is not that evidence.

The classic failure is a profile that sits in complain mode for six months because enforcing it can wait until after the freeze. Complain mode does teach you where the noise is. The problem is the dashboard upstairs showing a green tick next to "MAC enabled" while nothing on that node has ever been blocked. That is lying with formatting. Put a date in the calendar for enforce, track the denial count down toward zero, and escalate any profile that never leaves complain. Otherwise you paid the whole complexity bill and bought none of the control.

When aa-status or getenforce disagrees with your node bootstrap documentation, believe the node. Golden images drift, a daemonset that stamps labels does not change the LSM (Linux Security Module, the kernel framework AppArmor and SELinux both plug into) mode, and nobody notices until it matters. During an incident the first MAC question is never philosophical. It is whether this node was enforcing at the exact timestamp on the alert. Keep a small script that samples enforce state across the pool and shouts about the outliers.

MCS categories and svirt labels are the reason two root containers on a healthy Red Hat node still cannot read each other's files. Break the labels and you get denials that look exactly like application bugs, which is why a volume remounted with the wrong context can burn half a day before anyone suspects SELinux. Teach the storage and platform teams to set contexts on purpose. A blind chcon -R across a shared volume either opens a hole or breaks ten applications at once, and you will not know which until the tickets arrive.

The last habit is triage discipline. A denial is not automatically an application bug, and it is not automatically an intrusion either. Sometimes the app is genuinely wrong. Sometimes the profile never learned a new path after a library upgrade moved a file. Sometimes it is somebody probing to see how far the container reaches. Read the three fields, subject, object, permission, before you form an opinion, and you will stop flipping modes out of frustration. Frustration is how a mandatory control quietly becomes optional folklore.

Try this

Run this on a lab cluster or on one staging node. Read the output before you touch anything. A first look at a denial gives you information, not a diagnosis.

$ aa-status 2>/dev/null | head -5
apparmor module is loaded.
13 profiles are in enforce mode.
$ kubectl get pod payments -o jsonpath='{.status.phase}{"\n"}'
Running
$ dmesg | grep DENIED | tail -1
audit: apparmor="DENIED" operation="mount" profile="k8s-payments"

Takeaway

MAC confines root inside the container, which is the one thing seccomp and capability dropping cannot do for you. Stay in enforce mode, add exceptions you can justify by pointing at the denial line that caused them, and treat unconfined and permissive as incidents rather than conveniences.

Next step: check that every node reports the LSM modules you expect from aa-status or getenforce, and fix the odd one out before you move on to capabilities, namespaces and cgroups (control groups), the next wall in the same compartment.

Related