Container immutability at runtime

Read-only rootfs, dropped caps, no shell.

The first thing an attacker does after popping a shell inside a container is reach for their toolkit. curl down a second stage, drop a miner, rewrite the app binary so it beacons out. Every one of those moves needs the same thing underneath it: somewhere to write. Take writing away and most of the playbook stalls. An immutable container never changes after it starts. No new binaries land, no config gets rewritten, the root filesystem is mounted read-only. You get two wins from that one property. It hardens the workload, because there's nowhere to install tools or tamper with the running app. And it hands you a detection signal, because a container that's supposed to sit frozen has no honest reason to write to disk or open a shell. The moment one does, you have a high-confidence alarm instead of a maybe. A determined attacker can still run code that lives only in memory, so this isn't a magic wall. But memory-only payloads are noisier, they die on the next restart, and they can't silently swap your running binary for a backdoored one. You've pushed the fight onto ground where you hold the advantage.

You make a container immutable with the same securityContext fields you've already met, anchored by one: readOnlyRootFilesystem: true. There's a catch that trips everyone the first time. Almost every real app writes something while it runs, a temp file, a cache, a pid file. Bolt the whole root filesystem shut and the app crashes on boot. So instead of letting it write anywhere, you give it a couple of labeled drawers. Think of a hotel room where everything is bolted to the floor except the safe and one empty drawer. You mount an emptyDir or a volume at each path the app genuinely needs, and nothing else is writable. Now every writable location lives in the spec, in plain sight, where a reviewer can point at it and ask why. The same lock frustrates the intruder. They can't mark a downloaded payload executable on the root filesystem, can't edit /etc to wire up a cron job, can't overwrite the binary that's already running. Their options shrink to the two drawers you chose, and you already know to watch those.

apiVersion: v1
kind: Pod
metadata: { name: app }
spec:
  containers:
  - name: app
    image: registry.internal/app@sha256:9f2a...   # pinned by digest
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      capabilities: { drop: ["ALL"] }
    volumeMounts:
    - { name: tmp,   mountPath: /tmp }
    - { name: cache, mountPath: /var/cache/app }
  volumes:
  - { name: tmp,   emptyDir: {} }
  - { name: cache, emptyDir: {} }

$ kubectl apply -f pod.yaml
pod/app created
 
# Do what an attacker would: try to drop a file onto root
$ kubectl exec app -- sh -c 'echo pwned > /usr/local/bin/x'
sh: can't create /usr/local/bin/x: Read-only file system
command terminated with exit code 1
 
# The named drawers still work, exactly where you declared them
$ kubectl exec app -- sh -c 'echo ok > /tmp/scratch && cat /tmp/scratch'
ok

Stop the mutable pod at the door

One hardened pod means nothing if the next deploy ships a soft one. You need a gate that checks the setting before the pod ever runs, the way a bouncer checks a guest list before anyone gets inside. Pod Security Admission (PSA), the gate built into the API server, gets you most of the way with its restricted profile: non-root, dropped capabilities, no privilege escalation, a seccomp (secure computing mode) profile pinned to the runtime default. But read the standard closely and you'll notice it leaves readOnlyRootFilesystem out on purpose, because too many stock images can't run that way yet. So for that one field you write your own rule with Kyverno or Gatekeeper, and you write it against three lists, not one. A pod carries regular containers, init containers that run and exit before them, and ephemeral containers that kubectl debug attaches to a pod that is already running. A rule reading only spec.containers waves the other two through, and the ephemeral one is the interactive shell an attacker would ask for by name. The policy below checks all three.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-ro-rootfs }
spec:
  rules:
  - name: readonly-root
    # Pod/ephemeralcontainers is the subresource kubectl debug writes to (Kyverno 1.10+)
    match: { any: [{ resources: { kinds: [Pod, Pod/ephemeralcontainers] } }] }
    validate:
      failureAction: Enforce   # per-rule; spec.validationFailureAction is deprecated
      message: "readOnlyRootFilesystem must be true"
      pattern:
        spec:
          containers:
          - securityContext: { readOnlyRootFilesystem: true }
          =(initContainers):        # =() means "only check this list if it exists"
          - securityContext: { readOnlyRootFilesystem: true }
          =(ephemeralContainers):
          - securityContext: { readOnlyRootFilesystem: true }

$ kubectl apply -f kyverno-readonly.yaml
clusterpolicy.kyverno.io/require-ro-rootfs created
 
# Ship a pod without the setting, the way a careless deploy would
$ kubectl run bad --image=nginx --restart=Never
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/default/bad was blocked due to the following policies
 
require-ro-rootfs:
  readonly-root: 'validation error: readOnlyRootFilesystem must be true.
    rule readonly-root failed at path /spec/containers/0/securityContext/'
 
# Now the sneaky one: bolt a debug shell onto the pod that already passed
$ kubectl debug -it app --image=busybox:1.36 --target=app
error: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/default/app was blocked due to the following policies
 
require-ro-rootfs:
  readonly-root: 'validation error: readOnlyRootFilesystem must be true.
    rule readonly-root failed at path /spec/ephemeralContainers/0/securityContext/'

Watch the ones that slip through

Admission stops the pod you can see coming. It does nothing about the pod that was already compliant when it launched and then got compromised at runtime. That's where Falco earns its place. Falco watches the kernel's system calls and turns behavior into alerts, and an immutable container is the easiest thing in the world to watch. It's a motion sensor in a museum after closing: nothing should be moving, so any movement is worth a look. A shell starting inside a distroless image (a stripped image that ships no shell and no package manager) is not a false positive waiting to happen. It's an intruder, because there was no shell to run until someone put one there. This is why scope matters. A blanket 'shell in any container' rule buries you in noise, because plenty of normal images run shell scripts as their entrypoint. Point the same rule at your inert, immutable workloads and the false positives fall away, because those pods have no legitimate reason to spawn a shell at all.

- list: shell_binaries
  items: [sh, bash, dash, ash, busybox, zsh]
 
- rule: Shell in immutable workload
  desc: A shell started inside a container that should never run one
  condition: >
    spawned_process and container
    and proc.name in (shell_binaries)
    and container.image.repository = "registry.internal/app"
  output: "Shell in immutable pod (pod=%k8s.pod.name proc=%proc.cmdline image=%container.image.repository)"
  priority: CRITICAL
  tags: [container, runtime, mitre_execution]

# Reload Falco with the rule, then trip it on purpose
$ kubectl exec -it app -- sh
 
# On the node, Falco fires the instant the shell spawns:
$ kubectl logs -n falco ds/falco | tail -1
14:11:03.882: Critical Shell in immutable pod
  (pod=app proc=sh image=registry.internal/app)
 
# The write attempt needs no rule of its own: the kernel already
# blocked it, so a read-only workload stays quiet in the logs
$ kubectl logs -n falco ds/falco | grep -c 'immutable'
1

Memory-only malware still exists. Immutability raises cost and creates a clean signal when something tries to write anyway.

App needs writable cache? Mount a dedicated volume at the narrowest path that works. Don't count on noexec, the mount flag that stops files on a volume from being run, because an emptyDir gives you no way to set it and whatever lands in /tmp stays executable. That flag comes from a PersistentVolume or a CSI (Container Storage Interface) volume with mountOptions, so plan for that storage if you need it. On emptyDir, the drawers you opened are the paths you watch.

Policy engines should reject pods that claim to be hardened yet set readOnlyRootFilesystem: false without an exception label.

Combine immutability with image provenance. A read-only container running an unsigned mutable tag is still a supply-chain risk. Lock the filesystem and the digest together. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Run a read-only root pod, attempt to write to /, and confirm failure. Optionally add a detector rule for unexpected writes.

$ kubectl -n payments apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: frozen }
spec:
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep","3600"]
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 1000
      capabilities: { drop: ["ALL"] }
    volumeMounts:
    - { name: tmp, mountPath: /tmp }
  volumes:
  - { name: tmp, emptyDir: {} }
EOF
pod/frozen created
 
$ kubectl -n payments exec frozen -- sh -c 'echo pwn > /bin/pwn' || echo IMMUTABLE_OK
IMMUTABLE_OK
$ kubectl -n payments exec frozen -- sh -c 'echo ok > /tmp/x && cat /tmp/x'
ok

Takeaway

Immutable containers remove the attacker's toolkit drop zone. Combine read-only root with admission bans on mutable privileged pods.

Related