Pod Security Standards: enforce restricted without breakage

Roll out the restricted profile namespace by namespace, find violators with audit mode first, and carve exceptions that expire.

That failure mode is the whole argument for a phased rollout. Pod Security Admission enforces the three Pod Security Standards with one namespace label, and enforce rejects Pods, not the Deployment that creates them: kubectl apply succeeds, the ReplicaSet fails to create replicas, and the person who finds out is whoever notices the service has no endpoints. The warn and audit modes exist to move that discovery earlier, to the developer's terminal and to the audit log, while nothing is blocked.

Three profiles, and which namespaces get which

Pod Security Standards

Step one: warn and audit at the target level

Label the namespace with warn and audit at restricted and change nothing else. warn returns a warning on every kubectl apply that would fail under enforcement, including applies of Deployments and other workload objects, because warn and audit evaluate the Pod template inside them. audit records the same violations as annotations on the API server's audit events, which is the complete list, including the workloads nobody has redeployed since the label went on. Both modes admit the object; in the run the non-compliant Deployment was created with the warning below and its replica was available a second later.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.37
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.37

Harvesting the list needs the audit log to be on; the violation text is in the pod-security.kubernetes.io/audit-violations annotation of each event. One sprint of normal deploys is usually enough for every workload in the namespace to have been evaluated at least once.

Step two: fix the securityContext, not the policy

Every violation maps to a field. The pod-level securityContext takes runAsNonRoot: true and seccompProfile: { type: RuntimeDefault }; each container takes allowPrivilegeEscalation: false and capabilities: { drop: ["ALL"] }. An image that runs as root needs a numeric runAsUser in the manifest and a filesystem that user can use, which is an image change rather than a YAML change. For Helm-managed workloads the fix goes into the chart values or an upstream issue, otherwise the next helm upgrade reverts it.

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]

Step three: enforce, pinned to a version

When the audit log has been quiet for the namespace, add the enforce label. Pin enforce-version to a minor version rather than latest: the standards can tighten with a Kubernetes release, and latest means a cluster upgrade can start rejecting Pods that passed the day before. Keep warn and audit on latest so the upcoming tightening shows up as warnings first. Adding or changing an enforce label triggers a dry-run evaluation of the namespace's existing Pods, and the resulting warnings are your last chance to catch something before a Deployment rollout tries to replace those Pods. Two things about the version label from the run, on a 1.35.0 server: a label naming a version that server did not have yet (v1.37) was accepted without any warning and echoed in the violation text as restricted:v1.37, so in that run a wrong minor number was not caught by the API server; and 1.35 without the v was rejected outright with must be "latest" or "v1.x". The Pod Security Admission documentation says how a version is resolved; what the run shows is that the server will not catch the mistake for you, so pin deliberately to the version you actually run and review the label when the cluster is upgraded.

labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/enforce-version: v1.37
  pod-security.kubernetes.io/warn: restricted
  pod-security.kubernetes.io/warn-version: latest
  pod-security.kubernetes.io/audit: restricted
  pod-security.kubernetes.io/audit-version: latest

When enforce bites: recovery in order of preference

The right-hand column is where Gatekeeper or Kyverno come in, after PSA has set the floor. Their rollout discipline is the same one described above, under a different name: dry run, fix, then deny.

Related posts

Quick reference