PSA, Kyverno & Gatekeeper

The pod-hardening floor plus custom policy.

Role-Based Access Control (RBAC, the system that decides who is allowed to do what) will happily let a developer create a pod. It has nothing to say about whether that pod runs as root, mounts the host's disk, or flips on privileged mode. Something else has to answer that second question, and this lesson is about the three tools that do. Think of a bouncer working from a written rulebook. A few rules are house standard, checked at every single door. At some doors the bouncer only warns you and waves you through. At others you're turned away flat. That rulebook is admission control: the stage where Kubernetes inspects an object after you've been allowed to submit it, but before it's saved to the cluster. For an attacker who already has permission to create pods, admission control is the wall standing between "I can make a pod" and "I can make a pod that owns the node." Pod Security Admission is the rulebook that ships in the box. Kyverno and Gatekeeper are the rules your own venue writes.

Pod Security Admission: the built-in floor

Pod Security Admission (PSA) is the built-in replacement for PodSecurityPolicy (PSP), the older feature Kubernetes ripped out in v1.25. PSA invents no rules of its own. It enforces the Pod Security Standards, which are three named profiles you switch on by putting a label on a namespace. The privileged profile allows anything. The baseline profile blocks the settings that are plainly dangerous: sharing the host's process or network namespaces, privileged containers, most host mounts. The restricted profile is the hardened floor. It insists a pod run as a non-root user, forbid privilege escalation, and drop every Linux capability (the fine-grained slices of root power, like changing file ownership or binding to low ports). It also wants a seccomp profile (a filter that limits which system calls a container can make) set to RuntimeDefault, and no hostPath volumes at all. Each profile runs in up to three modes at once, and the modes behave quite differently. In enforce mode the pod is rejected outright. In warn mode it's admitted, but whoever ran kubectl gets the violation printed back at them. In audit mode the user sees nothing, while a line describing the violation lands in the audit log. That separation is what makes a rollout safe. Switch on warn and audit first, watch what would break, then flip enforce when you're ready. Skip that staging and you learn which of your workloads were quietly non-compliant by watching them fail to schedule in production.

$ kubectl label namespace prod \
    pod-security.kubernetes.io/enforce=restricted \
    pod-security.kubernetes.io/enforce-version=v1.31 \
    pod-security.kubernetes.io/warn=restricted \
    pod-security.kubernetes.io/audit=restricted
 
namespace/prod labeled

$ kubectl apply -n prod -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: shell
spec:
  volumes:
    - name: host
      hostPath: { path: / }
  containers:
    - name: shell
      image: busybox:1.36
      command: ["sleep", "3600"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: host
          mountPath: /host
EOF
 
Error from server (Forbidden): error when creating "STDIN": pods "shell" is forbidden: violates PodSecurity "restricted:v1.31": privileged (container "shell" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "shell" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "shell" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "shell" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "shell" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Two of those modes make PSA safe to turn on without calling a war room. warn sends the violation straight back to whoever ran kubectl, so they see it at apply time, on their own screen. audit writes it into the API server's audit log, which is the running record of every request the server has handled. Set audit=restricted across the whole cluster before you flip a single enforce label, and you've built an exact inventory of which workloads would break, without breaking one of them yet. This is the difference between a control you can turn on during a quiet Tuesday and one that pages you at 2 a.m. because half of production stopped scheduling. Here's how you pull those violations back out of the audit log to preview what an enforce rollout would catch.

$ jq -c 'select(.annotations["pod-security.kubernetes.io/audit-violations"])
  | {user: .user.username, ns: .objectRef.namespace, pod: .objectRef.name,
     viol: .annotations["pod-security.kubernetes.io/audit-violations"]}' \
  /var/log/kubernetes/audit.log
 
{"user":"[email protected]","ns":"prod","pod":"shell","viol":"would violate PodSecurity \"restricted:v1.31\": privileged (container \"shell\" must not set securityContext.privileged=true), runAsNonRoot != true (pod or container \"shell\" must set securityContext.runAsNonRoot=true)"}

Custom rules: Kyverno and Gatekeeper

PSA stops there. It understands a pod's security settings and nothing else about your business. It can't say "only images from our own registry," or "every pod needs a cost-center label," or "no public load balancer in this namespace." For rules like those you install a policy engine. The engine hooks into that same admission stage as a webhook, which means Kubernetes pauses on each matching request and phones out to the engine for a yes or a no before it saves anything. The engine also teaches the cluster brand-new object types through Custom Resource Definitions (CRDs, the mechanism that lets you add your own kinds of objects next to the built-in ones like Pod and Service). The two popular engines take different routes to the same place. Kyverno writes policy as ordinary Kubernetes YAML: match some resources, then validate, mutate, or generate. Open Policy Agent Gatekeeper (OPA is the general-purpose policy project sitting underneath it) writes policy in Rego, a small query language built for exactly this, wrapped in reusable ConstraintTemplates. The good habits are identical whichever you pick. Keep the policy as reviewed, version-controlled code, ship it in audit before enforce, and make every exception both narrow and time-limited.

$ kubectl apply -f - <<'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-registries
spec:
  background: true
  rules:
    - name: allowed-registry
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Audit          # roll out in Audit, then flip to Enforce
        message: "Images must come from registry.acme.io"
        pattern:
          spec:
            containers:
              - image: "registry.acme.io/*"
EOF
 
clusterpolicy.kyverno.io/restrict-registries created

$ kubectl run badimg --image=docker.io/library/nginx:1.27 -n apps
pod/badimg created
 
$ kubectl get policyreport -A -o json \
  | jq -r '.items[].results[] | select(.result=="fail")
           | "\(.policy) FAIL \(.resources[0].name)"'
restrict-registries FAIL badimg

$ kubectl patch clusterpolicy restrict-registries --type=json \
    -p='[{"op":"replace","path":"/spec/rules/0/validate/failureAction","value":"Enforce"}]'
clusterpolicy.kyverno.io/restrict-registries patched
 
$ kubectl run badimg2 --image=docker.io/library/nginx:1.27 -n apps
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/apps/badimg2 was blocked due to the following policies
 
restrict-registries:
  allowed-registry: 'validation error: Images must come from registry.acme.io.
    rule allowed-registry failed at path /spec/containers/0/image/'

That leaves you with three tools sitting at different heights. PSA is fixed. You don't add rules to it, you only pick a profile, and that limit is the whole point: its job is the universal pod-hardening floor every namespace should stand on. Kyverno and Gatekeeper are where your own organization's rules live. Kyverno keeps everything in YAML and can mutate and generate objects on top of validating them, which makes it the gentler on-ramp for most teams. Gatekeeper trades some of that ease for Rego, which handles cross-object logic (checking a new pod against every Service that already exists, say) that Kyverno expresses less naturally. It records its audit findings in each Constraint's status field, so a plain kubectl describe on a constraint shows you what's failing. Running both engines at once means two systems to reason about on every single admission, so most clusters run one and layer it over PSA. And both engines answer through admission webhooks, which turns their own uptime, and their fail-open or fail-closed setting, into the next thing an attacker pokes at. That's where this course heads next.

Try this

Work through “Custom rules: Kyverno and Gatekeeper” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: pSA checks the pod, so a Deployment can be admitted while its pods die in silence. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related