Admission control internals

The pipeline, webhook order, and enforcement.

A developer who's allowed to create pods is not automatically allowed to create any pod she likes. Those are two different questions, and Kubernetes answers them at two different gates. Walk through an airport and you hit both. Your boarding pass gets checked first: are you ticketed for this flight? That's authorization. Then you step into the body scanner, which can turn you back even with a perfectly valid ticket. Admission control is the scanner. Authorization asks whether you may touch pods at all. Admission looks at the actual pod you handed over and decides whether that specific object gets to exist. The distinction sounds pedantic right up until it costs you a node.

The screening lanes, in order

Every write to the API server runs down a fixed set of lanes before anything gets saved. (The API server is the front door of the cluster, the process kubectl actually talks to.) Authentication comes first: who is this. Then authorization, usually RBAC, short for Role-Based Access Control, which decides whether this identity may run this verb on this resource. Then admission, where object-level policy lives. Admission runs in two rounds. Mutating admission webhooks go first. A webhook is just an HTTP callback: the API server POSTs the pod to your service, and your service can patch it, maybe injecting a logging sidecar or filling in a default value. After mutation, the object gets checked against the schema to confirm it's well-formed, and only then do validating admission webhooks vote to accept or reject. There are also built-in admission controllers compiled into the API server itself, but webhooks are how you plug in rules of your own. Whatever survives all of that is written to etcd, the key-value database that holds every object in the cluster. The order isn't an accident. Mutation runs before validation so the validator always judges the final, patched object, never a draft that some later mutator could quietly rewrite.

Turn a privileged pod back

Here's the gap RBAC leaves wide open. Say a build service account has create-pods in the dev namespace. Totally normal. With nothing else in place, that account can create a pod with privileged set to true, and a privileged container shares the node's kernel capabilities, a well-worn path to breaking out of the container and onto the node underneath it. RBAC said yes, because create-pods means create pods, full stop. Privileged isn't the only dangerous field, either. hostPath volumes mount the node's own filesystem into the pod, and hostPID and hostNetwork drop the pod straight into the node's process and network namespaces. RBAC waves all of them through, because none of them changes the verb or the resource, and the verb and the resource are the only things RBAC ever looks at. The object-level rule, no privileged containers, has to live somewhere else. That somewhere is admission. A ValidatingWebhookConfiguration is how you put it there: it registers your policy service with the API server and spells out exactly which requests get forwarded for a vote.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: deny-privileged.acme.io
webhooks:
  - name: validate.pods.acme.io
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail
    matchPolicy: Equivalent
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
        scope: Namespaced
    clientConfig:
      service:
        namespace: policy-system
        name: pod-policy
        path: /validate
        port: 443
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t

kubectl apply -f deny-privileged-webhook.yaml
kubectl run breakout --image=busybox --privileged -n dev -- sleep 3600

validatingwebhookconfiguration.admissionregistration.k8s.io/deny-privileged.acme.io created
Error from server: admission webhook "validate.pods.acme.io" denied the request: privileged containers are not permitted in namespace dev

The rejection is doing you a favor. It names the exact webhook that voted no, validate.pods.acme.io, and that's where troubleshooting starts. The message travels all the way back to kubectl, so whoever ran the command sees precisely why, no digging through server logs required. When a pod that should be fine gets bounced, or you just want to confirm which policy fired, pull the configuration and read what it matches and how it fails.

kubectl get validatingwebhookconfiguration deny-privileged.acme.io \
  -o jsonpath='{range .webhooks[*]}{.name}{" fail="}{.failurePolicy}{" resources="}{.rules[*].resources}{"\n"}{end}'

validate.pods.acme.io fail=Fail resources=[pods]

failurePolicy set to Fail is the setting you want here. It means that if your policy service is unreachable, the API server rejects the request instead of waving it through. There's a real tradeoff buried in that choice. Fail-closed protects you when the webhook is down, but it also means a broken policy service can freeze every pod create in the whole cluster, so you keep that service boringly reliable and scoped tight. Now flip to the detection side. Every one of these decisions lands in the API server audit log, and the audit record is more honest than the error a user sees. It shows that authorization said allow and admission still said no, which lets you pick out exactly the requests RBAC alone would have let onto your nodes.

jq 'select(.objectRef.resource=="pods" and .verb=="create" and .responseStatus.code>=400)
    | {user:.user.username, ns:.objectRef.namespace,
       authz:.annotations."authorization.k8s.io/decision",
       msg:.responseStatus.message}' /var/log/kubernetes/audit.log

{
  "user": "system:serviceaccount:dev:build-service",
  "ns": "dev",
  "authz": "allow",
  "msg": "admission webhook \"validate.pods.acme.io\" denied the request: privileged containers are not permitted in namespace dev"
}

Mutate first, then judge

You can watch the ordering happen for yourself. A MutatingWebhookConfiguration looks almost identical to the validating one, just with the kind changed and one extra internals knob: reinvocationPolicy, which asks to be called again if some later webhook changes the object. Set to IfNeeded, it covers the case where a mutator that ran early needs a second look because a later one touched the same field. It's a second chance, not a promise of order. Install a mutator that injects a default securityContext, then create a pod that never asked for one. The stored pod comes back with the injected field already set, which proves the mutator ran before anything hit etcd. And because validation runs after mutation, the validating lane judged that patched pod, not the bare one you typed at the terminal.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: default-nonroot.acme.io
webhooks:
  - name: mutate.pods.acme.io
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Ignore
    reinvocationPolicy: IfNeeded
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    clientConfig:
      service:
        namespace: policy-system
        name: pod-mutator
        path: /mutate
        port: 443
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t

kubectl run web --image=nginx -n dev
kubectl get pod web -n dev -o jsonpath='{.spec.securityContext.runAsNonRoot}{"\n"}'

pod/web created
true

Writing and running these webhook services yourself is a lot of moving parts: a pod that serves traffic over TLS (the encrypted transport your browser uses for HTTPS), a certificate authority bundle the API server has to trust, and a set of careful failurePolicy choices on top. Most teams don't hand-roll any of it. Kubernetes ships Pod Security Admission as a built-in validating controller, and policy engines like Kyverno and OPA Gatekeeper let you write these accept, reject, and mutate rules as plain data instead of code. Rules as data means you can review a policy change in a pull request and test it without compiling and shipping a Go binary first. That's exactly where the next lesson goes.

Try this

Work through “Mutate first, then judge” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: don't put your security in a mutating webhook. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related