Admission controllers

The gate that mutates and validates every write.

You apply a Deployment and it bounces back in half a second: Error from server: admission webhook "validate.kyverno.svc" denied the request: every container needs a memory limit. No Role said no. Nothing was written. That message came from admission control, the checkpoint every write crosses after Kubernetes already knows who you are and that you're allowed, but before it agrees the object exists.

Think of a building's mailroom. Security at the door checked your badge (that's authentication) and confirmed you're cleared for this floor, which is authorization, driven by RBAC (Role-Based Access Control). The mailroom is the last step. It opens every package, sometimes adds a routing label or a required stamp, sometimes refuses to deliver it, and only then files it in the log. Admission control is that mailroom. It runs inside the API server (the front door every kubectl command and every controller talks to), on every create, update, and delete. It's the last place your policy gets a say before the object lands in etcd, the cluster's key-value database.

Mutate first, then validate

Admission runs in two ordered passes, and the order is the whole trick. Mutating controllers go first. They can change the incoming object. One might inject a sidecar (a helper container that rides along next to yours), another might fill in a default memory request, a third might stamp on a label. Next the API server checks the now-modified object against its schema. Then validating controllers run and get a plain yes-or-no vote, accept or reject, no edits allowed. Because mutation always happens before validation, a validating rule always sees the final object, sidecars and defaults included, not the half-formed thing you submitted. One subtlety a busy admin hits eventually: if a mutating webhook edits an object, other mutating webhooks can be re-invoked (that's reinvocationPolicy: IfNeeded), because one plugin's change might make another's rule apply.

$ kubectl get validatingwebhookconfigurations
NAME                                       WEBHOOKS   AGE
kyverno-resource-validating-webhook-cfg    1          84d
kyverno-policy-validating-webhook-cfg      1          84d
 
$ kubectl get mutatingwebhookconfigurations
NAME                                     WEBHOOKS   AGE
istio-sidecar-injector                   1          112d
kyverno-resource-mutating-webhook-cfg    1          84d

The plugins baked into the API server

Most of admission is compiled straight into the API server. These built-in plugins are toggled with the --enable-admission-plugins flag and cover a lot of everyday behavior. NamespaceLifecycle stops you creating objects in a namespace that's being deleted. ResourceQuota enforces per-namespace limits. LimitRanger applies default requests. ServiceAccount wires the identity token into each Pod (a Pod is Kubernetes' smallest unit: one or more containers that share a network identity). NodeRestriction stops a compromised kubelet, the agent on each node, from editing other nodes. PodSecurity enforces the Pod Security Standards. Two of the built-ins are special: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. Those are the plugins that call out to code you supply.

One gotcha reading the flag: it doesn't list everything that's on. It only adds to (or with --disable-admission-plugins, removes from) a default set that's already compiled in and enabled, which includes PodSecurity, ResourceQuota, ServiceAccount, and both webhook plugins. So a flag that shows just NodeRestriction does not mean admission is barely running.

$ kubectl -n kube-system get pod kube-apiserver-cp-1 \
    -o jsonpath='{.spec.containers[0].command}' | tr ' ' '\n' | grep admission
--enable-admission-plugins=NodeRestriction

Adding your own rules

There are two ways to plug in policy the built-ins can't express. The older way is a webhook: you register a ValidatingWebhookConfiguration or MutatingWebhookConfiguration that tells the API server "for every Pod create, POST it to my HTTPS service and do what it says." That service is usually a policy engine like Kyverno or OPA Gatekeeper (OPA = Open Policy Agent), so you write policy as data instead of running your own server. The newer way is built into Kubernetes itself and has been generally available since v1.30. It skips the network hop entirely. ValidatingAdmissionPolicy lets you write the rule in CEL (Common Expression Language, a small language for writing yes-or-no checks) and the API server evaluates it itself. No webhook pod to keep alive, no HTTPS certificate to rotate, no extra failure mode. Webhooks aren't dead, though. They can mutate objects, reach out to external data, and run logic too tangled for a single expression, so plenty of clusters run both side by side.

A policy doesn't have to see every object. namespaceSelector and objectSelector narrow what it matches, and you almost always exclude kube-system so a broken rule can't take down the control plane's own pods. On webhooks, matchConditions (also CEL) can filter finer still, so the backend only gets called for requests that actually matter.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-memory-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        operations:  ["CREATE", "UPDATE"]
        resources:   ["pods"]
  validations:
    - expression: "object.spec.containers.all(c, has(c.resources.limits) && has(c.resources.limits.memory))"
      message: "every container must set a memory limit"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-memory-limits-binding
spec:
  policyName: require-memory-limits
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels: { environment: prod }

$ kubectl apply -f require-memory-limits.yaml
validatingadmissionpolicy.admissionregistration.k8s.io/require-memory-limits created
validatingadmissionpolicybinding.admissionregistration.k8s.io/require-memory-limits-binding created
 
$ kubectl label ns prod environment=prod --overwrite
namespace/prod labeled
 
$ kubectl -n prod run nolimits --image=nginx:1.27
The pods "nolimits" is forbidden: ValidatingAdmissionPolicy 'require-memory-limits'
  with binding 'require-memory-limits-binding' denied request:
  every container must set a memory limit

When it breaks

The first move when writes start failing is to read the rejection, because it names its own source. A webhook denial reads admission webhook "..." denied the request:. A CEL policy denial names the ValidatingAdmissionPolicy and its binding, exactly like the message above. That tells you which gate fired. Then check that config's failurePolicy and timeoutSeconds, because those two decide what happens when the backing service is slow or unreachable.

$ kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
    -o jsonpath='{range .webhooks[*]}{.name}{"  "}{.failurePolicy}{"  "}{.timeoutSeconds}{"\n"}{end}'
validate.kyverno.svc-fail  Fail  10

If that config points at a Service with no healthy pods behind it, every matching request waits out the timeout and then failurePolicy decides. Fail rejects it. Ignore lets it through with no policy applied at all. The nastiest version is a fail-closed webhook whose own backend is down: nothing its rules match can be created, including a replacement for the webhook itself, so the outage blocks its own fix. Before touching the config, confirm the backend actually has endpoints with kubectl -n kyverno get endpoints.

A dead fail-closed webhook blocks creates cluster-wide. Monitor webhook latency and availability.

Mutating then validating order matters. Never assume a mutate ran if validate failed first in your mental model — learn the chain.

Lab with dry-run and policy audit modes before enforce. Friday evening enforce is how outages start. Monitor webhook latency and availability.

Mutating then validating order matters.

Try this

List validating and mutating webhook configurations. Apply a pod that your policy should reject and capture the deny message from the API.

$ kubectl get validatingwebhookconfigurations
NAME                                       WEBHOOKS   AGE
kyverno-resource-validating-webhook-cfg    1          84d
kyverno-policy-validating-webhook-cfg      1          84d
 
$ kubectl get mutatingwebhookconfigurations
NAME                                     WEBHOOKS   AGE
istio-sidecar-injector                   1          112d
kyverno-resource-mutating-webhook-cfg    1          84d
 
$ kubectl -n kube-system get pod kube-apiserver-cp-1 \
    -o jsonpath='{.spec.containers[0].command}' | tr ' ' '\n' | grep admission
--enable-admission-plugins=NodeRestriction
 
$ kubectl apply -f require-memory-limits.yaml
validatingadmissionpolicy.admissionregistration.k8s.io/require-memory-limits created
validatingadmissionpolicybinding.admissionregistration.k8s.io/require-memory-limits-binding created
 
$ kubectl label ns prod environment=prod --overwrite
namespace/prod labeled
 
$ kubectl -n prod run nolimits --image=nginx:1.27
The pods "nolimits" is forbidden: ValidatingAdmissionPolicy 'require-memory-limits'
  with binding 'require-memory-limits-binding' denied request:
  every container must set a memory limit
 
$ kubectl get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg \
    -o jsonpath='{range .webhooks[*]}{.name}{"  "}{.failurePolicy}{"  "}{.timeoutSeconds}{"\n"}{end}'
validate.kyverno.svc-fail  Fail  10

Takeaway

Admission mutates and validates after authz. Webhooks extend policy; built-in controllers cover defaults and limits.

Related