Gatekeeper: Kubernetes admission

Constraints and templates in-cluster.

Conftest checks your YAML while the pipeline runs, and that is genuinely useful. It is also easy to walk around: anyone with kubectl, the command line tool that talks to the cluster, can run apply -f from a laptop and never touch CI (continuous integration, the automated build and test pipeline) at all. Gatekeeper is the bouncer standing at the door of the cluster itself. It is OPA (Open Policy Agent) running as a validating Kubernetes admission webhook, and optionally a mutating one, which means the API server calls it for permission before it writes anything down. Every create and every update meets your Rego first. The policy ships as CRDs (Custom Resource Definitions, new object kinds you teach Kubernetes to understand): a ConstraintTemplate carries the Rego, a Constraint says where that Rego applies and with what settings. A Pod that breaks the rule never gets scheduled, whichever door it came through.

The template and the constraint

A ConstraintTemplate is the recipe. It holds the Rego plus an OpenAPI schema, which is a short machine-readable description of the parameters the rule accepts and what type each one is, so another team can tune the rule without editing policy code. A Constraint is someone actually cooking that recipe: which kinds and namespaces it matches, what the parameters are set to, and the enforcementAction, one of deny, dryrun or warn. Gatekeeper compiles every template into OPA and evaluates the matching constraints each time the API server asks a question.

kubectl apply -f constrainttemplate.yaml
kubectl apply -f constraint.yaml

constrainttemplate.template.gatekeeper.sh/k8srequiredlabels created
k8srequiredlabels.constraints.gatekeeper.sh/must-have-owner created

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          required := input.parameters.labels[_]
          not input.review.object.metadata.labels[required]
          msg := sprintf("missing required label: %v", [required])
        }

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: must-have-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]
  enforcementAction: deny

What a rejection looks like

With enforcementAction: deny, a matching object gets a Forbidden response from the API server and nothing is stored. The text in that error is the string your violation rule built, so write messages the person on call can act on without opening your Rego. kubectl apply --dry-run=server sends the object all the way through the webhook and then throws the result away. It is the cheapest way to ask whether something would have been allowed.

kubectl apply -f pod-no-labels.yaml --dry-run=server

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [must-have-owner] missing required label: owner

Audit first, deny later

Audit is Gatekeeper walking through the cluster and re-checking objects that are already running, so you learn about the mess you inherited before you start blocking new work. Roll a new constraint out with enforcementAction: dryrun or warn. Read the violation counts on the constraint status and the metrics. Fix the workloads. Then flip to deny. Jump straight to deny on a busy cluster and the next deploy of every old object fails at once.

kubectl get k8srequiredlabels.constraints.gatekeeper.sh/must-have-owner -o jsonpath='{.status.totalViolations}'

14

kubectl patch constraint must-have-owner --type merge -p '{"spec":{"enforcementAction":"deny"}}'

constraint.constraints.gatekeeper.sh/must-have-owner patched

What happens when Gatekeeper is down

failurePolicy on the ValidatingWebhookConfiguration decides what the API server does when it cannot reach Gatekeeper at all. Fail means the door stays locked: every matching create and update is refused, across the whole cluster. Ignore means the door swings open and policy is skipped without a word to anyone. Both are defensible. Neither is something you want to discover during an outage. Choose one on purpose, run more than one Gatekeeper replica, and alert on webhook latency and error rate. This service now sits in the path of every single deploy you make.

Rewriting objects vs rejecting them

Gatekeeper can also change objects on the way in. Mutating policies (Assign, ModifySet) patch the object before it is stored, which is handy for filling in a default such as a missing label. Validation says no. Mutation quietly edits. Most teams stay validation only at first, because a rejection is easy to explain and an invisible edit is not, then add mutation once the deny rules have settled down.

The webhook needs a valid TLS certificate (Transport Layer Security, the same certificate machinery that sits behind HTTPS). If the CA bundle, the certificate authority record the API server trusts, goes stale, every apply fails closed. That looks exactly like a very strict policy until you go and read the API server logs. Rotate the certificates with cert-manager or with the install chart, and treat expiry as a date in the calendar rather than a surprise.

Ship policy the way you ship apps

Treat ConstraintTemplates and Constraints as ordinary manifests under GitOps, the practice of running a cluster from a Git repo where a controller syncs whatever is committed. Flux or Argo CD applies the CRDs and Gatekeeper compiles them on its own. While you are writing a rule, keep a copy of the template Rego in policy/ so opa test can run against it locally, then paste the finished rule into the CRD spec for deployment.

kubectl get constraints
kubectl describe constraint must-have-owner

NAME               ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
must-have-owner    deny                 0

Start with a narrow match: one namespace, one label selector, one kind. If the template turns out to be wrong, the damage stays small enough to fix in an afternoon. Inside the Rego, input.review.object is the whole object the API server was handed. For a Pod, input.review.object.spec is the pod spec exactly as it looks before the scheduler has seen it.

kubectl apply -f bad-pod.yaml --dry-run=server 2>&1 | head -3

Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied

Patch per environment with Kustomize

Keep the templates in Git and let Kustomize overlays, the small patch files that change a few fields of a base manifest per environment, vary the parameters. Dev runs the constraint in dryrun. Prod runs the same rule with deny. The required label list can differ too. Platform teams already patch application config this way, so there is no new workflow to teach anyone.

Gatekeeper publishes audit results to Prometheus, the metrics database most clusters already run, including totalViolations for each constraint. Alert when that number climbs, rather than waiting for admissions to start failing. A sudden jump after a cluster upgrade or a newly synced namespace usually means manifests drifted to a different API version, not that somebody is attacking you.

Install Gatekeeper into its own namespace with resource quotas on it. The audit and controller pods can eat a lot of CPU during a sync, and that noise should not land on a namespace shared with anything you care about. Check that your NetworkPolicy already allows apiserver → webhook and webhook → apiserver before you switch deny constraints on across the cluster.

Name the control plane namespaces as exclusions in the match spec from day one. When an incident is running, the temptation is to turn the webhook off entirely, and an exemption already written into the manifest is what stops that from being anyone's first move.

kubectl logs -n gatekeeper-system deploy/gatekeeper-controller-manager --tail=5

level=info msg="constraint updated" name=must-have-owner

An AssignMetadata mutation can add exactly the label your validating constraint then insists on, which is lovely when the order is right and baffling when it is not. Kubernetes runs mutating webhooks first and validating webhooks second, so a mutation that landed after validation would never have helped.

Private and air-gapped clusters cannot pull the public Gatekeeper chart or its images. Mirror both into your internal registry before install day, otherwise you find out during the install, with pods stuck trying to pull.

Make break-glass a resource rather than a habit. An exemption scoped to namespaces labelled break-glass=true, with a review date on it and every use recorded, is a much better emergency lever than someone deleting the ValidatingWebhookConfiguration at midnight and forgetting to put it back.

Upgrade Gatekeeper alongside the cluster, and always in a lower environment first. When the template API version changes, existing ConstraintTemplates can stop being valid without saying anything, and you learn about it on the next apply instead of during the upgrade.

kubectl explain constraint.template.gatekeeper.sh gives you field-by-field reference straight from the cluster when the published docs lag behind. The same trick works for a Constraint kind, because its schema is the OpenAPI block embedded in the template you already wrote.

Violation records live in etcd, the cluster's own database, and etcd is not an archive. If compliance asks for retention, export constraint violations to object storage nightly or ship them to your SIEM (Security Information and Event Management, the searchable warehouse where security teams keep logs).

Policies can target custom resources too, and Gatekeeper only looks at what your match block names. Lock down Pods but forget the CRD kinds your platform team invented, and people will happily deploy unscanned custom resources straight through the gap.

During a migration, a label such as policy.opt-out=true lets a namespace stay audited while not being denied. Give that label a sunset: a second policy that rejects any new opt-out namespace after your deadline, so the temporary escape hatch cannot quietly become permanent.

Across many clusters

The same ConstraintTemplate YAML goes to dev, stage and prod. The only field the Kustomize patch changes is enforcementAction. One Rego test suite covers all three, and enforcement graduates per environment instead of per hand-edited copy of the rule.

Run an audit and export the results before every Kubernetes upgrade. API deprecations change the shape of objects, and a deny rule that reads a field path which no longer exists stops matching quietly rather than failing loudly.

dryrun and warn exist so you can measure the damage before you cause it. During adoption, read the audit results every day and treat the count as a backlog with owners' names against it. A constraint sitting on four thousand violations is a migration plan, not a switch to flip on a Friday afternoon.

Most installs start at failurePolicy: Ignore, which is sensible while you are still restarting things and getting the webhook healthy. Leaving it there forever is how a broken control becomes a silent hole. Write down the conditions for moving to Fail and hold yourself to them: more than one replica running, alerts live on webhook latency and errors, constraints already sitting in deny without complaints.

Know which of your templates rewrite objects and which only reject them, because a mutation shows up as drift in an Argo CD diff. The stored object stops matching the file in Git, Argo CD reports it out of sync, and it can end up fighting the webhook. Label the mutating policies clearly so the next person reading that diff is not chasing a ghost.

Promote templates through the same Git path your applications use, cluster by cluster. A template that exists only because somebody ran kubectl apply on one cluster is drift, and drift in a policy control is the kind that turns into an incident.

Try this

Apply a ConstraintTemplate and its Constraint, then throw a non-compliant object at the cluster. Read two things: the denial the API server hands back, and TOTAL-VIOLATIONS on the constraint. That pair is your proof that Gatekeeper is enforcing rather than merely installed.

kubectl apply -f constrainttemplate.yaml
kubectl apply -f constraint.yaml
kubectl apply -f bad-pod.yaml
kubectl get k8srequiredlabels.constraints.gatekeeper.sh
kubectl get constraint -A

constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
constraint.constraints.gatekeeper.sh/ns-must-have-owner created
Error from server: admission webhook "validation.gatekeeper.sh" denied the request: [ns-must-have-owner] you must provide labels: {"owner"}
 
NAME              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
ns-must-have-owner   deny                 3

Takeaway

Gatekeeper packs your Rego into a ConstraintTemplate, and a Constraint decides where that rule bites. Start in dryrun, work the audit list down to something small, then move to deny, and keep watching failurePolicy so a webhook that is down does not stop enforcing without telling you.

Keep the same rule in Conftest so your pipeline catches it first. Gatekeeper is there for everything that skipped the pipeline anyway.

Related