Enforcement points

The same policy in CI, admission, and audit.

In short. Enforcement points in Policy-as-code at scale: The same policy in CI, admission, and audit.

Think about how an international airport treats your passport. It is checked when you book the ticket, again at check-in, again at the security line, again at the boarding gate — and sometimes once more on arrival. No single desk is trusted to be the *only* check, because every desk can be skipped under the right circumstances: online check-in, a rushed gate agent, a systems outage. Policy-as-code works the same way. An enforcement point is any place in your delivery pipeline where a policy decision is requested and — crucially — *acted on*: a merge blocked, an API request rejected, a violation surfaced as an alert.

The vocabulary comes from access-control literature. The policy decision point (PDP) is the engine that evaluates rules and returns allow or deny — OPA or Kyverno, which earlier lessons covered in depth. The policy enforcement point (PEP) is the component that intercepts an action, asks the PDP, and enforces the answer. That decoupling is the entire trick: one set of policies, authored and tested once, enforced everywhere. The remaining work is choosing and wiring the *where*.

Why one checkpoint is never enough

Each enforcement point, used alone, has a well-documented bypass. Enforce only in CI, and anyone with cluster credentials can kubectl apply a manifest that never saw a pipeline — so can a compromised CI token, a GitOps controller syncing an unreviewed branch, or an engineer acting mid-incident. Enforce only at admission, and developers get no feedback until deploy time (slow, expensive to fix), the thousands of resources admitted *before* the policy existed are never re-checked, and a webhook outage silently becomes a policy outage. Enforce only with runtime audit, and you are writing incident reports instead of preventing incidents.

Production setups therefore layer three canonical points: CI validation on every pull request, admission control at the cluster API server, and a continuous audit loop over what is already running. The model generalizes beyond Kubernetes — conftest evaluates Terraform plan JSON just as happily as manifests, and OPA sits behind Envoy's ext_authz filter to authorize service-to-service traffic — but the three-layer Kubernetes pattern is the one to build first.

Layer 1: fail fast in CI

conftest is a CLI that runs Rego policies against local files — YAML, JSON, HCL, Dockerfiles — with no cluster involved. It is the cheapest enforcement point you own: feedback lands in seconds, on the developer's branch, before anything exists to roll back. Pull the same versioned bundle your cluster enforces, then test the rendered manifests.

$ conftest pull oci://ghcr.io/acme/policy-bundle:v1.4.2   # downloads into ./policy/
$ conftest test deploy/api.yaml -p policy/
FAIL - deploy/api.yaml - main - Deployment "api": container "api" must set resources.limits.memory
FAIL - deploy/api.yaml - main - Deployment "api": set securityContext.runAsNonRoot=true
 
6 tests, 4 passed, 0 warnings, 2 failures, 0 exceptions

The exit code is non-zero on failure, so a required CI job blocks the merge with no extra glue. But be honest about what this layer is: it governs only changes that flow *through the pipeline*. Treat CI enforcement as a developer-experience feature — the fastest feedback loop — never as your security boundary.

Layer 2: admission control, the only synchronous gate

When any client creates or updates an object, the Kubernetes API server — after authentication, authorization, and mutation — serializes the request into an AdmissionReview JSON document and POSTs it over TLS to every registered validating webhook whose rules match. Gatekeeper and Kyverno are, at this layer, just HTTPS servers that answer allowed: true/false within a deadline. This is the *only* enforcement point that can prevent an object from ever existing, regardless of who created it or by what path — which is exactly why its failure behavior deserves scrutiny.

$ kubectl apply -f deploy/api.yaml
Error from server (Forbidden): error when creating "deploy/api.yaml":
admission webhook "validation.gatekeeper.sh" denied the request:
[require-memory-limits] Deployment "api": container "api" must set resources.limits.memory
 
# The webhook's failure mode is configuration, not code — inspect it:
$ kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration \
    -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.failurePolicy}{"\t"}{.timeoutSeconds}{"\n"}{end}'
validation.gatekeeper.sh	Ignore	3
check-ignore-label.gatekeeper.sh	Fail	3

Layer 3: audit what admission missed

The audit loop re-evaluates policies against objects that already exist — Gatekeeper's audit controller sweeps the cluster every 60 seconds by default and writes findings into each constraint's status. This catches the four populations admission structurally cannot: resources created before the policy, resources admitted while the webhook was down or set to Ignore, resources in exempted namespaces, and violations introduced by *changing the policy itself*.

$ kubectl get constraints
NAME                                                                    ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8srequiredresources.constraints.gatekeeper.sh/require-memory-limits   deny                 12
 
$ kubectl get k8srequiredresources require-memory-limits -o json | jq '.status.violations[0]'
{
  "enforcementAction": "deny",
  "group": "apps",
  "kind": "Deployment",
  "message": "Deployment \"legacy-worker\": container \"worker\" must set resources.limits.memory",
  "name": "legacy-worker",
  "namespace": "batch",
  "version": "v1"
}

Note the ceiling: constraint status stores a capped list of violations (20 by default, tunable via --constraint-violations-limit), so at scale you consume audit results through the exported Prometheus metrics or a violation-export sink, not by scraping status fields.

Hardening and what breaks at scale

Admission webhooks sit on the API server's hot path, so every hardening decision is a latency budget decision. Keep timeoutSeconds at 3 (never above 10 — a slow webhook stalls *every* matching request cluster-wide), and scope webhook rules narrowly: intercepting high-churn objects like events or leases multiplies webhook QPS for zero policy value. Watch apiserver_admission_webhook_admission_duration_seconds for creep as policy count grows. And instrument the *relationship between layers*: if audit violations climb while CI failures and admission denials stay flat, something is deploying around your pipeline — that divergence is your bypass detector, and it only exists because you enforce the same bundle at every point.

With engines chosen, policies tested, a lifecycle governing change, and enforcement layered from pull request to running workload, the single-cluster story is complete. The problems that remain are organizational rather than technical — distributing bundles to fleets of clusters, granting time-boxed exceptions without eroding the rules, and deciding who owns a policy when it blocks someone else's deploy. Those are the scale problems the rest of this course takes on.

Related