Performance, safety & the landscape
Fast, safe Rego; Sentinel & Kyverno.
Policy lives in the hot path, the stretch of code every single request has to walk through. Every admission review in your cluster waits on it. Every authorization check in your app waits on it. So the speed of your Rego (the language that OPA, the Open Policy Agent, evaluates) belongs on the same list as capacity and uptime, not on the list of things to tidy up later. A toll booth on the only bridge into town can search every trunk perfectly and still back traffic up for miles.
Measure before you rewrite. opa eval --profile shows you where the time actually goes. The usual trap is nested iteration over big lists, so reach for set lookups instead of scans. Partial evaluation, the --partial flag, does the fixed work once up front and leaves a smaller query for request time. Correctness still comes first. Latency decides whether a policy bug stays a bug or turns into an outage.
Profile before you rewrite
Guessing which line is slow is like guessing which appliance ran up the electricity bill. Put a meter on it. opa eval --profile prints the time and the number of evaluation steps per expression. Feed it input the size you really get: a real admission payload, a real authorization document, not a three-line toy. Then fix the lines the profiler points at and leave the rest alone.
opa eval -d policy.rego -i large-input.json --profile 'data.main.deny'
+------+---------+----------+| TIME | NUM EVAL| LOCATION |+------+---------+----------+| 4.2ms| 1240 | line 18 || 1.1ms| 380 | line 9 |+------+---------+----------+
Avoid accidental O(n²)
Checking every container against every row of a list is like finding a name in an unsorted stack of paper by reading the whole stack once per name. Ten containers against a thousand allowed images is ten thousand comparisons. That is what O(n squared) means: double the input and the work goes up fourfold. Key the data as a set or object instead, so each check is one hop: allowed_images[image] rather than image == allowed_list[_]. Lift shared work into helper rules too, so it runs once per query instead of once per turn of the inner loop.
cat > fast.rego <<'EOF'package main# slow: nested iteration# deny if { img := input.containers[_].image; not img == data.allowed[_] }# fast: set membershipallowed_images contains img if { img := data.allowed[_] }deny contains msg if {c := input.containers[_]not allowed_images[c.image]msg := sprintf("image %v not allowed", [c.image])}EOF
# indexed lookup pattern
Partial evaluation
A kitchen preps stock and sauces in the morning so each order is a short finish instead of a full cook. opa eval --partial does that for policy. It works through everything it can already decide from the data it holds, then hands back a residual, the leftover rules that still need the per-request input. That pays off when a big allowlist sits unchanged for hours while input changes on every call. Compile once in a sidecar, evaluate many times.
opa eval --partial -d policy.rego -i input.json 'data.main.allow'
# residual query + supported_rules in output — smaller eval at request time
Safety: fail closed under load
An admission webhook (the callback Kubernetes makes to your policy service before it accepts an object) runs on a stopwatch. The default budget is often 10 seconds, and plenty of teams dial it lower. Slow policy means timeout, and a timeout under failurePolicy: Fail blocks every admission in scope. Load-test that path after policy changes. Watch p99 latency (the slowest one request in a hundred) and rejection rates on their own graphs, kept apart from how many violations you are catching.
kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration -o yaml | grep timeoutSeconds
timeoutSeconds: 3
Other policy engines, and when they win
Kyverno writes Kubernetes policy as YAML resources, which is a relief for teams who will not learn Rego. HashiCorp Sentinel aims at Terraform Cloud and Enterprise. Checkov and KICS ship stacks of prewritten rules for IaC (infrastructure as code, the files that describe your servers, networks and clusters). OPA wins when one engine has to cover CI (continuous integration, the pipeline that builds and checks every commit), admission control, and running services, all sharing one set of tests. Choose breadth or choose a single-purpose tool on purpose. Doing both badly is the failure mode.
Regal and opa check for perf traps
Regal is a linter for Rego, a spell-checker for policy. It flags the slow shapes and the undefined-field mistakes before they reach production. opa check --strict catches compile errors while you still have the terminal open. Run both in CI next to a profiling smoke test on a large fixture input, so a slow rule fails a pipeline instead of a cluster.
regal lint --disable opa-fmt policy/
policy/k8s.rego:12: regal lint rule: prefer-set-membership
Twenty constraints that each walk every pod cost far more than one parameterized template doing the same job once. Treat a policy refactor the way you treat a change to a hot API: measure admission latency before, measure after, keep the numbers.
Keep OPA warm in sidecars
A long-lived OPA server pays the compile cost once and spreads it across millions of requests. Starting a fresh opa eval per request is like reheating the oven for every slice of toast, which is the wrong shape for an authorization hot path. Run OPA with --server and reuse HTTP connections from the app side instead of opening a new one each time.
ab -n 1000 -c 10 -p post.json -T application/json localhost:8181/v1/data/authz/allow
Requests per second: 8200 [#/sec]
Sentinel and Kyverno give up OPA's reach across domains in exchange for a nicer fit inside one. That trade is fine when your scope really is only Terraform Cloud, or only Kubernetes, and the rules stay modest. When the same team owns the pipeline, the cluster and service authorization, putting all of it in OPA means one language to remember and one test harness to keep alive.
Admission timeout budgets
The stock webhook timeout is often 10 seconds, though plenty of platform teams cut it to two or three so developers are not left staring at a spinner. Before you promote a policy, profile its p99 evaluation time at peak admission QPS (queries per second, how many requests arrive each second). If it runs past the budget, split the constraints up or narrow the match statements by kind, so OPA never gets handed objects it was never going to care about.
Caching in the app (remember an allow decision for one user and one resource for 30 seconds) buys throughput and pays for it in freshness. Write down the TTL (time to live, how long a cached answer stays usable) and clear the cache when roles change. OPA stays the source of truth. The cache is an optimization with a staleness risk you have agreed to in advance.
opa eval --stats prints the count of evaluation steps. Paste the before and after numbers into the pull request when you refactor, so reviewers argue about data rather than taste.
Some teams run both engines, and that can be a sound call: Kyverno handling the easy mutation defaults, OPA handling the cross-field checks that need real logic. If you split it that way, write the boundary into an ADR (architecture decision record, a short dated note explaining why a choice was made) so nobody has to guess which tool to extend next quarter.
To rehearse an admission storm after a cluster upgrade, generate manifests in bulk and push them through kubectl create --dry-run=server. The API server runs the full admission round trip and stores nothing.
Publish a maximum Rego evaluation time on the platform wiki and treat it as an SLO (service level objective, a number you promise to stay under). A policy pull request that busts the budget then owes you the same written justification you would demand from an app change that added 50ms to checkout.
If you have optimized and the budget is still gone, split the policy itself. The admission sidecar loads only the Kubernetes constraints, and the Terraform rules never compile there at all. Smaller bundles load faster, and a broken rule takes fewer things down with it.
In review, hunt for [_] nested inside [_]. When you find one, ask the author two questions: how big does that input get in production, and what did --profile say about the evaluation count?
Do the arithmetic on the whole round trip, not on evaluation alone. If Rego takes 0.5ms and the HTTP call to the sidecar adds 2ms, the network is your bill, and the in-process OPA SDK (software development kit, the library version you link straight into your service) may be worth reaching for where your language has one. You give up the clean isolation of a separate process to win those milliseconds back. Make that trade knowingly.
Give the sidecar a startupProbe that POSTs a representative input to /v1/data/authz/allow before the pod is marked ready. Otherwise the first real request of the day pays for loading and compiling the policy, and the first real request of the day is usually a customer.
Capacity planning
Deploys cluster. Monday morning after a frozen weekend looks nothing like Wednesday afternoon. Take your peak admissions per second, multiply by p99 evaluation latency, and you have the CPU your OPA replicas need to ride out the storm without queueing.
Measure evaluation time with decision logs switched on, then again with them off. Writing every input to the log at info level can cost more than the policy itself once authorization traffic gets heavy.
That hard timeout, often around three seconds, deserves a second look, because both ways out of it hurt. If your Rego routinely eats most of the budget, Kubernetes falls back on failurePolicy. Set to Fail, every request in scope is rejected and nobody can deploy. Set to Ignore, every request sails through unchecked and your guardrail is gone. Both are incidents. Profile against production-shaped objects, not the tidy YAML in your test fixtures.
Partial evaluation earns its keep when roles, allowlists and exception tables barely move between requests. Compile that static half ahead of time and leave a shorter residual query for each input. Then measure it. Some policies get much faster and some gain nothing at all, and the benchmark is the only way to tell which one you have.
Warmth needs help from your probes. A cold OPA paying compile cost on the first request of a burst will wreck p99 for that whole minute. Readiness should stay false until the policy is loaded. Liveness should not kill an OPA that is busy evaluating unless something is genuinely wrong with it.
The capacity arithmetic is dull and you still have to do it: requests per second times average evaluation time, plus headroom for the spike when a new bundle activates. Add replicas before you start inventing exotic Rego micro-optimizations. A second replica is cheaper than a policy nobody on the team can read.
Try this
Profile against an input the size you actually get. Try partial evaluation if most of your data sits still. Then benchmark before and after a rewrite so you can prove the rewrite helped. Fix what the profiler names, and expect nested comprehensions over long lists to be the villain.
opa eval -d policy.rego -i large-input.json --profile 'data.main.deny'opa eval -d policy.rego -i large-input.json --partial 'data.main.deny'opa bench -d policy.rego -i large-input.json 'data.main.deny' --count 50
# profile table: time + eval steps per expression# partial leaves a residual query with static data compiled away# bench prints ns/op style timings across iterations
Takeaway
Policy latency is availability. Swap repeated list scans for set lookups, keep OPA warm behind --server instead of spawning it per request, and stay inside the admission timeout so a slow rule never forces Kubernetes to pick between rejecting every deploy and waving everything through.
Speed does not excuse a policy that goes undefined the moment a field is missing. A fast wrong answer is still a wrong answer, and at admission time it is a wrong answer that ships.