The decision API & bundles

OPA as a service; distributing policy.

Everything so far ran policy as a one-shot errand. You hand opa eval some input, read the answer, and the process exits. OPA (Open Policy Agent, the policy engine you have been driving from the command line all course) can work that way forever on your laptop. A production app cannot. Starting a whole process for every request is like hiring a courier each time you need to look up a phone number. Real systems treat policy the way they treat DNS (the Domain Name System, the always-on lookup service that turns a name into an address): one long-running answer machine sits nearby, and you ask it over the network.

Two pieces make that happen. opa run --server starts OPA listening on port 8181 and exposes the Decision API (a plain HTTP endpoint you POST a question to and read an answer from). Bundles are versioned tarballs, your Rego files plus a data.json, that OPA downloads, verifies, and swaps in while it keeps serving. Nothing restarts. A sidecar OPA (a second container running next to your app inside the same pod) answers on localhost:8181, and it never forks opa eval per request.

Policy on a port: the Decision API

Start OPA with --server and every policy it has loaded appears as a tree of documents under /v1/data. The URL path is the same path you queried in the REPL (the interactive OPA prompt): package authz, rule allow, so /v1/data/authz/allow. You POST the facts about the request wrapped in a top-level input key, and OPA replies with {"result": ...}. Run it as a sidecar and that call never leaves the pod, so a warm engine answers well inside a millisecond. The policy below opens with default allow := false, and that one line is why the second call answers false instead of coming back empty. Further down you will see what the same request does without it.

cat > authz.rego <<'EOF'
package authz
 
default allow := false
 
allow if {
    "editor" in input.user.roles
    input.action == "write"
}
EOF
 
opa run --server authz.rego &
sleep 1
curl -s localhost:8181/v1/data/authz/allow \
  -H 'Content-Type: application/json' \
  -d '{"input": {"user": {"roles": ["editor"]}, "action": "write", "resource": "docs/x"}}'

{"result": true}

curl -s localhost:8181/v1/data/authz/allow \
  -H 'Content-Type: application/json' \
  -d '{"input": {"user": {"roles": ["viewer"]}, "action": "write", "resource": "docs/x"}}'

{"result": false}

Build a bundle

opa build -b takes a directory and produces one gzipped tarball: your .rego files, your data.json, and a .manifest that records which parts of the data tree this bundle owns. Build it in CI (continuous integration, the pipeline that runs on every commit), test it, version it, sign it, publish it. Open one up once so the format stops feeling like magic. It is files in a tarball with a small index at the front.

opa build -b policies/ -o bundle.tar.gz
tar tzf bundle.tar.gz

/.manifest
/authz.rego
/data.json

opa build -b policies/ --signing-key private.pem -o bundle.tar.gz

# signed bundle — OPA verifies before activate

Point OPA at a bundle service

A short config file tells OPA where to shop. You name one or more services (an S3 bucket on Amazon, a Google Cloud Storage bucket, an OCI registry from the Open Container Initiative, the same kind of registry that already holds your container images, or a plain nginx serving files) and the bundles to fetch from each. OPA pulls once at boot, then polls on a randomised interval so a thousand sidecars do not all knock in the same second. It checks the signature and activates the new policy in one atomic step. Gate your readiness probe on /health?bundles=true so a pod holding no policy never joins the load balancer. Read the status code there rather than the body: a healthy OPA answers 200 with an empty JSON object, and an unhealthy one answers 500. An empty body from the health endpoint means nothing is wrong, which is the opposite of what an empty body means from the Decision API.

cat > config.yaml <<'EOF'
services:
  - name: bundle-registry
    url: https://bundles.example.com
bundles:
  authz:
    service: bundle-registry
    resource: authz/bundle.tar.gz
    polling:
      min_delay_seconds: 30
      max_delay_seconds: 60
EOF
 
opa run --server -c config.yaml

# server starts; polls for bundle

curl -s -w '\nHTTP %{http_code}\n' 'localhost:8181/health?bundles=true'

{}
HTTP 200

On your own machine, opa run --watch reloads policy straight from disk and no bundle is involved. Production uses bundles because they carry provenance: a revision, a signature, a manifest. Keep the two paths straight in your head. A rule that behaves beautifully under --watch but never lands in the built artifact never reaches a single sidecar.

Undefined on the wire: fail closed

Ask for /v1/data/authz/allow when allow is undefined for that input and OPA answers HTTP 200 with an empty object, {}, and no result key at all. There is the trap. A client that checks response.result == false finds no false, falls through, and lets the request in. Two habits close it. Write default allow := false so the rule always has a value, and make every client wrapper treat a missing result key as a deny before it treats it as anything else.

curl -s localhost:8181/v1/data/bad/allow -d '{"input":{"user":"stranger"}}'

{}

Test the bundle you ship

Testing loose files proves the files pass. It does not prove the tarball you are about to publish passes. opa test -b bundle.tar.gz runs your tests against the compiled artifact itself, so a policy file that never made it into the build fails loudly instead of quietly. Wire it into CI right after opa build and before the publish step. Ask for coverage in a second run rather than the same one: --coverage replaces the friendly PASS line with a JSON coverage report, and --threshold 85 makes the command exit non-zero when the overall figure drops below 85 percent.

opa test -b bundle.tar.gz
opa test -b bundle.tar.gz --coverage --threshold 85 | jq .coverage

PASS: 8/8
91.2

The polling interval is a straight trade: shorter means fresher policy and more load on the bundle server. Thirty to sixty seconds is normal for sidecars. In locked-down environments an OCI registry is often the easiest source to get approved, because the image pull credentials your cluster already carries work for policy too.

Decision logs

opa run --server -c config.yaml --set decision_logs.console=true

# JSON decision logs to stdout for Loki/Datadog

Decision logs turn every query into an audit record: a summary of the input, the path that was asked for, the answer, and the bundle revision that produced it. Ship them to your SIEM (security information and event management, the system your security team already searches for audit trails) the same way you ship API gateway logs. When somebody asks why their deploy was rejected at 14:02 on Tuesday, the decision log plus the revision answers it in minutes instead of a week spent reading old Rego.

Running more than one bundle

One OPA can pull several bundles at once, each on its own polling interval. Large organisations usually land on three: service authorization, Kubernetes admission rules, and the Terraform checks that run in CI. Each activates independently, so a change to one does not reload the others. The one hard rule is that their roots must not overlap.

Inside the tarball, .manifest declares which roots under data this bundle owns, plus the revision string that later shows up in decision logs. Ownership is exclusive. If a second bundle claims a root this one already holds, activation fails and OPA keeps serving the last policy that did activate. Plain /health still says the process is fine, because the process is fine. Only /health?bundles=true knows whether the policy is current, which is why that is the URL your readiness probe should call.

The reason to split bundles is usually people rather than bytes. The identity team owns an allowlist that changes hourly. The platform team owns admission rules that change every few weeks. Give each its own bundle and the chatty one can poll fast without dragging the slow one through a reload every time, while a bad package from one team cannot land on another team's roots.

Sizing an OPA sidecar

OPA's memory footprint tracks the size of the policy and data it holds, so load test with bundles the size you actually ship rather than a toy one. Then set Kubernetes memory limits deliberately. A sidecar that is killed cleanly and restarts is recoverable. One that runs out of memory halfway through a request hands the outcome to the admission webhook's failurePolicy (the field that decides whether Kubernetes allows or blocks a request when the webhook does not answer).

Agree a maximum bundle size and write it down where your team will find it. Activation is not free: a big data set has to be parsed and indexed on every successful poll, so the two numbers worth watching are resident memory per sidecar and the gap between a poll finishing and the new policy going live. A 50 MB data.json full of exception rows moves both. When either drifts, the fix is rarely a bigger memory limit. Pull the large static reference table into its own slow-polling bundle, keep the small allowlist that actually changes on a fast schedule of its own, and set limits so an oversized sidecar gets restarted rather than left half-serving undefined under load.

The Horizontal Pod Autoscaler (HPA, the Kubernetes controller that adds and removes replicas) scales on CPU by default. OPA's pressure point is memory once bundles grow, so watch for OOMKilled (the status Kubernetes records when the kernel kills a container for using too much memory) in your sidecar metrics instead of trusting a CPU graph that looks calm.

Locking down the bundle path

mTLS (mutual TLS, where the client presents a certificate too, so both ends prove who they are) between OPA and the bundle server stops anyone sitting on the network path from swapping the tarball in flight. In the OPA config, the certificate OPA presents lives under that service's credentials.client_tls block, as a cert and private_key pair, while the services.tls block beside it holds the trust side, where ca_cert names the authority you accept for the server's own certificate. It sits alongside signing rather than replacing it: signing proves who built the bundle, mTLS proves who you are talking to.

Signing closes the gap between "CI built this" and "this sidecar is running it." Generate the key pair offline, keep the private half in your CI secret store, hand the public half to OPA through its config, and require verification before activation. After that, somebody who steals write access to the bundle bucket can upload whatever they like and OPA will refuse every byte of it. All of that depends on verification being required in the OPA config. If you leave it switched off, a tampered bundle and a signed one are treated exactly the same.

Give the OPA pod a network policy that permits egress to the bundle URL and nothing else. Polling keeps working, and a compromised OPA cannot phone home or ship the input documents it sees, which contain user identities and request bodies, anywhere you did not choose.

Back up the bundle registry and the signing keys at the same disaster recovery tier as your container registry. Losing the bundle history takes rollback off the table during an outage your own policy caused. Practise restoring into a blank OPA instance once a quarter.

Rolling a new bundle out

A bundle behaves like a container image tag. The tarball is immutable once published, and the name authz/bundle.tar.gz, or an OCI tag like v42, is only a pointer to it. When CI promotes v42, every sidecar that polls converges on it within a poll interval or two, and not one application process restarts. That separation is why platform teams like bundles.

Blue/green works for sidecars the same way it works for apps. Run two deployments pinned to different bundle tags, wait until the green one reports healthy on /health?bundles=true, shift traffic across, and leave blue running so rollback is one traffic switch rather than a rebuild.

For a canary, split the config so roughly 5% of sidecars point at the new bundle tag, and do it before the fleet-wide poll interval elapses. Evaluation errors and accidental denies then show up on a handful of pods rather than all of them at once.

Roll new bundles out the way you roll out apps. Promote the tag on a small subset of the Deployment, watch decision-log error rates and query latency for a few minutes, then widen. Keep the previous artifact published so a rollback is only a change of pointer.

Test the delivery path too

Point one integration test at a real OPA server that pulls its bundle from a MinIO or S3 test container (a throwaway object store started by the test run). The bug it usually catches is a typo in the services URL, which otherwise turns up in production as an empty policy tree.

Test the bundle download from a network zone that matches production, not from your laptop over the office VPN. Firewall rules that block the bundle URL are one of the most common go-live surprises, and they show up as pods that start, pass a naive health check, and answer every query with undefined.

Alert when a bundle goes stale

OPA's /metrics endpoint can tell you when it last activated a bundle, but only if you ask for it. Set status.prometheus: true in the config; without it the endpoint still serves HTTP and evaluation metrics, the bundle gauges are simply absent, and the alert query comes back empty with no hint as to why. Then alert when last_success_bundle_activation is older than twice your maximum polling interval. That one alert catches stuck polling, expired registry credentials, and a signature that quietly stopped verifying, all of which otherwise look like a healthy pod happily serving last month's rules.

Try this

Run the whole loop once on your own machine. Build a bundle from a directory, list what ended up inside it, run the tests against the artifact, then serve it with opa run --server -b and ask it a real question over HTTP. The last curl throws the body away and prints only the status code from /health?bundles=true, because the status code is the part a readiness probe actually reads. One caveat about that last line: a bundle loaded from disk with -b has no bundle plugin behind it, so the check passes as soon as the server is up. It only starts earning its place once OPA is polling a configured bundle service.

mkdir -p policies
cat > policies/authz.rego <<'EOF'
package authz
default allow := false
allow if input.user.role == "admin"
EOF
cat > policies/authz_test.rego <<'EOF'
package authz
test_admin_allowed if {
  allow with input as {"user": {"role": "admin"}}
}
EOF
echo '{}' > policies/data.json
opa build -b policies/ -o bundle.tar.gz
tar tzf bundle.tar.gz
opa test -b bundle.tar.gz
opa run --server -b bundle.tar.gz &
sleep 1
curl -s localhost:8181/v1/data/authz/allow -H 'Content-Type: application/json' -d '{"input":{"user":{"role":"admin"}}}'
curl -s -o /dev/null -w '\nhealth: %{http_code}\n' 'localhost:8181/health?bundles=true'
kill %1 2>/dev/null || true

/.manifest
/authz.rego
/authz_test.rego
/data.json
PASS: 1/1
{"result": true}
health: 200

Takeaway

Production OPA stays current by polling, not by redeploying. CI builds and signs a tarball, OPA fetches it, verifies the signature, and swaps it in between requests. Give the bundle registry the same care you give the container registry: TLS, signing, a short publisher list, and backups of both the artifacts and the keys.

And pin one rule to the wall for whoever writes the client wrapper. HTTP 200 with {} means undefined, and undefined means deny. Any code that reads a missing result key as approval is a security bug waiting for the right input.

Related