Delivering policy securely
Version, test, and enforce guardrails.
A car plant tests the airbag before it bolts it into the door. Being behind schedule is not a reason to skip that step. Policy is the same class of part. The rules that decide what reaches production get reviewed, tested, signed, versioned and rolled out with the same care as the applications they guard. A guardrail nobody checked is worse than no guardrail at all, because everyone drives like it is holding.
Every pull request against your policy repository should run the same short list: opa fmt to check formatting, a linter, opa check --strict to compile, opa test with a coverage floor, a bundle build, and a signature check. CI (continuous integration, the automated jobs that run on every proposed change) blocks the merge if any of those go red. Conftest, Gatekeeper and any sidecar OPA (Open Policy Agent, the engine that evaluates your rules) then pull that one signed bundle. Two failures turn policy-as-code into decoration: policy nobody tested, where a broken rule quietly allows everything, and policy nobody enforces, where the answer comes back and the caller ignores it.
One pipeline stage, six checks
One stage, on every pull request: format check, lint, strict compile, unit tests against a coverage threshold, integration tests over Conftest fixtures, then the signed bundle build. Any failure blocks the merge. When the change lands on the main branch, the publish step pushes the bundle to your registry and bumps the tag that Gatekeeper and the sidecars are pointed at.
opa fmt --list policy/ && test -z "$(opa fmt --list policy/)"regal lint policy/opa check --strict policy/opa test policy/ --coverage --threshold 90
PASS: 24/24Coverage: 92.1%
opa build -b policy/ -o bundle.tar.gz --signing-key "$CI_SIGNING_KEY"conftest test test/fixtures/ --policy policy/
24 tests, 24 passed, 0 failures# bundle.tar.gz ready to upload
policy-ci:script:- opa fmt --list policy/ && test -z "$(opa fmt --list policy/)"- regal lint policy/- opa check --strict policy/- opa test policy/ --coverage --threshold 90- opa build -b policy/ -o bundle.tar.gz --signing-key key.pem- conftest test test/fixtures/ --policy policy/artifacts:paths: [bundle.tar.gz]
The same rules, three gates
Conftest runs in application CI and stops bad configuration before merge. Gatekeeper runs at admission and stops bad resources entering the cluster. The Decision API answers services while they are running. That overlap is deliberate. A publicly readable storage bucket gets caught in the JSON (JavaScript Object Notation, a plain-text data format) that terraform plan writes out, and caught a second time at admission if somebody pushes past CI with an override. Version the whole set together, so tag v2026.07.24 means the same rules in all three places.
Roll out in dryrun before you deny
A new speed camera usually photographs for a month before it starts mailing tickets. New Gatekeeper constraints work the same way. Start them in dryrun or warn, and the audit loop counts how many live objects already break the rule without blocking anyone. Fix those workloads. Then flip enforcementAction to deny. New Conftest deny rules deserve the same courtesy: warn for a sprint if teams need it, then promote. Write the promotion date down, because warn rules that nobody dated are still warning three years later.
kubectl patch constraint no-privileged --type merge \-p '{"spec":{"enforcementAction":"dryrun"}}'
constraint.constraints.gatekeeper.sh/no-privileged patched
kubectl get constraint no-privileged -o jsonpath='{.status.totalViolations}'
7
Prove the gate actually blocks
Fire drills exist because nobody trusts an alarm that has never gone off. Do the same to your gates. Commit a violating fixture on a throwaway branch and confirm CI fails. Run kubectl apply on a bad pod, first against the dryrun constraint, then against deny, and confirm the second one rejects it. Send the Decision API an input you know is unauthorized and confirm the calling service returns 403, not 200. Policy that merged without that check is paperwork.
git checkout -b test/bad-pod && cp fixtures/violating-pod.yaml k8s/git commit -am "test: expect conftest fail" && git push# CI pipeline status:
failed — conftest test: 1 failure# merge blocked as intended
Tags you can roll back to
Bundle tags are immutable. Nobody edits what v2.4.0 means after it ships. If a policy release starts rejecting admissions it should never have touched, you point the sidecars and the Gatekeeper sync sources back at the previous tag. It is the same reflex you already have for a bad application deploy. Running opa test -b against every tag before you promote it keeps that reflex rarely needed.
Where the signing key lives
Signing keys belong in the CI secrets manager, never in the policy repository. Rotate them on a schedule. OPA's trust configuration lists the key IDs it will accept, so a rotation is a config change rather than a rebuild of everything downstream. Keep the credential that pulls a bundle separate from the credential that publishes one.
Put a CODEOWNERS file on the Rego and data paths so the right reviewers are pulled into every change automatically. Record the bundle's semantic version tag (v2.4.0) in each environment's manifest. Rolling back is then a manifest revert through the normal review path, not somebody hand-editing a live cluster at 2am.
Agree on the shape of input
Rego reads whatever fields your services actually send. Rename one, and the rule that referenced the old name goes undefined instead of failing loudly, which usually means it fails open. Validate your input fixtures against a JSON Schema in CI, so a rename breaks the pipeline rather than the guardrail.
git tag v2.4.0 && opa build -b policy/ -o bundles/v2.4.0.tar.gz
# immutable release artifact
Alarms rot quietly. Someone bypasses CI with an admin token, a warn rule never gets promoted, a constraint is deleted during an incident and nobody puts it back. Schedule deliberate "red team" commits in a sandbox pipeline that break policy on purpose, and check that the gates still fire. Monthly, automated, with the result written down.
conftest test --policy policy/ test/fixtures/violating/ ; echo exit:$?
1 failureexit:1
From merge to production tag
A normal flow looks like this. The pull request runs opa test and conftest against the changed policies. Merge builds the signed bundle. Publish sends it to the staging registry. Staging Gatekeeper's syncConfig pulls the staging tag, and you let it soak. Then you promote the exact same digest to the production tag. Never rebuild a production bundle from a different commit than the one staging tested. The signature would be fresh, the content would be untested, and you would have proved nothing.
Keep the last few bundle tags in the registry so there is somewhere to go back to. The runbook should name the command: patch the syncConfig or the OPA config resource to the previous tag, then check /health?bundles=true inside one polling interval to confirm the old bundle really activated. Rehearse it quarterly, while nothing is on fire.
Scan the OPA binary image on the same pipeline as your application images, and produce an SBOM (software bill of materials, an inventory of everything inside the image) for it. A CVE (Common Vulnerabilities and Exposures entry, a publicly catalogued flaw) in OPA is a platform CVE, because OPA sits in the admission path for every namespace you have.
Line policy versions up with how your organization handles change. A breaking Rego change is a major bump. A new deny rule is a minor. Data-only edits are a patch. Consumers then pin on purpose, and they know roughly what a bump will cost them before they take it.
Keep policy in its own repository, away from the applications. Application repos pin a policy version; the policy repo never imports application code. The blast radius stays obvious, and nobody slips a rule change through inside a feature branch.
Auditors for SOC 2 (a security audit of a service provider) or FedRAMP (the US federal cloud security program) do not take your word that policy is enforced. They ask for evidence. Keep CI logs, bundle signatures and Gatekeeper audit exports, with retention that matches whatever compliance window applies to you. Collect them automatically, because nobody reconstructs a year of evidence by hand.
Publish policy releases into the platform changelog. Application teams already watch for Kubernetes minor upgrades; give them the same signal for a policy major bump. A deny that arrives with two weeks of notice is a planning item. The same deny arriving unannounced on a Friday afternoon is an incident.
Treat OPA's own configuration as immutable infrastructure. The ConfigMap that points at your bundle URL belongs to GitOps (keeping cluster state in Git and letting a controller sync it), not to a kubectl edit typed during an outage. Break-glass access exists for genuine emergencies, and every use of it should be visible afterwards.
After a bundle promote, send the Decision API one input you know should be refused and check that you get false back. The /health endpoint can go green while the new bundle failed to activate, so health alone never proves the rules actually changed.
Split ownership so one queue does not hold up every change. Security owns the deny rules. The platform team owns the bundle pipeline. Product owns the data allowlists. Write that split into the policy repo README as a short table saying who decides, who reviews, and who only needs telling. Skip it and every pull request waits on the same overloaded person.
A release bot on the policy repo can write a changelog entry for each merged deny rule. That changelog is what application teams read before they bump the pinned policy version in their own CI, so make it say plainly what started being blocked and from which version.
What app teams need from you
Publish an integration guide and keep it short: the JSON shape of input, the query path to ask for, and a fail-closed snippet in Go, Java and Python. Your team owns the Rego. Their team owns three lines of glue that call it and treat any error as a deny.
Protect the policy repo's default branch with a two-person rule, the same as your production infrastructure. One engineer should never be able to self-approve a change that quietly removes a deny rule.
A healthy policy job fails closed. If opa test fails, if coverage drops below the floor, if Conftest fails, or if the bundle build fails, nothing gets published. Then add the step most teams skip: a post-deploy probe that replays the fixture from your last real incident against Gatekeeper or the sidecar and expects a rejection. That probe is how you learn the guardrail you built six months ago is still standing.
Record the bundle digest in the change ticket, not only the tag. When somebody asks what was running at 03:14 last Tuesday, the digest answers exactly. Rollback then reads "point OPA at yesterday's digest", never "hotfix the Rego on a live volume". Do that rollback in staging once a quarter so the runbook stays honest.
Publishing to production deserves dual control: two humans, or a pipeline identity that no individual can borrow. A developer laptop able to sign a production bundle is a supply-chain hole with a keyboard attached. Whoever takes that laptop gets to write the rules everything else trusts.
Write the input contract on one page: required fields, allowed enum values, and a working example payload. App teams integrate faster from a real example than from prose, and you head off the most common fail-open cause, a missing key nobody knew was mandatory turning a rule undefined.
Try this
Run the whole loop on your own machine: compile, test with coverage, Conftest over the manifests, build a signed bundle. Then finish it properly by asking the cluster to admit a known-bad privileged pod and watching it get refused. Evaluation without enforcement is half a pipeline.
opa check policy/opa test policy/ -v --coverage --threshold 85conftest test manifests/ --policy policy/opa build -b policy/ --signing-key ci.pem -o bundle.tar.gz# example gate after deploy:kubectl run probe --image=busybox --restart=Never --rm -i -- privileged=true 2>&1 | head -n 5
PASS: 20/20Coverage: 92.1%FAIL - manifests/bad.yaml - main - privileged not allowed# bundle.tar.gz published to registryError from server: admission webhook "validation.gatekeeper.sh" denied the request
Takeaway
Ship policy the way you ship software. Test it, measure coverage, sign the bundle, publish it, then check that CI, admission and the Decision API each still refuse a violation you can name out loud. Dryrun first, deny second, and keep rollback down to pointing at yesterday's tag.
Hand app teams the input schema and a written exception path with an expiry date on every exception, so policy pull requests do not turn into a ticket queue nobody staffs.