>_SecOpsLog_
CoursesPolicy-as-code at scalePolicy lifecycle

Policy lifecycle

Versioning, rollout waves, exceptions with expiry.

In short. policy platforms focus on Policy lifecycle: Versioning, rollout waves, exceptions with expiry.

Advanced30 min · lesson 4 of 5

A city that wants a new speed limit does not switch on the cameras overnight. The rule gets drafted, debated, published, posted on warning signs for a month, and only then enforced with tickets — and years later, when the road changes, it gets repealed. A policy in infrastructure is the same kind of object: a rule a machine enforces, such as *every container image must come from our internal registry*. Writing the rule is the easy part. The dangerous part is everything around it — how it is reviewed, shipped to the engines that enforce it, switched on without breaking production, and eventually retired.

That surrounding process is the policy lifecycle: the managed path a rule takes from proposal to retirement — author, review, test, package, distribute, audit, enforce, monitor, retire. Because policy-as-code stores rules as files in Git and evaluates them with an engine such as OPA, the lifecycle is a software-delivery problem, and every discipline you already apply to application code — versioning, CI, staged rollout, telemetry — applies to policy too. Earlier lessons covered how an engine evaluates a decision and how to test Rego; this one covers what happens between git merge and a request being denied in production.

The failures a lifecycle prevents

Skipping the lifecycle fails in two opposite ways. Big-bang enforcement: a policy merges straight into deny mode. A new *require a team label* rule looks harmless in review, but 137 existing workloads lack the label, every redeploy is suddenly blocked, and the platform team spends the night hand-writing exemptions. Policy rot is the mirror image: rules sit in audit-forever mode, exemptions never expire, and nobody remembers why a rule exists — enforcement in name only. There is also a security dimension: the distribution channel is an attack path. Whoever can serve policy to your engines decides what is admitted to your clusters, so an unsigned bundle fetched over plain HTTP is roughly equivalent to handing out cluster-admin. Signing, versioning, and staged rollout are the mitigations — and they are lifecycle features, not engine features.

Packaging: bundles, revisions, signatures

OPA does not read Git. It consumes a bundle: a gzipped tarball containing .rego files, JSON data, and a .manifest that declares a *revision* (a version string you choose) and *roots* (the namespaces the bundle owns, which stops two bundles from silently overwriting each other's rules). CI builds the bundle, signs it with a private key, and pushes it to a bundle server — any static HTTP server or OCI registry works. Put both the semantic version and the Git SHA in the revision, because that string is what engines later report back, and it should answer *exactly which policy is running in cluster X right now* with a single field.

shell
# Build a signed bundle from the policies/ directory
opa build -b policies/ -o bundle.tar.gz \
  -r "v1.4.0+3f9c2d1" \
  --signing-key /keys/bundle-private.pem
 
# Verify what you just built
opa inspect bundle.tar.gz
# -> MANIFEST:
# -> +----------+----------------+
# -> |  FIELD   |     VALUE      |
# -> +----------+----------------+
# -> | Revision | v1.4.0+3f9c2d1 |
# -> | Roots    | kubernetes     |
# -> +----------+----------------+
# -> NAMESPACES:
# -> +---------------------------+-------------------------------+
# -> |         NAMESPACE         |             FILE              |
# -> +---------------------------+-------------------------------+
# -> | data.kubernetes.admission | /policies/admission/deny.rego |
# -> +---------------------------+-------------------------------+

Distribution is pull-based. Each engine polls the server on a jittered interval, verifies the signature against a public key, and hot-swaps the new bundle atomically — no restarts. Two feedback channels close the loop: the status API reports which revision each engine has activated, and decision logs stream every allow/deny along with the input that produced it. Decision logs are both your audit trail and the evidence you use to promote a policy to its next stage.

opa-config.yaml
services:
  bundle_registry:
    url: https://bundles.internal.example.com
    credentials:
      bearer:
        token_path: /var/run/secrets/opa/token  # never inline tokens
 
keys:
  bundle_signer:
    algorithm: RS256
    key: ${BUNDLE_PUBLIC_KEY}    # public key only; private stays in CI
 
bundles:
  admission:
    service: bundle_registry
    resource: bundles/admission/bundle.tar.gz
    polling:
      min_delay_seconds: 30      # jittered poll window
      max_delay_seconds: 120
    signing:
      keyid: bundle_signer       # reject unsigned or tampered bundles
 
status:
  service: bundle_registry       # report active revision upstream
 
decision_logs:
  service: bundle_registry
  reporting:
    min_delay_seconds: 5
    max_delay_seconds: 10
Policy promotion pipeline
1Author & review
Git PR, unit tests in CI
2Build & sign
opa build -r v1.4.0+sha
3Distribute
bundle server, engines poll
4Audit (dryrun)
drain the violation debt
5Enforce (deny)
gate: 0 violations + soak
Retirement runs the same pipeline in reverse: deny -> warn -> delete. The status API and decision logs feed evidence back into every gate.

Staged enforcement: dryrun, warn, deny

An enforcement action is what the engine does when a rule is violated. Gatekeeper defines three: dryrun records the violation and does nothing else; warn admits the request but returns a warning the client sees (kubectl prints it); deny rejects it. The iron rule of the lifecycle: every new policy enters production in dryrun. Gatekeeper's audit controller also rescans objects that already exist in the cluster (every 60 seconds by default), so dry-run shows your true violation debt — including workloads created long before the policy existed.

shell
# Stage 1: ship the constraint in dry-run
kubectl apply -f require-team-label.yaml   # spec.enforcementAction: dryrun
# -> k8srequiredlabels.constraints.gatekeeper.sh/require-team-label created
 
# The audit controller rescans existing objects (~60s); read the debt
kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{.status.totalViolations}'
# -> 137
 
# Stage 2: remediation has driven the count to zero; surface warnings
kubectl patch k8srequiredlabels require-team-label \
  --type merge -p '{"spec":{"enforcementAction":"warn"}}'
# -> k8srequiredlabels.constraints.gatekeeper.sh/require-team-label patched
 
# Stage 3: soak period passed, violations still zero -> enforce
kubectl patch k8srequiredlabels require-team-label \
  --type merge -p '{"spec":{"enforcementAction":"deny"}}'

Promotion between stages should be gated, and the gate should be automated: flip dryrun to warn only when every violation is either remediated or covered by a dated exemption; flip warn to deny only after totalViolations has held at zero through a soak period — a week of real deploys, not a quiet weekend. A CI job that reads the constraint status enforces this more reliably than a human who remembers to check.

Zero admission denials does not mean safe to enforce
Admission control only evaluates objects on create and update. Those 137 pre-existing violators keep running untouched, so after you flip to deny the cluster looks healthy — until a node drain or an eviction recreates their pods and admission rejects them, turning routine maintenance into an outage. Gate promotion on the audit controller's count of *existing* violations, never on the absence of new denials.

Retirement, exceptions, and what breaks at scale

The lifecycle does not end at deny. Exemptions accumulate: require every one to carry an owner and an expiry date in metadata, and fail CI when one is past due — otherwise exceptions quietly become permanent policy. Dead rules accumulate too: a rule that decision logs show has not fired in 90 days is a repeal candidate, and removal runs the rollout in reverse (deny to warn to delete) so nothing was depending on it silently. Know the trade-offs you have bought. Polling means minutes of lag between merge and enforcement — deliberate, but it surprises people expecting instant effect. The bundle server becomes a tier-0 dependency: engines keep the last good bundle when a fetch fails, but a freshly started node has nothing until its first successful pull. And one giant bundle serving hundreds of clusters costs activation time and memory on every engine — split bundles along team or domain boundaries with disjoint roots before that hurts.

The stages are universal; the machinery is not. Revisions and status reports are OPA's spelling, enforcementAction and totalViolations are Gatekeeper's, and Kyverno expresses the same ideas as validationFailureAction: Audit and PolicyException resources. The next lesson puts those two Kubernetes-native engines side by side — how each implements validation, mutation, and this lifecycle, and how to choose between them.

Quick check
01A require-team-label constraint has sat in warn for a week and Gatekeeper has denied zero admission requests the whole time. What actually tells you it is safe to flip to deny?
Incorrect — admission control only evaluates objects on create/update, so it never sees the pre-existing violators that keep running; zero denials says nothing about them.
Correct — the audit controller rescans existing objects; only a zero existing-violation count guards against a node drain or eviction recreating a violating pod and getting rejected at admission, turning maintenance into an outage.
Incorrect — bundles hot-swap atomically with no restarts, and restarting engines does nothing to reveal or fix pre-existing violators.
Incorrect — 90 days without firing is the threshold for repealing a rule, not for promoting to deny; the promotion soak is roughly a week of real deploys.
Kyverno vs GatekeeperEnforcement points

Related

What OPA is: decoupled policyOPA & RegoWhat IaC scanning findsCheckov & IaC scanningFrameworks demystifiedCompliance as codePolicy-as-code interview questionsInterview prep