Multi-cluster distribution

GitOps, bundles, and wave rollouts.

In short. Fifty clusters with hand-applied policies become fifty truths.

Fifty clusters with hand-applied policies become fifty truths. Multi-cluster distribution needs a control plane for policy: Git as the source of truth, progressive delivery waves (canary clusters → nonprod → prod), and health signals before promotion. OPA bundles, Gatekeeper configs via GitOps, or Kyverno policies synced by Fleet/Argo/Anthos Config Management are mechanisms — the operating model is the product.

Include breakers: if a policy version spikes denials or breaks a canary namespace, rollback the bundle the same way you rollback an app chart.

Wave rollouts

git tag policy-bundle-2026.07.24
# GitOps repo pin for canary:
echo 'bundle: ghcr.io/acme/policy:2026.07.24' > clusters/canary/policy-version.yaml
git add . && git commit -m 'pin policy bundle on canary'

[main …] pin policy bundle on canary

Data plane differences

Not every cluster shares namespaces or CRDs. Parameterize constraints per cluster class (PCI, sandbox, edge). Keep templates common; keep parameters local. Sync only the inventory you need — full-cluster sync into OPA can be expensive.

Going deeper

Cluster classes (sandbox, pci, edge) should be first-class in your GitOps layout. Parameters differ; templates do not. Resist per-cluster forks of Rego — that is how drift wins.

Include a kill switch: a known ConfigMap or annotation that platform on-call can flip to dry-run during a false-positive SEV. Log and ticket every use. A kill switch you pretend you do not need will be invented under fire as "delete the webhook."

Multi-cluster also means multi-region latency for webhook calls if you centralize oddly. Keep admission local to the cluster; distribute policy data, not the PEP itself.

Chaos-test webhook outages: what happens when Gatekeeper/Kyverno is down? Fail-closed protects security and can halt deploys; fail-open protects deploys and can halt security. Choose explicitly per cluster class and document it for IR.

Record a short promotion ADR template: bundle version, canary results, deny rates, abort criteria, and approvers. Future you will need that during the next SEV.

Align policy bundle versions with platform release trains. Shipping policies ad-hoc on Fridays recreates the same risk as shipping untested app charts on Fridays.

Smoke-test admission on a canary namespace that always hosts a known-good and known-bad workload after every bundle promote.

Try this

Sketch a three-wave pin file layout in a GitOps repo and a promotion checklist with denial-rate abort criteria.

printf '%s\n' 'waves: [canary, nonprod, prod]' 'abort_if_denies_per_min: 50' 'bundle_pin: 2026.07.24'

waves: [canary, nonprod, prod]
abort_if_denies_per_min: 50
bundle_pin: 2026.07.24

Takeaway

Multi-cluster policy is versioned distribution plus waves plus rollback — not SSH and hope. Common templates, local parameters, pinned bundles.

Next: measure whether policies earn their keep.

Related