Policy as code & rollout

Version, review, audit→enforce the call graph.

Every office building keeps a key register. It is a book that says who holds a key to which door, when it was issued, and who signed for it. The locks work fine without the book. What you lose without it is the answer to one question, the one asked after something has gone wrong: who could have opened that door? A service mesh (the layer that runs a small proxy beside every workload and carries all the traffic between them) collects hundreds of allow rules across dozens of services, and each rule is worth exactly as much as the record of how it got there. If an on-call engineer can widen access with kubectl edit at 3am and nobody notices for a quarter, you do not have an authorization system. You have a suggestion.

The register lives in git, one file per door

In Istio, permission to make a call is written down as an AuthorizationPolicy: a Kubernetes object saying which callers may reach which workload, on which paths and methods. Treat every one of them as source code. It sits in git beside the workload it protects, changes land through pull requests, and a GitOps controller (Argo CD or Flux, a controller that continuously drags the cluster back to whatever the repo says) applies it. Git becomes the register, and the commit is the signature in the book.

Write one policy per workload rather than one sprawling file per namespace. Ownership then falls out for free. The orders team owns apps/orders/authorizationpolicy.yaml, and a careless edit to one service cannot quietly widen another service's access. git log answers who allowed the frontend to call orders, and when. A bad rule becomes a git revert and a sync instead of a live-cluster scramble with the pager going off.

The price is real discipline. No emergency kubectl edit. Anything applied outside the repo is drift, and Argo CD will either stamp on it (if self-heal is turned on) or leave it running and mark the app OutOfSync. The second case is worse. The locks and the register now disagree, and the register is the thing your auditors read.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-allow-frontend
  namespace: orders
spec:
  selector:
    matchLabels:
      app: orders                  # binds to the orders workload only
  action: ALLOW
  rules:
  - from:
    - source:
        principals:                # peer identity, read from the caller's certificate
        - cluster.local/ns/frontend/sa/frontend
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/orders", "/api/orders/*"]

What the rule proves, and what it does not

The principals list carries all the weight here. A principal is the caller's cryptographic identity, lifted out of the client certificate the receiving proxy checked during the TLS handshake (Transport Layer Security, the same negotiation your browser runs with a bank, except here both ends present a certificate rather than only the server, which is what mTLS, mutual TLS, means). Call it the photo ID the caller had to show at the door. That certificate is an SVID (SPIFFE Verifiable Identity Document), and the name printed on it is a SPIFFE ID (Secure Production Identity Framework For Everyone), a URI such as spiffe://cluster.local/ns/frontend/sa/frontend. Nobody can present that ID without the matching private key, which the proxy received over a Unix socket on the node (a socket that is a file path, so it never leaves the machine) and never wrote to disk. A header can be typed by anyone. This cannot.

Two details trip almost everyone. The first is spelling. In principals you drop the spiffe:// scheme and write <trust-domain>/ns/<namespace>/sa/<service-account>. The certificate keeps the scheme and so do the Prometheus labels you will read later. Same identity, two spellings, and the policy field wants the short one.

The second is subtler, and it is usually taught backwards. Inside an authorization policy, cluster.local is not a literal string. Istio treats it as a pointer to whatever your mesh's current trust domain is (the name of the authority that signs identities, the equivalent of which office issued the badge), plus any aliases configured at install time. So a rule saying cluster.local/ns/frontend/sa/frontend keeps matching on a mesh whose real trust domain is prod.example.com, and it survives a trust domain rename without an edit. Write the literal prod.example.com/ns/frontend/sa/frontend instead and you get an exact string comparison with none of that forgiveness: rename the trust domain later and every literal rule silently stops matching. Use cluster.local for callers inside your own mesh. Save literal trust domains for the day you have to name somebody else's.

You still need to know your real trust domain, because SPIRE entries, federation bundles and every metric label use the literal value even when your rules do not. Read it off the certificate the proxy is actually holding rather than guessing.

$ istioctl proxy-config secret deploy/frontend -n frontend -o json \
    | jq -r '.dynamicActiveSecrets[]
             | select(.name == "default")
             | .secret.tlsCertificate.certificateChain.inlineBytes' \
    | base64 -d | openssl x509 -noout -subject -ext subjectAltName

subject=
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/frontend/sa/frontend

Note the empty subject=. Mesh certificates carry no useful common name; the identity lives only in the URI SAN (Subject Alternative Name, the certificate extension holding names other than the subject). If you run SPIRE with a trust domain of prod.example.com, set Istio's meshConfig.trustDomain to the same value, and this output, the SPIRE entries and every metric label will read prod.example.com while your policies go on saying cluster.local. One wrong character in a policy fails closed: the rule matches nothing, requests get denied, and the YAML still looks perfect on screen.

Be precise about what that certificate establishes, because operators routinely read too much into it. It proves the peer that terminated the TLS connection holds a private key for that SPIFFE ID, issued by your trust domain's CA (certificate authority, the service that signs identities). That is the whole claim. It does not prove which container in the pod made the call, because one sidecar (the proxy container sitting beside your application container, handling its network traffic) serves every process in that pod. It says nothing about the human or the customer behind the request; for that you want requestPrincipals, which matches <issuer>/<subject> out of a validated JWT (JSON Web Token, a signed bundle of claims about an end user) and is a separate check entirely. And it does not prove the code running under frontend's service account is the code you reviewed. Anyone who can create a pod in the frontend namespace with the frontend service account has that identity handed to them, legitimately, by the mesh. Your Kubernetes RBAC (role-based access control, the rules deciding which accounts may do what in the cluster) on pod creation is part of your authorization boundary whether you think of it that way or not.

source.namespaces and source.serviceAccounts come out of the same verified certificate, so they are not forgeable either. They are blunter. namespaces: ["frontend"] grants every service account in that namespace, including the debug pod somebody left running since March. Prefer principals, and make a broad scope something a human has to approve on purpose.

The identity registry is policy too

If SPIRE mints your identities, there is a second register sitting upstream of the first. The AuthorizationPolicy is the list of who may open the door. The registration entry is the badge office deciding who gets issued the badge in the first place. An entry with a sloppy selector quietly undoes every policy that names that identity, because more workloads can now wear it. Delete the sa:frontend line below and every pod in the frontend namespace can obtain the frontend SVID and call orders, with a valid certificate and a clean audit trail behind it.

Entries are created with spire-server entry create. Typing them by hand against a running server is the same 3am problem as kubectl edit, so keep them in a file and feed it with -data, which reads JSON and refuses to be combined with the per-field flags. The parent_id below is a node alias: an entry the SPIRE agents themselves match, which every workload entry then hangs off.

{
  "entries": [
    {
      "spiffe_id": "spiffe://cluster.local/ns/frontend/sa/frontend",
      "parent_id": "spiffe://cluster.local/ns/spire/sa/spire-agent",
      "selectors": [
        { "type": "k8s", "value": "ns:frontend" },
        { "type": "k8s", "value": "sa:frontend" }
      ],
      "x509_svid_ttl": 3600
    }
  ]
}

$ spire-server entry create -data spire/entries.json

Entry ID                : 4c9e6f0b-2a71-4c8d-9a3e-71f0c2d5b8a4
SPIFFE ID               : spiffe://cluster.local/ns/frontend/sa/frontend
Parent ID               : spiffe://cluster.local/ns/spire/sa/spire-agent
Revision                : 0
X509-SVID TTL           : 3600
JWT-SVID TTL            : default
Selector                : k8s:ns:frontend
Selector                : k8s:sa:frontend

That x509_svid_ttl of 3600 is the certificate's lifetime in seconds, one hour, after which the agent hands the workload a fresh one. Re-running the command is where pipelines come unstuck: entry create fails on a duplicate rather than doing nothing, and spire-server entry update -data needs each entry's server-generated ID inside the file, which you never wrote and do not want to maintain by hand. Most teams run the SPIRE Controller Manager instead. It watches ClusterSPIFFEID objects (a CRD, a custom resource definition, meaning an object type an add-on teaches Kubernetes about) and reconciles them into entries the way Argo CD reconciles your policies. Either way the source of truth is a reviewed file, and both registers ride through the same pull request: who may hold an identity, and what that identity may call.

CI has to catch what the cluster will not

A mistyped principal is not a syntax error. It is a well-formed string that matches nothing. The API server accepts it, Argo CD goes green and Synced, and traffic starts dying with RBAC: access denied. Nothing anywhere tells you somebody wrote sa/fronted. That silent-deny failure mode is why authorization policy earns a stricter pipeline (CI, continuous integration: the checks that run automatically on every pull request) than the rest of your YAML.

Three checks cover most of it. istioctl analyze catches schema and cross-reference problems; --use-kube=false makes it read files only, which is all a pull-request runner usually has, and --failure-threshold Warning turns warnings into a failed build instead of scrollback. kubectl apply --dry-run=server pushes the object through a real API server and its validating webhooks without storing it, so that step does need credentials to a cluster, normally staging. Conftest, a test runner for OPA (Open Policy Agent) and its Rego policy language, enforces the house rules no schema can express.

package main
 
import rego.v1
 
# a principal is <trust-domain>/ns/<ns>/sa/<sa>, never the full spiffe:// URI
deny contains msg if {
	input.kind == "AuthorizationPolicy"
	some r in input.spec.rules
	some src in r.from
	some p in src.source.principals
	not regex.match(`^[a-z0-9.-]+/ns/[^/]+/sa/[^/]+$`, p)
	msg := sprintf("principal %q is not <trust-domain>/ns/<ns>/sa/<sa>", [p])
}
 
# a policy with no selector in the root namespace applies to the entire mesh
deny contains msg if {
	input.kind == "AuthorizationPolicy"
	input.metadata.namespace == "istio-system"
	not input.spec.selector
	msg := sprintf("%s has no selector in the root namespace: mesh-wide blast radius",
	               [input.metadata.name])
}

That second rule only fires when the namespace is written in the file, so pin metadata.namespace in every policy manifest instead of letting a deploy tool inject it. Here is the pipeline running against a pull request where somebody pasted the identity straight out of the certificate, scheme and all.

$ istioctl analyze --use-kube=false --failure-threshold Warning apps/orders/
$ kubectl apply --dry-run=server -f apps/orders/authorizationpolicy.yaml
$ conftest test --policy policy/ apps/orders/

✔ No validation issues found when analyzing apps/orders/.
 
authorizationpolicy.security.istio.io/orders-allow-frontend configured (server dry run)
 
FAIL - apps/orders/authorizationpolicy.yaml - main - principal "spiffe://cluster.local/ns/frontend/sa/frontend" is not <trust-domain>/ns/<ns>/sa/<sa>
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

Two of the three checks were perfectly happy, which is the point. Rego proves shape, never truth. cluster.local/ns/frontend/sa/fronted sails straight through that regex because it is a well-formed path to a service account that does not exist. A spell-checker can confirm a phone number has ten digits. It cannot tell you anybody answers. For truth you need the cluster, and the check is four lines of shell: pull every principal out of the repo and confirm each one names a real ServiceAccount. Run it against staging on every pull request.

$ grep -rhoE '[a-z0-9.-]+/ns/[a-z0-9-]+/sa/[a-z0-9-]+' apps/ | sort -u | while read -r p; do
    ns=${p#*/ns/}; ns=${ns%%/*}
    sa=${p##*/sa/}
    kubectl get serviceaccount "$sa" -n "$ns" >/dev/null 2>&1 \
      || echo "MISSING ServiceAccount for principal: $p"
  done

MISSING ServiceAccount for principal: cluster.local/ns/frontend/sa/fronted

Read the call graph off the wire, not off memory

A dry run tells you what a written policy would do. It cannot tell you which policy to write. Get that from the mesh itself. Nobody builds a guest list from memory; you read the door log first. Put mTLS in PERMISSIVE mode (the proxy accepts both mutual TLS and plaintext), leave authorization off, let real traffic run long enough to include the infrequent jobs, then ask Prometheus (the time-series database that scrapes metrics off every proxy) who actually called whom. Istio's istio_requests_total counter carries source_principal and destination_principal labels taken straight from the certificates, plus connection_security_policy, which reads mutual_tls when the connection was authenticated and none when it was not.

$ kubectl -n istio-system port-forward svc/prometheus 9090:9090 >/dev/null &
$ curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode 'query=sum by (source_principal, source_workload, connection_security_policy) (
      increase(istio_requests_total{reporter="destination",
                                    destination_workload="orders",
                                    destination_workload_namespace="orders"}[7d]))' \
    | jq -r '.data.result[] | [ .metric.connection_security_policy,
                                (.metric.source_principal // "<none>"),
                                .metric.source_workload,
                                (.value[1] | tonumber | floor) ] | @tsv'

mutual_tls	spiffe://cluster.local/ns/frontend/sa/frontend	frontend	503088
mutual_tls	spiffe://cluster.local/ns/checkout/sa/checkout	checkout	90114
mutual_tls	spiffe://cluster.local/ns/billing/sa/reconciler	reconciler	712
none	<none>	unknown	1843

Four callers in a week, and the last two rows are the ones worth your afternoon. The reconciler at 712 requests is a batch job that a two-hour observation window would have missed completely, and missing it means breaking month-end billing on the day you enforce. The bottom row is worse. It has no principal, because a plaintext connection presents no certificate, so no principals rule can ever match those 1843 requests. Its source_workload reads unknown too, which tells you the caller has no sidecar at all; a caller that has a sidecar but talks plaintext would still name itself there, because the proxies swap workload metadata in a header. For a genuine unknown, go to the access log and start from the remote address. Get that caller into the mesh before you enforce anything, or its calls die the moment default-deny lands.

Metrics give you edges, not paths. Istio deliberately keeps the request path off istio_requests_total, because the label cardinality would flatten Prometheus. If your rules restrict methods and paths, take those from the access log instead: add %DOWNSTREAM_PEER_URI_SAN% to meshConfig.accessLogFormat and every line carries the caller's SPIFFE ID next to the method and path it asked for.

Rehearse the denial before you cause it

You now have a candidate allow-list. Run the fire drill before you change the locks. Istio's dry-run annotation, istio.io/dry-run: "true", compiles the policy into the proxy and evaluates it against live traffic, then throws the verdict away. The request goes through either way. Envoy counts what would have happened in a parallel set of shadow counters, so you can measure how many requests a policy would kill before it kills any of them.

One thing the annotation does not do, and this surprises people. A normal ALLOW policy flips its target workload into allow-list mode, where anything unmatched is denied. A dry-run ALLOW policy does not flip that switch. Enforcement stays exactly as it was, which is what makes the rehearsal safe to ship to production on a Tuesday afternoon.

# in the repo this is one line under metadata.annotations; shown here as a command
$ kubectl annotate authorizationpolicy orders-allow-frontend -n orders \
    istio.io/dry-run=true --overwrite
 
# let real traffic run for a full business cycle, then read the shadow counters
$ kubectl exec deploy/orders -n orders -c istio-proxy -- \
    pilot-agent request GET stats | grep 'rbac\.istio_dry_run'

authorizationpolicy.security.istio.io/orders-allow-frontend annotated
 
http.inbound_0.0.0.0_8080.rbac.istio_dry_run_allow_shadow_allowed: 593202
http.inbound_0.0.0.0_8080.rbac.istio_dry_run_allow_shadow_denied: 712

712, the same figure as the reconciler. Istio also tags that counter for Prometheus as envoy_http_inbound_0_0_0_0_8080_rbac{authz_dry_run_action="allow",authz_dry_run_result="denied"}, which is the version you alert on, because raw counters reset whenever a proxy restarts. Counters tell you how many. They never tell you who. For that, turn the proxy's RBAC logger up for a few minutes.

$ istioctl proxy-config log deploy/orders.orders --level "rbac:debug" | grep rbac
$ kubectl logs deploy/orders -n orders -c istio-proxy | grep "shadow denied" | tail -3

rbac: debug
 
2026-07-14T02:00:11.482913Z	debug	envoy rbac	shadow denied, matched policy none
2026-07-14T02:00:11.611204Z	debug	envoy rbac	shadow denied, matched policy none
2026-07-14T02:00:12.044771Z	debug	envoy rbac	shadow denied, matched policy none

matched policy none is the ALLOW engine reporting that nothing in your allow-list covered the request. Look at what the line is missing: the caller. Correlate those 02:00 timestamps against the access log (which carries the peer URI SAN if you added it) or against the Prometheus breakdown above. Add the reconciler's principal, redeploy, and keep watching until shadow_denied sits flat at zero across a period long enough to include your slowest-moving caller.

Treat the rehearsal as evidence, not proof. The annotation is still an experimental Istio feature with sharp edges. It supports only ALLOW and DENY actions, so a CUSTOM policy that hands the decision to an external authorizer never gets rehearsed at all. ALLOW and DENY produce two independent shadow results, because the proxy enforces those stages separately, so one request can be shadow-allowed by one and shadow-denied by the other and you have to read both. Istio states plainly that the dry-run log, metric and tracing output are troubleshooting aids and not an API, so they can change between releases: use them to make a decision, never as a hard CI gate. And the whole exercise only ever measures traffic that happened while you were watching.

Flip default-deny in waves

Default-deny in Istio is a policy with an empty spec. No action means ALLOW, and no rules means nothing can ever match it, so every request that no other ALLOW policy covers gets refused. Scope it to one application namespace and let a sync wave put it behind your allow-list, so there is never a moment where the deny exists and the allows do not.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-nothing
  namespace: orders                        # one namespace, never istio-system
  annotations:
    argocd.argoproj.io/sync-wave: "1"      # lands after the ALLOW policies in wave 0
spec: {}

Across all the policies on one workload, the proxy evaluates CUSTOM first, then DENY, then ALLOW. A bouncer checks the banned list before the guest list. A narrow DENY (block /api/orders/refund from everyone outside the platform namespace) therefore beats a broad ALLOW, which lets you layer a hard prohibition over an allow-list without rewriting it. None of this counts for much until PeerAuthentication is set to STRICT, meaning the proxy refuses plaintext outright. principals are read from a client certificate, a plaintext connection has none, so in PERMISSIVE mode such a caller matches no ALLOW rule, falls through to default-deny and collects a 403 whose message says nothing whatsoever about TLS. Chasing that 403 as an authorization bug is a well-worn afternoon.

# wave 1: drop the rehearsal annotation, add allow-nothing, let Argo CD sync the commit
$ kubectl annotate authorizationpolicy orders-allow-frontend -n orders istio.io/dry-run-
 
# prove enforcement is live: call from an identity that is NOT on the allow-list
$ kubectl exec deploy/curl -n test -c curl -- \
    curl -s -o /dev/null -w '%{http_code}\n' http://orders.orders.svc.cluster.local:8080/api/orders
 
$ kubectl exec deploy/orders -n orders -c istio-proxy -- \
    pilot-agent request GET stats | grep -E 'rbac\.(allowed|denied)'

authorizationpolicy.security.istio.io/orders-allow-frontend annotated
 
403
 
http.inbound_0.0.0.0_8080.rbac.allowed: 41822
http.inbound_0.0.0.0_8080.rbac.denied: 1

That 403, with a response body of RBAC: access denied, plus the non-shadow rbac.denied counter moving by exactly one, is your proof. A green Argo CD tile proves the object exists in the cluster. It does not prove any proxy is refusing anything. Run the unauthorized-caller probe as a scheduled job rather than once by hand, so the day somebody deletes the allow-nothing policy you hear it from a test instead of from an incident.

Do one namespace per wave, each as its own commit. If denials spike, the rollback is a git revert and a sync that finishes in seconds and needs nobody holding cluster credentials. Waves also cap the damage: a bad rule stops at one namespace instead of taking the mesh with it.

Write the mesh's real trust domain into the README beside those policies, one line, because the certificates, the SPIRE entries and every metric label use the literal value even while your rules say cluster.local. The next lesson, Trust-domain federation, puts a caller signed by somebody else's CA on the other end of the connection. At that point the pointer stops covering the caller you care about, and the trust domain in your rules becomes a string your own mesh does not control.

Try this

Run spire-server entry create -data spire/entries.json on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: allow-nothing in the root namespace takes down the whole mesh. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related