Identity-based authz

AuthorizationPolicy, default-deny, least privilege.

A hotel key card does two jobs that are easy to blur together. The reader at the lobby door checks that the card is genuine and learns whose card it is. The lock on room 412 asks a completely different question: is this particular card on the list for this particular door? Mutual TLS is the lobby reader. It proves, cryptographically, who is calling on every single connection. It has no opinion at all about which rooms that caller may open. Identity-based authorization is the room lock, and it is the control that decides how far an intruder gets after they pick up one card.

What mTLS Proves, and What It Refuses To

STRICT mTLS (mutual Transport Layer Security, where both ends of a connection present a certificate and each verifies the other's) gives you an encrypted, mutually authenticated channel between every pair of sidecars in the mesh. A sidecar here is the Envoy proxy that Istio runs in the same pod as your container, quietly intercepting the pod's traffic on the way in and out. Leave authorization at its default on top of that and every workload can still reach every other workload. You have encrypted the lateral traffic, which helps an attacker hide it about as much as it protects you.

Encryption stops someone tapping the wire. It does nothing to contain a compromised pod, because the attacker's traffic is authenticated too. It leaves a real workload carrying a real certificate that the mesh CA (certificate authority, the service that signs workload certificates, normally istiod, Istio's control-plane process) issued that morning. Those certificates last about a day by default and the sidecar renews them without anybody noticing. Pop your reports pod through a vulnerable dependency and its sidecar will open a perfectly valid mTLS connection to the payments API, and the handshake will succeed on both sides.

Authorization is the decision mTLS deliberately does not make. In Istio it lives inside the destination workload's Envoy sidecar, in an RBAC (role-based access control) filter that runs before the request ever reaches your application process. The filter reads the caller's verified identity out of the client certificate, looks at the shape of the request (method, path, headers, port), and answers allow or deny. Your code is never consulted, and never has to be trusted to get it right.

A Principal Is a SPIFFE ID With the Scheme Chopped Off

An Istio AuthorizationPolicy matches callers on the principal: the SPIFFE ID (Secure Production Identity Framework For Everyone identifier, a URI, or Uniform Resource Identifier, that names exactly one workload) lifted out of the peer certificate with the spiffe:// scheme removed. A certificate carrying the SAN (Subject Alternative Name, the certificate extension listing the names a certificate is valid for) spiffe://cluster.local/ns/prod/sa/frontend is matched by principals: ["cluster.local/ns/prod/sa/frontend"]. Under the hood Istio glues the scheme back on and compares the whole string exactly, so there is no fuzziness and no partial credit.

Read the shape of that string and you learn how coarse your identities really are. cluster.local is the trust domain, prod is the namespace, and frontend is the pod's Kubernetes ServiceAccount. Identity stops there. It is not per pod and not per Deployment. Two Deployments sharing one ServiceAccount are the same caller as far as every policy in the mesh is concerned. Give each workload its own ServiceAccount before you write a single rule, or your careful least-privilege grants will quietly cover services you never meant to include.

Because the rule names an identity, it survives reality. Reschedule the pod onto another node with another IP address and the policy still describes the same thing. IP allowlists and NetworkPolicies (the Kubernetes object that filters pod traffic by address and label) rot for the mirror-image reason: they name an address, and addresses churn on every deploy.

That principal string is not something to type from memory. Get one character wrong and the rule matches nothing, and under default-deny "matches nothing" means "every request returns 403" with no error anywhere telling you why. Read the identity off the live certificate before you write policy against it.

# Pull the frontend workload's live certificate out of its Envoy and read the URI SAN
istioctl proxy-config secret deploy/frontend -n prod -o json \
  | jq -r '.dynamicActiveSecrets[] | select(.name=="default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -subject -ext subjectAltName

subject=
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/prod/sa/frontend

The Subject line is empty on purpose. SPIFFE puts the identity in the URI SAN and nowhere else, so reading the certificate's Common Name would tell you nothing at all. Strip spiffe:// off that URI and you have the exact string your policy needs.

If you have swapped istiod's built-in CA for SPIRE (the SPIFFE Runtime Environment, an open-source issuer that attests a workload before it hands over a certificate), the string comes out of a registration entry you wrote by hand, and SPIRE will happily mint any path you ask it for. Istio's integration expects one shape: spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>. Register something like spiffe://example.org/team/frontend instead and a literal principals: entry will still match it, which is exactly what fools people into thinking the shape is optional. What breaks is everything Istio works out by reading the ID. The namespaces and notNamespaces fields compile down to a regular expression over the path, roughly .*/ns/prod/.*, so they silently stop matching. Telemetry loses its namespace attribution. Stay on the documented shape.

kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/ns/prod/sa/frontend \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s:ns:prod \
    -selector k8s:sa:frontend \
    -selector k8s:pod-label:spiffe.io/spire-managed-identity:true \
    -socketPath /run/spire/sockets/server.sock

Entry ID         : 1f9c4d8e-6b21-4a55-9f0c-3c7d2b8a5e10
SPIFFE ID        : spiffe://example.org/ns/prod/sa/frontend
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:ns:prod
Selector         : k8s:pod-label:spiffe.io/spire-managed-identity:true
Selector         : k8s:sa:frontend

Those -selector flags are the attestation: SPIRE will only hand this identity to a pod that genuinely sits in namespace prod, runs as ServiceAccount frontend, and carries the label. The certificate SPIRE returns is an X.509-SVID (SPIFFE Verifiable Identity Document, which is a normal X.509 certificate whose only meaningful contents are that URI SAN and a short lifetime).

With SPIRE as the issuer the trust domain changes too, so the policy principal becomes example.org/ns/prod/sa/frontend. Set meshConfig.trustDomain to example.org so both halves agree. Skip that step and Istio notices: it logs Trust domain example.org from principal ... does not match the current trust domain or its aliases and falls back to comparing your string literally. The rule can still work by accident, and the next person to add a trust domain alias will discover that it never really did.

Close the Namespace, Then Hand Back One Key at a Time

The safe way to run an office building is not "lock the doors people complain about". It is "every door is locked, and each badge is granted the rooms its holder actually needs". Authorization works the same way. You close the whole namespace with one policy, then re-open exactly the flows your services really use.

# 1) Close prod. An ALLOW policy that carries no `rules` can never match,
#    and with no `selector` it covers every workload in the namespace.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: prod
spec: {}
---
# 2) Hand back exactly one key: frontend may POST /checkout on api. Nothing else.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-checkout
  namespace: prod
spec:
  selector:
    matchLabels:
      app: api                  # binds only to pods labelled app=api
  action: ALLOW                 # ALLOW is the default, spelled out for reviewers
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/frontend"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/checkout"]

Two things are doing the work. spec: {} is an ALLOW policy with no rules, so it can never match, and every request to every pod in prod now falls through to a deny. The second policy grants one flow back. Everything else in the namespace stays unreachable until somebody writes it down and somebody else reviews it.

Scope is the part people get wrong, and it is the whole difference between a namespace that is closed and one that only looks closed. selector binds a policy to sidecar-injected pods by label, inside the policy's own namespace. targetRefs is the alternative binding, used to attach a policy to a Gateway or a Service, and it is how waypoint proxies get their rules in ambient mode (Istio's sidecar-less option, where a shared proxy handles a namespace or a service instead of one proxy per pod). You may set selector or targetRefs, never both. One escape hatch is worth memorising: a policy with no selector at all, sitting in the mesh root namespace (istio-system unless you changed it), reaches workloads in every namespace, which is how you close an entire mesh with a single object.

That matters because Istio's closed default is per workload, not per namespace. Scope your deny-all with selector: {matchLabels: {app: api}} and you have secured exactly one service. A billing Deployment that ships next Tuesday, with no policy selecting it, still accepts calls from anything in the mesh, encrypted and completely unauthorized, and nothing on your dashboards will look wrong. The selector-less version above is what makes workloads nobody has thought of yet closed on arrival. Apply it first, grant flows back second.

The Order the Sidecar Decides In

Every policy that selects the destination workload is evaluated in a fixed order: CUSTOM first (these hand the decision to an external authorizer you configure yourself, an outsourced doorman), then DENY, then ALLOW. A DENY match ends the conversation, and no later ALLOW can undo it. That property is what makes a DENY policy a usable emergency brake when you need to cut one caller off in a hurry. A fourth action exists, AUDIT, which records that a request matched and never changes the outcome.

The part that catches teams is the baseline underneath all of it. If no ALLOW policy selects a workload at all, the request is allowed. Default-deny is not a setting you turn on anywhere in the mesh config. It falls out of having at least one in-scope ALLOW policy, and the zero-rule policy above is the idiom that produces it on purpose rather than by luck.

None of that means anything unless the caller actually presented a certificate, and that is a separate switch with its own object.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: STRICT   # refuse plaintext: every caller must present a certificate

Prove the Deny Happened, Do Not Assume It

An applied policy and a working policy are two different claims. Test both directions from inside the mesh: the same call from an identity you did not grant, and from one you did.

kubectl apply -f authz-policies.yaml

authorizationpolicy.security.istio.io/deny-all created
authorizationpolicy.security.istio.io/allow-frontend-checkout created

# Same request, two identities. 'reports' was never granted anything.
kubectl exec -n prod deploy/reports -c reports -- \
  curl -sS -w '\n-> %{http_code}\n' -X POST http://api.prod.svc.cluster.local/checkout
 
kubectl exec -n prod deploy/frontend -c frontend -- \
  curl -sS -w '\n-> %{http_code}\n' -X POST http://api.prod.svc.cluster.local/checkout

RBAC: access denied
-> 403
{"order":"6f21c9","status":"created"}
-> 200

RBAC: access denied is Envoy's own response body, not yours. The request stopped at the sidecar and your application never heard about it. If you ever see a 403 carrying your app's own error page instead, the mesh allowed the call and your code rejected it, which is a completely different bug in a completely different place.

A 403 on its own does not tell you why. Envoy records the reason in a field called response-code-details, which Istio's default access log format prints right after the response code and the response flags. Access logging is switched off in a default install, so turn it on before the day you need it.

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system   # the mesh root namespace, so this covers everything
spec:
  accessLogging:
    - providers:
        - name: envoy       # built-in provider: writes to the sidecar's stdout

kubectl logs -n prod deploy/api -c istio-proxy --tail=1

[2026-07-21T09:14:07.512Z] "POST /checkout HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/8.6.0" "b7f1a0c2-3d4e-4a8b-9c11-5f7a2d0e6b93" "api.prod.svc.cluster.local" "-" inbound|8080|| - 10.244.2.31:8080 10.244.1.17:41216 outbound_.8080_._.api.prod.svc.cluster.local default

Read rbac_access_denied_matched_policy[none]. The [none] says no ALLOW rule matched, so the request hit the default-deny floor and the fix is to add a grant. Had an explicit DENY fired, the brackets would name the culprit, something like rbac_access_denied_matched_policy[ns[prod]-policy[block-admin]-rule[0]], and the fix would be to remove or narrow that deny instead. One bracket, two opposite remediations. Note the 19 in the byte count as well: that is the length of the RBAC: access denied body, a handy fingerprint when you are scanning thousands of log lines.

To confirm the closed default actually reached this pod's proxy, rather than merely landing in the API server, ask the proxy directly.

istioctl x authz check api-7d9f6c8b4-2xk9p.prod

ACTION   AuthorizationPolicy               RULES
ALLOW    allow-frontend-checkout.prod      1
ALLOW    deny-all.prod                     1

Both entries report one rule, which surprises people. deny-all has no rules in its YAML at all, yet Istio compiles a zero-rule ALLOW policy into a single generated rule, named ns[prod]-policy[deny-all]-rule[0] inside Envoy and built so that nothing can ever satisfy it. That generated rule is the floor you are relying on. Seeing deny-all.prod in this list is your evidence that the closed default is live in the proxy. If it is missing here, the pod is open regardless of what kubectl get authorizationpolicies tells you.

kubectl exec -n prod deploy/api -c istio-proxy -- \
  pilot-agent request GET stats | grep -E '\.rbac\.(allowed|denied)'

http.inbound_0.0.0.0_8080.rbac.allowed: 128
http.inbound_0.0.0.0_8080.rbac.denied: 3

Those two counters are the ones to alarm on. A denied rate that climbs right after a deploy is almost always a real flow nobody wrote a policy for, and it reaches you far faster than a support ticket does. One catch: the sidecar's admin endpoint always has these numbers, but Istio trims which Envoy stats it exports to Prometheus, so add the prefix to meshConfig.defaultConfig.proxyStatsMatcher if the counters are missing from your scrape.

Flipping the Switch Without an Outage

Nobody knows their own call graph. The set of services that actually talk to each other is always longer and stranger than the architecture diagram, and the gap between the two is what takes a namespace down at 4pm on the day somebody applies deny-all cold.

Build the list from evidence instead of memory. Istio's telemetry already stamps every request with the verified identity at both ends, so a week of data hands you the real edges. Prometheus, the metrics database those counters land in, will read them back to you.

# Every caller/callee identity pair seen in prod over the last 7 days.
# Each row is one ALLOW rule you owe the namespace before you close it.
sum by (source_principal, destination_principal, destination_service_name) (
  increase(istio_requests_total{destination_workload_namespace="prod",
                               reporter="destination"}[7d])
) > 0

Two warnings about that query. source_principal keeps its spiffe:// prefix in metrics, unlike the policy field, so strip the scheme before you paste anything into YAML. And the label comes back empty or unknown for any call that arrived without a certificate, which makes a blank caller its own finding: something is still speaking plaintext to prod.

Those default metrics carry identities but no methods or paths, so use them to draw the graph and your access logs to fill in the L7 detail (layer 7, the application layer, where HTTP methods and URL paths live) on the handful of routes you want to constrain tightly.

For a DENY policy you are about to enforce, Istio gives you a rehearsal: full performance, no audience. Add the istio.io/dry-run annotation and the rules compile into Envoy's shadow ruleset, evaluated on every request, counted and logged, never enforced.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-legacy-batch
  namespace: prod
  annotations:
    "istio.io/dry-run": "true"   # evaluate and report, do not enforce
spec:
  selector:
    matchLabels:
      app: api
  action: DENY
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/legacy-batch"]

# Turn on RBAC debug logging on the destination sidecar (output is long, discard it)
istioctl proxy-config log deploy/api.prod --level "rbac:debug" >/dev/null
 
# Let real traffic run, then read what the shadow ruleset would have done
kubectl logs -n prod deploy/api -c istio-proxy --tail=500 | grep 'shadow denied'

2026-07-21T09:31:44.118820Z	debug	envoy rbac	shadow denied, matched policy ns[prod]-policy[deny-legacy-batch]-rule[0]
2026-07-21T09:31:46.902114Z	debug	envoy rbac	shadow denied, matched policy ns[prod]-policy[deny-legacy-batch]-rule[0]

Every shadow denied line is a request that would have been rejected. Watch them until the volume and the callers match what you expect, then delete the annotation to enforce. The same results land on the proxy's RBAC metric in Prometheus, tagged authz_dry_run_action="deny" and authz_dry_run_result="denied", which is far easier to graph than log lines. Istio is blunt that the log, metric and tracing output for dry-run exists for manual troubleshooting and may change between releases, so read it with your eyes and do not wire it into a release gate.

Be honest about the limits. Dry-run is an alpha feature. It covers ALLOW and DENY policies only, and it cannot rehearse the default-deny floor at all, because that floor is the absence of a match rather than a policy that fires. The call-graph query above is what you use for the floor, and the rbac.denied counter is what you watch on the way in.

Where This Control Stops

L7 rules (methods, paths, headers, hosts) only work on traffic the sidecar can parse as HTTP or gRPC. Point one at an opaque TCP connection to Postgres and Istio does not politely ignore the mismatch. An ALLOW rule carrying an HTTP-only field becomes invalid on a TCP port and gets dropped, so the traffic it was written to permit is now refused. A DENY rule carrying an HTTP-only field goes the other way and denies the entire port. For TCP you get identity and port matching, nothing more, and a rejected TCP connection is closed rather than answered, so the client sees a reset with no explanation attached.

Path matching is the fragile part of any policy. /checkout, /checkout/, /CHECKOUT and %2fcheckout are four different strings, and the history of web authorization is full of bypasses that were really normalization bugs. Istio's own guidance is to pick one pattern and stay inside it: ALLOW using only positive fields such as paths and methods, or DENY using only negative fields such as notPaths and notValues, and never mix the two. Break that rule in the safe direction and you get a puzzling 403. Break it the other way and you get a silent bypass. Choose meshConfig.pathNormalization.normalization deliberately as well, from NONE, BASE (what the default currently resolves to: RFC 3986 tidying plus backslashes converted to forward slashes), MERGE_SLASHES and DECODE_AND_MERGE_SLASHES. Better still, prefer identity plus method over long lists of literal paths wherever the design lets you.

Anything that never reaches the sidecar sits outside this control entirely. Ports listed in traffic.sidecar.istio.io/excludeInboundPorts, port 22 and the sidecar's own ports (inbound capture skips those by default), pods running with hostNetwork, UDP traffic, and anything a container does over localhost inside its own pod are all invisible to it. Istio's documentation says plainly that relying on all traffic being captured is not secure. Keep Kubernetes NetworkPolicy underneath as a second, independent layer that does not depend on a sidecar being healthy.

Two boundaries are worth keeping straight in your head. Mesh authorization governs the data plane, meaning the running service-to-service calls. Kubernetes RBAC governs the control plane, meaning who may kubectl apply a Deployment or read a Secret. Neither substitutes for the other, and a stolen CI (continuous integration) token walks straight past your AuthorizationPolicies by editing them. The cost at scale, meanwhile, is not CPU: the Envoy RBAC filter runs in-process and is cheap. The cost is sprawl, hundreds of narrow rules that drift away from the services they were written for until nobody dares delete one.

One gap this control cannot close by design is sitting in that policy right now. principals: ["cluster.local/ns/prod/sa/frontend"] proves the frontend service made the call. It says nothing about which of your ten thousand customers was riding inside it, which means a compromised frontend can still replay anybody's checkout as often as it likes. Authorizing the request itself, against the end user's verified token, is what RequestAuthentication and the requestPrincipals field are for, and the tokens they read are JWTs (JSON Web Tokens, the signed blobs your login system hands out). That is the next lesson.

Try this

Run kubectl apply -f authz-policies.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: pERMISSIVE mTLS Quietly Guts Identity Rules. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related