mTLS at scale

STRICT mode and the PERMISSIVE migration path.

A normal HTTPS connection is a one-sided introduction. The server shows its badge, your browser reads it, and then the two of you talk. The server has no idea who you are. For a shop's website that is exactly the right trade, because the whole point is serving strangers. Inside a cluster it is a bad trade. Every caller in there should be someone you can name, and the caller is almost always another one of your own services.

Mutual TLS (mTLS: both ends of the connection prove who they are with a certificate, instead of only the server doing it) closes that gap. TLS is Transport Layer Security, the same machinery behind the padlock in your browser. In a service mesh, mTLS becomes the standing posture for east-west traffic (service talking to service inside the cluster, as opposed to north-south traffic entering and leaving it). Every hop encrypted. Every caller named. Turning it on for one service takes five minutes. Turning it on everywhere, in a cluster that is currently serving customers, without cutting connections you did not know existed, is the part that needs a plan.

What the handshake actually proves

Every meshed pod gets a second container standing next to the application: a proxy called Envoy, usually called the sidecar. Linux packet-redirect rules (iptables, written either by a small init container or by the Istio CNI plugin, where CNI is the Container Network Interface, the thing that wires up pod networking) bend all traffic in and out of the pod through that proxy first. Treat the sidecar as a receptionist sitting between the office and the front door. Your application keeps speaking plain HTTP to its own receptionist over the pod's loopback interface and never knows anything changed. That is why mTLS costs you no application code. It is also why the application never handles the caller's certificate itself. For HTTP traffic the sidecar does pass the verified caller identity along in an X-Forwarded-Client-Cert header, so an app can read who called, but that header is only trustworthy because nothing unmeshed can reach the app directly.

When frontend calls api, the two sidecars run a TLS handshake and each side presents an X.509 certificate (X.509 is the certificate format the whole internet uses). The field that matters is not the common name that web certificates lean on. It is the Subject Alternative Name (SAN, the list of names a certificate is actually valid for), and in a mesh that SAN is a URI (uniform resource identifier, a structured string that names a thing): spiffe://cluster.local/ns/prod/sa/frontend. Read it left to right like a postal address: trust domain (cluster.local), namespace (prod), Kubernetes service account (frontend). That string is a SPIFFE ID (Secure Production Identity Framework For Everyone, the open standard for naming workloads), and the certificate carrying it is an X.509-SVID (SPIFFE Verifiable Identity Document). The Subject field is left deliberately empty, which is why the SAN extension is marked critical. The name lives in the SAN and nowhere else.

So a finished handshake hands you exactly two facts, in the same instant. The channel is encrypted. And each side holds the private key for a certificate that the mesh certificate authority (CA, the service that signs certificates) issued to that specific SPIFFE ID. One-way TLS gives you the first fact and leaves the caller anonymous, which is precisely the unattributed lateral movement that zero trust exists to stop.

Now the part people skip, and it is the part that gets teams into trouble. A successful handshake does not mean the call is allowed. Every meshed workload gets a certificate, including the pod an attacker lands on. STRICT mTLS with no authorization policy behind it gives you an encrypted flat network: everything can still call everything, and now your network sensors cannot read any of it either.

Three more limits worth carrying around in your head. Identity is only as fine-grained as the Kubernetes service account behind it, so ten deployments sharing the default service account in prod are one identity to the mesh, and no policy can tell them apart. The peer identity says nothing about the human or the token behind the request; that is a separate check, usually on a JWT (JSON Web Token, a signed blob of claims about a user). And a certificate proves a workload is who it says it is. Never that it is behaving.

Three modes, and which one wins

Enforcement comes down to one resource and one field. The resource is PeerAuthentication in the security.istio.io/v1 API group (API here meaning the set of Kubernetes resource types Istio installs; older write-ups show v1beta1, which is the same resource under its previous name). The field is mtls.mode. Think of it as the policy on a door. DISABLE is the door propped open: plaintext only, which you want for a genuinely legacy port and nothing else. PERMISSIVE is a door with a lock you have not turned yet, accepting both mTLS and plaintext on the same port, so the proxy speaks mTLS to meshed callers and still answers everyone else in the clear. STRICT is the lock turned. A fourth value, UNSET, inherits from the next scope up, and it is what you have when you have written no policy at all. Install a mesh, configure nothing, and the behaviour you get is equivalent to PERMISSIVE, chosen so that installing Istio does not break your cluster on day one.

Scope comes from where the policy lives and whether it carries a selector, never from what you name it. In the root namespace (istio-system unless you changed it) with no selector, one object covers the entire mesh, and convention names that one default. In an ordinary namespace with no selector, it covers that namespace, and you want exactly one such policy per namespace. Add a selector and it covers matching workloads. Add portLevelMtls and it covers a single port on those workloads. Narrow wins: port beats workload, workload beats namespace, namespace beats mesh.

# Scope comes from location + selector, never from the resource name:
#   istio-system (root ns) + no selector -> the whole mesh
#   any namespace + no selector          -> that namespace
#   + selector                           -> matching workloads
#   + portLevelMtls                      -> one port on those workloads
# Narrow wins: port > workload > namespace > mesh.
 
# --- Migration state: accept mTLS AND plaintext across prod. -------------
# Step 2 is this same object with mode: STRICT, applied only once the
# destination-side metrics read mutual_tls on every row.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: PERMISSIVE
---
# --- One documented exception -------------------------------------------
# A Prometheus that lives outside the mesh still scrapes api:9990.
# portLevelMtls REQUIRES a workload selector, and the number is the
# workload's own container port, not the Service port sitting in front.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: api-metrics-exception
  namespace: prod
spec:
  selector:
    matchLabels:
      app: api
  mtls:
    mode: STRICT            # every other port on this workload: STRICT
  portLevelMtls:
    9990:
      mode: PERMISSIVE      # tech debt: owner + removal date in the PR

kubectl apply -f peer-auth.yaml
 
# ...and here is what a port-level exception without a selector gets you:
kubectl apply -f - <<'YAML'
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: metrics-exception-broken
  namespace: prod
spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    9990:
      mode: PERMISSIVE
YAML

peerauthentication.security.istio.io/default created
peerauthentication.security.istio.io/api-metrics-exception created
 
Error from server: error when creating "STDIN": admission webhook
"validation.istio.io" denied the request: configuration is invalid:
port level mTLS is only valid when workload selector is defined

Migrating a live namespace without cutting traffic

Sidecars are injected when a pod is created, never afterwards. Labelling a namespace therefore does nothing at all to the pods already running in it, so kubectl label ns prod istio-injection=enabled (or istio.io/rev=<revision> on a revisioned install) followed by kubectl rollout restart deploy -n prod is one step, not two optional ones. Anything you forget to restart is a workload with no proxy, which means no identity, no metrics, and nothing for STRICT to enforce on.

Before you touch the mode, check the other half of every connection. PeerAuthentication governs only what a workload accepts. What a caller sends is decided by automatic mTLS, where istiod (the mesh control plane, the component that configures every proxy and signs every certificate) tells each client proxy to use mTLS whenever the destination has a sidecar, unless a DestinationRule overrides it. A rule left over from an old debugging session with trafficPolicy.tls.mode: DISABLE will keep that client sending plaintext into a server you have moved to STRICT. The symptom is an HTTP 503 reading upstream connect error or disconnect/reset before headers. reset reason: connection termination, and it will not be obvious where it came from. Sweep for those rules first.

# A blunt sweep: catches tls.mode DISABLE wherever it hides, including
# inside subsets and portLevelSettings, which a top-level check misses.
kubectl get destinationrule -A -o json \
| jq -r '.items[]
         | select((.spec|tostring) | contains("\"mode\":\"DISABLE\""))
         | "\(.metadata.namespace)/\(.metadata.name)  host=\(.spec.host)"'

prod/api-debug-nodtls  host=api.prod.svc.cluster.local

Then measure, because "we injected everything" is a belief and what you need is a number. Istio's standard request metric carries a label that says exactly what happened on the wire: connection_security_policy, which reads mutual_tls, none, or unknown. Two details make the query trustworthy. Filter on reporter="destination", because the sending proxy records unknown here (it cannot know what the far end enforced). And remember that a workload with no sidecar reports nothing at all, so an empty result is not evidence of safety, it is evidence that nobody is watching.

# The addon Prometheus image ships without curl or jq, so forward the
# port and query it from your own machine.
kubectl -n istio-system port-forward svc/prometheus 9090:9090 >/dev/null &
 
# Which inbound traffic to prod is already mTLS, and who still sends
# cleartext?
QUERY='sum by (source_workload, destination_workload, connection_security_policy)
       (rate(istio_requests_total{reporter="destination",
                                  destination_workload_namespace="prod"}[5m]))'
 
curl -sG --data-urlencode "query=$QUERY" http://localhost:9090/api/v1/query \
| jq -r '.data.result[] |
    "\(.metric.source_workload) -> \(.metric.destination_workload)  \(.metric.connection_security_policy)  \(.value[1]|tonumber|round) rps"'

frontend -> api  mutual_tls  42 rps
checkout -> api  mutual_tls  12 rps
unknown -> api  none  3 rps
frontend -> web  mutual_tls  10 rps

One row there is your entire migration risk. unknown -> api none 3 rps means something with no mesh identity is pushing three requests a second of cleartext into api. Flip STRICT now and those three requests a second become connection resets, and whatever depends on them starts paging someone. Find the sender before you enforce. It is usually a virtual machine outside the cluster, a namespace nobody labelled, a load balancer health check, or a monitoring agent that was meshed by exactly nobody. Run the same query against istio_tcp_connections_opened_total as well, because istio_requests_total only counts HTTP and gRPC, and your database and cache traffic will be invisible in the first query. PERMISSIVE is where you do that hunting, and it is a migration state rather than a resting state, because while you sit in it anyone who can route a packet to the pod IP gets an unauthenticated plaintext session.

Proving STRICT actually took effect

kubectl apply proves the API server accepted your YAML. It proves nothing about live traffic. Start by having istioctl resolve the four possible scopes into one answer, so you are not guessing which policy won. istioctl experimental describe pod (short form istioctl x describe pod, still experimental in current releases) prints the effective mode and the policies that produced it.

istioctl experimental describe pod frontend-6b4c8f9-abcde -n prod

Pod: frontend-6b4c8f9-abcde
   Pod Revision: default
   Pod Ports: 8080 (frontend), 15090 (istio-proxy)
--------------------
Service: frontend
   Port: http 8080/HTTP targets pod port 8080
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.prod

Then run the negative test, which is the only one an auditor should accept. Call the service from a pod that has no sidecar, in a namespace with no injection label, and watch what comes back.

# The 'legacy' namespace is deliberately NOT labelled for injection,
# so this pod has no proxy and no mesh identity.
kubectl run probe --image=curlimages/curl:8.9.1 -n legacy \
  --restart=Never -it --rm -- \
  curl -sS --max-time 5 http://api.prod.svc.cluster.local:8080/health

curl: (56) Recv failure: Connection reset by peer
pod "probe" deleted
pod legacy/probe terminated (Error)

A reset, not a 403. That distinction is the whole point. Under STRICT the receiving Envoy has no configuration that accepts a plaintext connection, so the socket dies before a single byte of HTTP is parsed. If you ever get a real HTTP status back from this test, something answered you, and you should go find out what. You can see the reason in the proxy's own configuration. All captured inbound traffic is redirected to port 15006, and each filter chain on that listener declares which transport it matches. tls chains handle mTLS. raw_buffer chains handle plaintext, and PERMISSIVE is exactly what puts them there.

# Run before and after the flip. 15006 is where every captured inbound
# connection lands; raw_buffer chains are the plaintext doors.
istioctl proxy-config listener deploy/api -n prod --port 15006 -o json \
  | jq -r '.[0].filterChains[].filterChainMatch.transportProtocol // "any"' \
  | sort | uniq -c

# while prod was PERMISSIVE
      4 raw_buffer
      5 tls
 
# after mode: STRICT
      5 tls
 
# (the counts scale with how many ports the workload declares; what
#  matters is whether raw_buffer appears at all)

Certificates that die young

A hotel key card that stops working at checkout is a very different security proposition from a brass key that opens the door forever. Mesh identity is the key card. The agent inside each pod generates a private key in memory, sends a CSR (certificate signing request, a formal "please sign this public key for this name") to istiod, and hands the signed certificate to Envoy over SDS (Secret Discovery Service, the API Envoy uses to receive secrets at runtime). No key is written to disk, so kubectl cp out of the sidecar gets an attacker nothing, and a stolen node disk image gets them nothing. One honest caveat: the projected service account token the agent uses to prove who it is *is* mounted into the pod's filesystem. Anyone who can read that token and reach istiod can ask for a certificate for that workload from anywhere. It is audience-bound and rotates every 12 hours by default, which limits the damage, but that token is the thing worth guarding. You can read the live certificate straight out of the proxy.

istioctl proxy-config secret deploy/frontend -n prod
 
# and the leaf itself, decoded:
istioctl proxy-config secret deploy/frontend -n prod -o json \
  | jq -r '.dynamicActiveSecrets[]
           | select(.name=="default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

RESOURCE NAME   TYPE         STATUS   VALID CERT   SERIAL NUMBER   NOT AFTER              NOT BEFORE
default         Cert Chain   ACTIVE   true         229384...       2026-07-22T09:14:32Z   2026-07-21T09:14:02Z
ROOTCA          CA           ACTIVE   true         1e5f0a...       2036-07-18T08:59:24Z   2026-07-20T08:59:24Z
 
subject=
issuer=O = cluster.local
notBefore=Jul 21 09:14:02 2026 GMT
notAfter=Jul 22 09:14:32 2026 GMT
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/prod/sa/frontend

An empty subject, a URI SAN, and a 24 hour window. That lifetime is the default (SECRET_TTL on the agent, where TTL means time to live), and the agent asks for a fresh leaf at half of it, because SECRET_GRACE_PERIOD_RATIO defaults to 0.5. Renewal is a new SDS push into memory: no restart, no downtime, nothing for a human to forget. istiod caps whatever the agent requests at MAX_WORKLOAD_CERT_TTL, 90 days by default. Here is the honest trade. There is no revocation list and no OCSP responder (Online Certificate Status Protocol, the live "is this certificate still good?" lookup) for these certificates, so the short lifetime *is* the revocation mechanism. Shortening it further means more signing traffic hitting istiod, and on a mesh with twenty thousand proxies that is measurable control-plane CPU. Pick a number you can defend, then watch istiod's CPU when you change it.

If your identities have to reach past Kubernetes, out to virtual machines, another cloud, or a partner's cluster, you can put SPIRE underneath the mesh instead of istiod's built-in CA. SPIRE is the reference SPIFFE implementation: spire-server signs, and a spire-agent runs on every node. The SPIFFE CSI driver (CSI: Container Storage Interface, the standard way to mount things into pods) mounts the agent's Workload API socket into the sidecar, Envoy fetches SVIDs over that same SDS protocol, and identity now comes from attestation (the agent proving what a workload really is by inspecting the node and the pod) rather than from a service account token alone. Two things have to line up or nothing validates. Istio must be installed with values.global.caAddress pointing at the mounted socket, unix:///run/secrets/workload-spiffe-uds/socket, and SPIRE's trust domain must equal Istio's meshConfig.trustDomain. The cost is that entries become explicit. Nothing gets an identity until you register it.

# k8s_psat = the Kubernetes projected service account token node
# attestor: how the agent itself proved which node it runs on.
spire-server entry create \
  -spiffeID  spiffe://acme.internal/ns/prod/sa/frontend \
  -parentID  spiffe://acme.internal/spire/agent/k8s_psat/prod-cluster/8a1f...c9 \
  -selector  k8s:ns:prod \
  -selector  k8s:sa:frontend \
  -selector  k8s:pod-label:spiffe.io/spire-managed-identity:true \
  -dns       frontend \
  -dns       frontend.prod.svc

Entry ID         : 5f2f0a5c-9f3c-4c2e-9d3a-1a2b3c4d5e6f
SPIFFE ID        : spiffe://acme.internal/ns/prod/sa/frontend
Parent ID        : spiffe://acme.internal/spire/agent/k8s_psat/prod-cluster/8a1f...c9
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:ns:prod
Selector         : k8s:pod-label:spiffe.io/spire-managed-identity:true
Selector         : k8s:sa:frontend
DNS name         : frontend
DNS name         : frontend.prod.svc

Where STRICT stops covering you

STRICT is enforced by a proxy. If the proxy never sees the packet, the policy is a comment in a YAML file. Traffic escapes capture in five ordinary ways: ports listed in the traffic.sidecar.istio.io/excludeInboundPorts annotation; pods in namespaces nobody labelled for injection, or carrying sidecar.istio.io/inject: "false"; pods running on the host network; UDP services (User Datagram Protocol, the connectionless cousin of TCP), because the capture rules only redirect TCP; and anything a process sends to 127.0.0.1 inside its own pod. All five produce the same misleading picture. istioctl reports STRICT, the dashboards are green, the metrics show nothing at all for that path, and cleartext keeps flowing. Verify that the sidecar sees the traffic, not that the policy claims a mode.

The cost is real too, and it is mostly handshakes rather than bulk encryption. Modern CPUs encrypt at close to line rate; setting up a new TLS session is the expensive event. So connection reuse decides your bill. HTTP/2 and keep-alive spread one handshake across thousands of requests. A batch job that opens a fresh TCP connection per record pays for a handshake every single time, and surfaces later as proxy CPU nobody can account for. Istio's own published figures have sat around a third of a vCPU and 40 MB of memory per proxy per 1000 requests per second, with roughly two to three milliseconds added at the 90th percentile for a request crossing a pair of proxies. Treat those as a shape, not a promise, and measure your own version under your own traffic. Proxy memory tracks how many other services each sidecar has been told about, which is why a Sidecar resource (networking.istio.io/v1) that trims each proxy's view of the mesh is a standard scaling move.

Two escape hatches when the per-pod tax hurts. Istio's ambient mode moves mTLS out of the pod and into a per-node component called ztunnel, which carries traffic over HBONE (HTTP-Based Overlay Network Environment, an mTLS-wrapped HTTP/2 tunnel on port 15008): same identities, same STRICT semantics, one proxy per node instead of one per pod. Be careful with exceptions there, because ztunnel does not honour portLevelMtls carve-outs the way a sidecar does, so re-test every one of them before you move a namespace. Linkerd starts from the other end. Meshed pod-to-pod TCP is mTLS the moment both sides are meshed, with no PERMISSIVE switch to forget, and its equivalent of STRICT is the inbound policy default, config.linkerd.io/default-inbound-policy: all-authenticated. Its coverage check is linkerd viz edges deploy -n prod, where the SECURED column plays the role of the metric query above.

When you do reach that final flip, run the metric query once more with the namespace filter removed, and let it cover a full business day, including the 02:00 batch window when the reporting job wakes up and opens a plaintext connection nobody has thought about since 2019. Any row that still reads none is not a metric. It is tomorrow's incident, already labelled with a workload and a namespace, waiting for you to fix it while nobody is watching.

Try this

Run kubectl apply -f peer-auth.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: mTLS is a door policy, not a guest list. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related