Service mesh architecture

Sidecar vs ambient; what the mesh enforces.

A courier company can promise sealed, signed-for parcels in two ways. Put a notary at every desk, or put one notary in the mailroom on each floor. Same promise, very different bill. Desk or mailroom is the whole sidecar-versus-ambient argument in a service mesh, and it decides how much of your cluster goes to security plumbing instead of your product. A service mesh wraps every service-to-service call in identity, encryption, and authorization without your application writing a line of security code. This lesson takes the machinery apart so you can say exactly where each guarantee is enforced, what it costs, and what it still fails to prove.

Two Planes: Who Decides, Who Enforces

Every mesh splits into a control plane and a data plane, and almost everything about its security posture falls out of that split. The control plane is the back office. In Istio it is a single component, istiod. It runs the mesh CA (certificate authority, the office that signs identity documents and vouches for them), minting a short-lived X.509 certificate for every workload. It watches Kubernetes for your PeerAuthentication and AuthorizationPolicy resources and compiles them into proxy configuration. Then it streams that configuration to the proxies over xDS (a family of discovery service APIs Envoy uses to fetch its clusters, listeners, and routes while it is running, with no restart) and the certificates over SDS (Secret Discovery Service, the xDS flavour that carries private keys and certificates). Those certificates land straight in proxy memory and never touch disk.

istiod reissues each certificate long before it expires. The default workload certificate lives 24 hours and renewal starts at the halfway mark, set by SECRET_GRACE_PERIOD_RATIO (default 0.5). A little randomness on top, SECRET_GRACE_PERIOD_RATIO_JITTER (default 0.01, about 15 minutes spread across a 24-hour certificate), staggers renewals so ten thousand proxies do not all queue at the CA in the same second. If you need a different lifetime, SECRET_TTL on the proxy sets it, and anything past 90 days is refused. The data plane is the fleet of proxies sitting in the actual request path. They hold the private keys, run the mTLS handshake (mutual Transport Layer Security, where both ends present a certificate rather than only the server), and enforce policy on every live connection. Control plane decides, data plane enforces. istiod never sees a byte of your application traffic.

That split hands you a specific failure mode. Learn it before you meet it at 3 a.m. Kill istiod and existing proxies keep running on the configuration they already cached, so traffic survives. What you lose is change. No new policy lands, no new pod can get a certificate, and once running certificates hit their renewal window with no CA to answer, handshakes start failing. A control-plane outage is a fuse burning down, not a switch flipping off. istioctl proxy-status asks istiod which proxies are connected and what configuration they have subscribed to. Run it first, every time, when somebody says a policy "is not working".

istioctl proxy-status

NAME                             CLUSTER      ISTIOD                   VERSION   SUBSCRIBED TYPES
checkout-6d4b9c8f7-2xk9v.prod    Kubernetes   istiod-5c9b7d8f4-lm2rt   1.30.3    4 (CDS,LDS,EDS,RDS)
frontend-7c9f8b5d6-4qde2.prod    Kubernetes   istiod-5c9b7d8f4-lm2rt   1.30.3    4 (CDS,LDS,EDS,RDS)
payments-84f6d7c9b-h7n2p.prod    Kubernetes   istiod-5c9b7d8f4-lm2rt   1.30.3    4 (CDS,LDS,EDS,RDS)

Read that table twice. First for who is missing: a pod that is running but never appears here has no proxy talking to istiod at all, which normally means injection never happened or the proxy is crash-looping. Second for SUBSCRIBED TYPES, the count and names of the configuration streams that proxy asked for, where CDS is clusters, LDS listeners, EDS endpoints, and RDS routes. What this default view does not tell you is whether the last push actually landed. Add -v 1 and istioctl breaks the table into one column per configuration type. Each cell then reads SYNCED (2m) when the proxy acknowledged istiod's last push, and how long ago; STALE when istiod pushed and never got an acknowledgement back; NOT SENT when istiod had nothing to send; or IGNORED when that proxy does not subscribe to that type at all. STALE is the one that ruins your afternoon, because the proxy is quietly running older rules than the YAML in Git claims.

Sidecar: A Notary At Every Desk

In the sidecar model, which covers classic Istio and all of Linkerd, a proxy container is injected into every pod: Envoy for Istio, the purpose-built Rust linkerd2-proxy for Linkerd. Your application opens what it thinks is an ordinary connection to the destination. Its own sidecar catches that connection, wraps it in mTLS, and ships it to the destination pod, where the receiving sidecar checks the certificate, applies policy, and hands plaintext to the app. Your app is the person who drops a letter in the out-tray and never learns that somebody sealed it. The interception is iptables, the Linux kernel's packet-filtering rules. At injection time Istio adds an init container called istio-init that writes NAT rules (network address translation, rewriting where a packet is headed) redirecting the pod's outbound traffic to port 15001 and its inbound traffic to port 15006, both served by Envoy. Install the Istio CNI node agent (Container Network Interface, the plugin layer Kubernetes uses to wire up pod networking) and the same redirect gets programmed from the node instead, so your pods stop needing the NET_ADMIN and NET_RAW capabilities. With CNI in place the injected init container is named istio-validation and does nothing but confirm the redirect is really there. Either way, the application has no proxy setting, no library, no code change, and no idea.

# Enroll the namespace, then restart so pods come back with a proxy
kubectl label namespace prod istio-injection=enabled
kubectl rollout restart deployment -n prod
 
# Where did the proxy actually land in the pod spec?
kubectl get pod -n prod -l app=frontend \
  -o jsonpath='{.items[0].spec.initContainers[*].name}'; echo
kubectl get pod -n prod -l app=frontend \
  -o jsonpath='{.items[0].spec.containers[*].name}'; echo

namespace/prod labeled
deployment.apps/checkout restarted
deployment.apps/frontend restarted
deployment.apps/payments restarted
istio-init istio-proxy
frontend

That output comes from a cluster on Kubernetes 1.33, so istio-proxy turns up under initContainers rather than containers. That is the native sidecar layout: an init container carrying restartPolicy: Always, which tells the kubelet to start it before your app containers, keep it running alongside them, and stop it last. On an older cluster the first line reads istio-init on its own and the second reads frontend istio-proxy. Both shapes are healthy. One application container in the second line and nothing Istio-shaped in the first is the failure. Make that check part of your rollout, because the failure is silent. If the namespace label is missing, or a pod carries the annotation sidecar.istio.io/inject: "false", you get your application container and nothing else: no encryption, no policy, no warning anywhere. The pod runs happily outside the mesh while your dashboard shows a green namespace.

A sidecar is a full L7 proxy (layer 7, meaning it parses HTTP: methods, paths, headers) dedicated to one workload. That buys per-request routing, retries, and authorization on verbs and paths. It also costs. Istio's published benchmark puts one sidecar at roughly 0.20 vCPU (virtual CPU, one core's worth of scheduling time) and 60 MB of memory per 1,000 requests per second, measured with two worker threads. Multiply by every pod you run. On a 2,000-pod estate the proxy fleet can outweigh some of the applications it protects, which is precisely why ambient mode exists. Latency is harder to quote honestly. Istio publishes percentile charts now rather than one headline figure, so treat any single number you find in a blog post, including the 2.65 ms that still circulates from old documentation, as folklore. What you can say for certain is that sidecar mode adds two proxy hops to every call. Measure your own 99th percentile, on your own payloads.

Ambient: One Notary Per Floor

Ambient mode splits the sidecar's job in two and moves the cheap half down to the node, one mailroom notary serving the whole floor. A component called ztunnel (zero-trust tunnel) runs as a DaemonSet (a Kubernetes object that guarantees one copy of a pod on every node) and handles L4 identity and mTLS for every pod on that node. Layer 4 means TCP connections: who is talking to whom, on which port, with no view inside the HTTP request. No sidecar, no pod restarts, and it went generally available in Istio 1.24. Traffic between nodes rides in an HBONE tunnel (HTTP-Based Overlay Network Environment, an HTTP CONNECT tunnel carried inside mTLS on port 15008), which keeps the real source and destination legible to the mesh while the payload stays encrypted.

# No restart needed: ztunnel is already running on every node
kubectl label namespace prod istio.io/dataplane-mode=ambient
 
# Which pods is ztunnel actually securing, and how?
istioctl ztunnel-config workload --workload-namespace prod

namespace/prod labeled
NAMESPACE  POD NAME                    ADDRESS      NODE       WAYPOINT  PROTOCOL
prod       checkout-6d4b9c8f7-2xk9v    10.244.2.19  worker-2   None      HBONE
prod       frontend-7c9f8b5d6-4qde2    10.244.1.11  worker-1   None      HBONE
prod       payments-84f6d7c9b-h7n2p    10.244.2.41  worker-2   None      HBONE

PROTOCOL: HBONE is the confirmation you want: those pods are enrolled, and ztunnel will speak mTLS on their behalf. A pod showing TCP in that column is reachable but not tunnelled. WAYPOINT: None is the other half of the story. ztunnel never parses HTTP, so it cannot tell a GET /health from a DELETE /orders/42. For L7 rules you deploy a waypoint proxy, a shared Envoy that you opt a namespace or a single service into. It is an ordinary Deployment you can size, scale, and reason about, rather than a thousand copies you cannot.

istioctl waypoint apply -n prod --enroll-namespace
kubectl get gateway -n prod

waypoint prod/waypoint applied
namespace prod labeled with "istio.io/use-waypoint: waypoint"
NAME       CLASS            ADDRESS        PROGRAMMED   AGE
waypoint   istio-waypoint   10.96.108.72   True         14s

# What istioctl generated, and what you should keep in Git instead
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: prod
  labels:
    istio.io/waypoint-for: service   # service (default), workload, all, or none
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

The trade-off inverts cleanly. Sidecar gives you L7 everywhere and charges you for it everywhere, including the forty services that only ever needed encryption. Ambient gives you identity and encryption for close to nothing: the same benchmark puts ztunnel at about 0.06 vCPU and 12 MB per 1,000 requests per second, roughly a third of the processor time and a fifth of the memory a sidecar spends on that same L4 work. Then it charges you for HTTP-aware policy only where you asked for it, around 0.25 vCPU and 60 MB per waypoint, and you run a handful of those rather than one per pod. Where the private key lives also changes, and that has teeth. In sidecar mode the Istio agent generates the key inside the pod and it never leaves; compromise one sidecar and you hold one workload's identity. In ambient the key sits in ztunnel on the node, outside your application's pod entirely, which keeps it away from your app code but gathers every identity scheduled on that node into one privileged process. Own a sidecar, own a service. Own a ztunnel, own a node's worth of services.

What The Mesh Enforces, And How You Check It

Because a proxy sits inside every connection, four guarantees land uniformly, identically for your new Go service and for the Java monolith nobody wants to touch. Identity. Each workload gets an X.509 certificate whose URI SAN (Subject Alternative Name, the certificate field that says who this certificate is for) holds a SPIFFE ID (Secure Production Identity Framework For Everyone, a portable workload name written as a URI), derived from the pod's Kubernetes service account. spiffe://cluster.local/ns/prod/sa/frontend is the workload's name. A pod IP address is not a name; it gets recycled in minutes. Linkerd does the same job in a different shape, putting a DNS SAN like frontend.prod.serviceaccount.identity.linkerd.cluster.local on the certificate, readable with linkerd identity -n prod po/frontend-7c9f8b5d6-4qde2. Encryption. East-west traffic between proxies is mutual TLS, so both ends prove who they are before a request moves. Authorization. Policy is evaluated against the cryptographically verified peer identity from that handshake, not a header a caller can type and not a source IP address a caller can spoof. Observability. The proxy terminates every call, so you get request counts, latencies, and a live who-talks-to-what graph without instrumenting anything. That graph is the raw material for writing tight policy later.

None of this needs taking on faith. istioctl proxy-config pulls live configuration out of a running Envoy, so what comes back is the certificate the proxy is serving right now, not the one you assume you deployed.

istioctl proxy-config secret deploy/frontend -n prod -o json \
  | jq -r '.dynamicActiveSecrets[] | select(.name=="default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d \
  | openssl x509 -noout -subject -ext subjectAltName -dates

subject=
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/prod/sa/frontend
notBefore=Jul 21 08:12:41 2026 GMT
notAfter=Jul 22 08:14:41 2026 GMT

The subject line is empty on purpose. A SPIFFE certificate carries no common name and no DNS name, so the identity lives only in that URI SAN, and every authorization decision keys off it and nothing else. The dates confirm the 24-hour default: a certificate stolen tonight is scrap by tomorrow morning. In ambient the equivalent check is istioctl ztunnel-config certificate. If you have replaced istiod's CA with SPIRE, the same SVID (SPIFFE Verifiable Identity Document, the certificate SPIRE hands a workload once it has proved what it is) reaches Envoy over that same SDS socket and reads identically here, because the SPIRE agent serves SDS from its Workload API socket. Enforcement is equally inspectable. PeerAuthentication in mode: STRICT tells the receiving proxies in a namespace to refuse plaintext outright.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default          # name 'default' + a namespace = namespace-wide
  namespace: prod
spec:
  mtls:
    mode: STRICT         # PERMISSIVE accepts both; DISABLE turns mTLS off

kubectl apply -f peerauth-strict.yaml
 
# Call the service from a pod that is NOT in the mesh
kubectl run probe -n default --rm -i --restart=Never \
  --image=curlimages/curl:8.11.1 \
  -- curl -sS -m 5 http://frontend.prod.svc.cluster.local:8080/

peerauthentication.security.istio.io/default created
curl: (56) Recv failure: Connection reset by peer
pod "probe" deleted
pod default/probe terminated (Error)

A connection reset, produced by the receiving proxy before one byte reached your application. Notice where that happened: at the destination. The receiving side is the enforcement point, which leads straight to the part people get wrong.

What mTLS Does Not Prove

mTLS proves one thing, precisely: the process at the other end of this connection holds a private key for a certificate your mesh CA signed for identity X. Strong statement. Narrow statement. It does not prove a user. spiffe://cluster.local/ns/prod/sa/frontend tells you a pod running as the frontend service account is calling. It says nothing about whose request it is carrying, or whether that person should be anywhere near this data. End-user authorization stays with your application, or with a JWT (JSON Web Token, a signed bundle of claims about a user) validated by a RequestAuthentication policy layered on top. It does not prove good behaviour. If an attacker gets code execution inside the frontend pod, the mesh hands them the frontend identity and politely encrypts their traffic for them. Certificates authenticate a workload, never its intentions. It does not prove your admission controls. Anyone who can create a pod in prod with serviceAccountName: frontend receives the frontend identity, full stop. Your mesh identity is exactly as strong as your Kubernetes RBAC (role-based access control, the rules deciding who may create or change which objects) over pod creation and service accounts. Loose RBAC quietly downgrades a serious mesh into decoration. It does not prove encryption end to end. The hop between the proxy and the application inside the pod is plaintext over the loopback interface, by design. Traffic is encrypted proxy to proxy, not process to process, so a packet capture inside the pod's network namespace still sees cleartext, and so does anything with a debugger on the app container.

Rolling It Out Without An Outage

A premature STRICT flip is the classic mesh outage, and the cure is not vague caution. It is one specific check. Every proxy exports istio_requests_total with a connection_security_policy label. Filter to reporter="destination" and that label records how the arriving request was actually secured: mutual_tls or none. Filter to reporter="source" and it reads unknown, because the sending proxy cannot know what the far end decided, so the destination side is the only side worth asking. You can ask the receiving proxy directly, before you change anything.

kubectl exec -n prod deploy/frontend -c istio-proxy -- \
  pilot-agent request GET stats/prometheus \
  | grep '^istio_requests_total{.*reporter="destination"' \
  | grep -o 'connection_security_policy="[a-z_]*"' \
  | sort | uniq -c

    412 connection_security_policy="mutual_tls"
      6 connection_security_policy="none"

Six plaintext requests. Apply STRICT now and those six callers get their connections reset in production, and you spend the next hour guessing which client broke. Find them first: something is outside the mesh, or has injection disabled, or is dialling the pod IP address directly. Only when that count is a flat zero across a full traffic cycle, including the nightly batch job nobody remembers owning, is STRICT boring. PERMISSIVE is the effective default, and that is exactly why the plaintext gets through. A namespace with no PeerAuthentication at all accepts cleartext from anything that can reach the pod, so "we installed the mesh" and "we enforce mTLS" are two different claims separated by that one manifest.

Two more habits keep the rollout quiet. Adopt one namespace at a time and let it soak, because a cluster-wide flip destroys your ability to blame a specific change for a specific breakage. And decide the overhead deliberately rather than meeting the bill during a traffic spike: set CPU and memory requests on the proxies, or move the namespaces that only ever needed identity and encryption onto ambient. So pin that stats query to your rollout checklist, run against every workload in the namespace rather than one convenient pod. Zero none on all of them, held through a full day, is your green light. The full migration path, including port-level exceptions and the multi-cluster edges where that count refuses to reach zero, is where mTLS at scale takes over.

Try this

Run istioctl proxy-status on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the proxy may not be ready when your app starts. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related