mTLS & pod-to-pod encryption

Identity and encryption between services.

Two pods in the same cluster talk over the pod network in cleartext by default. No encryption, no identity check. Kubernetes hands every pod a routable IP and lets them reach each other flat, which is convenient and completely unauthenticated. This is the default almost everywhere, and most teams never notice until an audit or an incident makes them look. Whatever opens a TCP connection to payments-api on port 8080 gets served, and anything sitting on the path can read every byte. On a flat cluster network that path is short. A compromised pod, a malicious CNI (Container Network Interface) plugin, a node someone has rooted: all of them are already close enough to sniff or splice traffic that was never protected. Mutual TLS, or mTLS, closes both gaps in one handshake. TLS (Transport Layer Security) is the same encryption that locks down every HTTPS site; it scrambles the connection so a sniffer sees only ciphertext. The mutual part adds the second guarantee: each end presents a certificate that proves who it is, not just the server. So payments-api can tell the real web frontend from something that merely reached its port and started talking.

You could bolt TLS into every service by hand, but almost nobody does. The common move is a service mesh: Istio or Linkerd inject a small proxy container, a sidecar, next to each of your pods. Think of it as a translator who sits in on every call and redoes it over a scrambled line. Your app keeps speaking plain HTTP to localhost. The sidecar catches that traffic, wraps it in mTLS, and only the sidecar on the receiving pod holds the key to unwrap it. The apps never learn TLS was involved. That's how a mesh encrypts every connection in a namespace without touching a line of application code.

Turn it on. Then prove plaintext is dead.

A fresh mesh is polite about plaintext. Out of the box it accepts mTLS and cleartext both, which is exactly what you want while you're still rolling sidecars out one workload at a time. That mode is PERMISSIVE. The one you want to finish on is STRICT: refuse anything that isn't mutually authenticated. It's the difference between a door that opens for a keycard or a polite knock, and one that opens for the keycard only. Skip PERMISSIVE and jump straight to STRICT and you're not migrating, you're scheduling an outage. You set the mode per namespace with a PeerAuthentication object.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata: { name: default, namespace: payments }
spec:
  mtls:
    mode: STRICT      # refuse any non-mTLS traffic in this namespace

Applying it is one command. Proving it took is two more. First confirm the mode the mesh actually computed for the workload, because a mesh-wide policy or a workload-specific one can override what you assume is in effect. Then show that a client with no sidecar gets its connection dropped instead of a friendly reply.

$ kubectl apply -f peerauth.yaml
peerauthentication.security.istio.io/default created
 
$ istioctl x describe pod payments-api-7d9f8c-abcde.payments
...
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
 
$ kubectl exec -n legacy deploy/curl -- curl -sS payments-api.payments:8080
curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

Identity is the whole point

Encryption is the easy half. The reason mTLS matters for security is the certificate underneath it. A mesh issues those certs through SPIFFE (Secure Production Identity Framework For Everyone), which gives every workload a name like spiffe://cluster.local/ns/payments/sa/payments-api and bakes it into a short-lived certificate that the mesh CA (certificate authority, the component that signs and vouches for certs) rotates on its own, often every 24 hours. Picture a building badge that photographs you on the way in, expires at lunch, and reprints itself before you notice. Short lifetimes matter for a reason: a cert lifted off a compromised pod is worthless within a day, so the blast radius of a leak stays small. Because every connection is now tied to a specific, verified identity, you can stop reasoning about IP addresses, which lie and churn, and start writing rules about who is calling. Don't take the mesh's word for it, though. Read the identity straight out of the live cert.

$ istioctl proxy-config secret payments-api-7d9f8c-abcde.payments -o json \
  | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/payments/sa/payments-api

This is the part people get wrong. STRICT mTLS proves identity. It does not decide what an identity is allowed to do. Authentication answers who are you; authorization answers what you may touch, and mTLS only does the first. Once STRICT is on, every authenticated workload in the mesh can still reach every other one, because you haven't told the mesh otherwise. In a real breach that gap is exactly what lets a foothold in one namespace pivot into your payment path unchallenged. The fix is an AuthorizationPolicy that names the exact SPIFFE identities allowed to call a service, checked at L7 (the application layer, where the sidecar can read the request) by the receiving proxy.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata: { name: payments-allow-web, namespace: payments }
spec:
  selector: { matchLabels: { app: payments-api } }
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/storefront/sa/web"]   # SPIFFE identity

$ kubectl apply -f authz.yaml
authorizationpolicy.security.istio.io/payments-allow-web created
 
$ kubectl exec -n storefront deploy/web -- \
    curl -s -o /dev/null -w '%{http_code}\n' payments-api.payments:8080/charge
200
 
$ kubectl exec -n analytics deploy/reporter -- \
    curl -s payments-api.payments:8080/charge
RBAC: access denied

No mesh? Then it's your code's job.

Not every cluster runs a mesh, and you don't need one to get mTLS. Without a mesh, though, the wiring is yours to own. cert-manager is the usual in-cluster CA: you define an Issuer, request a Certificate, and it drops a TLS secret into the namespace for your pod to mount. Your client then verifies the server's cert and, because this is mutual, the server verifies the client's right back. Same guarantee as the mesh, more moving parts, and rotation you have to watch yourself instead of getting it for free. Miss a renewal and every client that trusted the old cert starts failing handshakes at the worst possible hour, which is the failure mode meshes exist to spare you. Two things are worth confirming: that the cert was actually issued, and that a real client-authenticated handshake completes end to end.

$ kubectl get certificate payments-tls -n payments
NAME           READY   SECRET         AGE
payments-tls   True    payments-tls   40s
 
$ kubectl exec -n storefront deploy/web -- \
    openssl s_client -connect payments-api.payments:8443 \
    -cert /certs/tls.crt -key /certs/tls.key -CAfile /certs/ca.crt </dev/null 2>/dev/null \
    | grep -E 'subject=|Verify return code'
subject=CN = payments-api
    Verify return code: 0 (ok)

Without a mesh, you own TLS in the app or sidecar you maintain. Half-enabled TLS (server only) still leaves spoofed clients in play.

Certificates need rotation. Short-lived workload identities beat long-lived shared PEMs checked into git.

Combine mTLS with NetworkPolicy. Encryption without authorization still lets any authenticated workload talk too widely if the mesh trusts too many identities.

Peer authentication STRICT will break bare curl probes and that is the point. Give developers a documented mesh-aware debug path so they do not "temporarily" switch back to PERMISSIVE and leave it there. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

With a mesh (or a lab Istio/Linkerd install), verify peer authentication is STRICT and that a plaintext curl from a non-mesh pod fails.

$ kubectl -n payments get peerauthentication -o yaml | head -30
apiVersion: security.istio.io/v1
kind: PeerAuthentication
...
  mtls:
    mode: STRICT
 
$ kubectl -n payments run plain --rm -it --image=curlimages/curl:8.7.1 --restart=Never -- \
  curl -s --max-time 3 http://payments-api:8080/health || echo PLAINTEXT_FAIL
PLAINTEXT_FAIL
 
$ kubectl -n payments exec deploy/payments-api -c istio-proxy -- \
  pilot-agent request GET /stats/prometheus | grep ssl.handshake | head
...ssl.handshake... 42

Takeaway

Cluster DNS and flat pod IPs are not identity. Turn on mTLS so every hop authenticates and encrypts, then prove plaintext dies.

Related