Encryption in transit & service-mesh mTLS

TLS everywhere, mesh identity, and short-lived cert rotation.

Your office building checks badges at the lobby turnstile. Past that turnstile, every interior door stands open. A courier who talks their way through reception can wander into any office, read whatever was left on a desk, and phone other floors claiming to be the CFO. That is the exact security you get when you terminate TLS (Transport Layer Security, the encryption behind the S in HTTPS) at the load balancer and then trust the network behind it. The encrypted lobby protects the one hop from the outside world. Everything past it, the calls your own services make to each other, runs in the clear. Mutual TLS (mTLS, where both ends of a connection prove who they are with certificates, the client as well as the server) closes both gaps at once: every hop is encrypted, and every caller shows cryptographic identification at every door.

North-South Guarded, East-West Wide Open

Two directions of traffic matter here. North-south traffic crosses your perimeter: a customer's browser talking to your load balancer, like mail arriving at the front desk. East-west traffic moves sideways between your own services, checkout calling ledger, like internal memos passed desk to desk. The old habit terminates TLS at the load balancer or Ingress and then forwards plaintext to the workload, because the internal network is "trusted." In a flat Kubernetes cluster or a shared VPC (Virtual Private Cloud, your own private slice of the cloud network) that trust is fiction. A compromised pod, a rogue sidecar, a mirrored switch port, or a packet capture on the node can all read east-west traffic, and can change it in flight. A plaintext hop also carries no proof of who is calling, so anything that reaches the wire can pretend to be anything else. A zero-trust data plane treats "internal" as hostile ground and insists on two things for every hop: encrypt it, and authenticate both ends.

How Mesh mTLS Actually Works

A service mesh puts a small proxy, called a sidecar, next to every workload. Istio uses Envoy for this. Think of it as a personal translator who sits beside each worker and handles every phone call in and out. Your application still speaks plain HTTP (HyperText Transfer Protocol, unencrypted) to localhost, which is cheap and simple. The sidecar catches that connection and wraps it in TLS, and here is the mutual part, it also presents a client certificate of its own. The two sidecars check each other's certificate against the mesh certificate authority (CA, the in-cluster service that signs identities) before one byte of your data moves.

What is written on that certificate is the interesting part. It does not carry an IP address or a hostname. It carries a workload identity in the form of a SPIFFE ID (Secure Production Identity Framework For Everyone, an open standard for naming workloads), something like spiffe://cluster.local/ns/payments/sa/checkout. Read it right to left: the checkout service account, in the payments namespace, in the cluster.local trust domain. It is a job title, not a desk number. Move the pod to another node, give it a fresh IP, scale it to fifty replicas, and the identity stays the same, because it comes from the pod's Kubernetes service account. The mesh CA signs these certs with a short life. Istio's default workload certificate lasts 24 hours, and the istio-agent inside each pod requests a new one at roughly half-life, about every 12 hours. The private key is generated in the pod and handed to Envoy over a local socket (the Secret Discovery Service, or SDS); it never touches disk. Nothing long-lived sits around to steal, and no human ever renews anything. Drain or revoke an identity and it stops being able to connect within one rotation window.

# PeerAuthentication: require mutual TLS for every workload in the namespace.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT          # plaintext is REFUSED, not silently upgraded
---
# AuthorizationPolicy: mTLS proves identity; THIS decides who may call the ledger.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ledger-allow-checkout
  namespace: payments
spec:
  selector:
    matchLabels:
      app: ledger
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/payments/sa/checkout"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charge"]

Two objects, two jobs. The PeerAuthentication sets mTLS mode to STRICT, which means the server side refuses any plaintext connection instead of quietly accepting it. That refusal is the thing that actually enforces an encrypted data plane. But encryption and identity are only half the story. mTLS proves the caller is checkout. It says nothing about whether checkout is allowed to call ledger. Proving who you are is authentication; deciding what you may do is authorization, and they are separate controls. The AuthorizationPolicy is the authorization half: it selects the ledger workload and permits a request only when the source identity is checkout and the call is a POST to /v1/charge. One rule of Istio to burn into memory: the moment any ALLOW policy attaches to a workload, everything that does not match is denied by default. Before this policy, every meshed identity could reach the ledger. After it, only checkout on that one path can.

A manifest is a statement of intent, not proof. Verify what the sidecar is actually enforcing on the running pod.

$ kubectl apply -f payments-mtls.yaml

peerauthentication.security.istio.io/default created
authorizationpolicy.security.istio.io/ledger-allow-checkout created

# What mTLS mode is really in effect on the pod?
$ istioctl experimental describe pod checkout-6b9f4c8d7-abcde -n payments
 
# And the live cert the proxy is serving, with its short lifetime:
$ istioctl proxy-config secret checkout-6b9f4c8d7-abcde -n payments

Pod: checkout-6b9f4c8d7-abcde
   Pod Revision: default
   Pod Ports: 8080 (checkout), 15090 (istio-proxy)
--------------------
Service: checkout
   Port: http 8080/HTTP targets pod port 8080
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.payments
 
RESOURCE NAME     TYPE           VALID CERT     SERIAL NUMBER      NOT AFTER                NOT BEFORE
default           Cert Chain     true           2f1a...c9          2026-07-22T14:03:11Z     2026-07-21T14:03:11Z
ROOTCA            CA             true           0a44...11          2034-06-01T00:00:00Z     2024-06-01T00:00:00Z

# Prove it from a pod with NO sidecar (the 'default' namespace is not injected):
# a plaintext client should be rejected outright.
$ kubectl run probe --image=curlimages/curl -n default --restart=Never -it --rm -- \
    curl -sS http://ledger.payments.svc.cluster.local:8080/v1/charge

curl: (56) Recv failure: Connection reset by peer
pod "probe" deleted
command terminated with exit code 56

The reset is the point. Under STRICT the ledger's sidecar will not even start a plaintext conversation, so a caller with no mesh identity gets its connection torn down before it sends a request. That is the encrypted data plane doing its job, live, in front of you.

Who Runs The Control Plane On Each Cloud

The Envoy sidecars and the Istio config behave the same on every cloud. What differs is who keeps the control plane alive: who patches Istiod (the Istio control-plane process that distributes config and signs certs), who rotates the CA root, who upgrades the sidecars. Running that yourself on one cluster is a chore. Running it on three clusters across three clouds is a second job. Google and Microsoft will do it for you. Amazon will not, and its old managed option is on the way out.

On GKE, Cloud Service Mesh (formerly Anthos Service Mesh) gives you a Google-managed control plane and CA. You enable it on the fleet and set management to automatic, so both the sidecars and the certificates upgrade themselves. If you want to hold the trust root yourself, you can back that managed CA with Google's Certificate Authority Service instead of the default mesh CA.

$ gcloud container fleet mesh enable --project my-prod
$ gcloud container fleet mesh update \
    --management automatic \
    --memberships prod-cluster \
    --location us-central1 \
    --project my-prod
$ gcloud container fleet mesh describe --project my-prod

Waiting for Feature Service Mesh to be created...done.
Waiting for Feature Service Mesh to be updated...done.
membershipStates:
  projects/812.../locations/us-central1/memberships/prod-cluster:
    servicemesh:
      controlPlaneManagement:
        details:
        - code: REVISION_READY
          details: 'Ready: asm-managed'
        state: ACTIVE
      dataPlaneManagement:
        state: ACTIVE
    state:
      code: OK
      description: 'Revision(s) ready for use: asm-managed.'

On AKS, the Istio-based service mesh add-on is a Microsoft-supported build of upstream Istio. You enable it on the cluster, and Azure owns the revision lifecycle and the mesh CA (you can back that CA with your own root held in Azure Key Vault for a trust chain you control). Revisions are how you upgrade one step at a time, canary style: get the available revisions, run the new one alongside the old, move workloads over, then retire the old.

$ az aks mesh enable --resource-group prod-rg --name prod-aks
$ az aks mesh get-revisions --location eastus -o table

# (managed-cluster JSON trimmed)
{
  "provisioningState": "Succeeded",
  "serviceMeshProfile": {
    "mode": "Istio",
    "istio": {
      "revisions": [ "asm-1-24" ]
    }
  }
}
 
Revision    Upgrades    Compatible K8s versions
----------  ----------  -----------------------
asm-1-24    asm-1-25    1.29, 1.30, 1.31
asm-1-25    asm-1-26    1.30, 1.31, 1.32

AWS is the outlier. App Mesh, its own Envoy control plane, is deprecated and reaches end of support on September 30, 2026, so anything new should not be built on it. There is no managed Istio on EKS either. That means on AWS you install and then own the upstream control plane yourself, including Istiod upgrades and CA rotation, which is exactly the operational weight the other two clouds lift off you. Price that in when you compare.

# AWS has no managed Istio; App Mesh is deprecated (end of support 2026-09-30).
# On EKS you install and own the upstream control plane:
$ kubectl config use-context prod-eks
$ istioctl install --set profile=default -y

Switched to context "prod-eks".
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete
Made this installation the default for cluster-wide operations.

One wrinkle the multi-cloud setup forces you to face: identity does not cross clouds by itself. Two companies whose front-desk guards will not honor each other's ID badges have to get their security offices to agree on a shared signing stamp first. Meshes work the same way. Each cluster signs certs under its own trust domain (the cluster.local part of the SPIFFE ID, which you should rename to something unique per cluster in production). A sidecar in GKE will not trust a cert signed by the AKS mesh CA until you tell it to. To let a service in one cloud call a service in another over mTLS, you federate the trust domains: either issue every cluster an intermediate CA under one shared root you hold in a cloud KMS (Key Management Service), or exchange the root CA bundles between meshes so each side can validate the other. Skip that step and your cross-cloud calls fail the handshake, which is the mesh working correctly, not a bug.

Harden The North-South Edge, Everywhere

The mesh handles east-west. The front door still needs a modern lock. If the load balancer will still speak TLS 1.0 or accept weak ciphers, that becomes your softest target regardless of how strict the interior is. The control is the same on every cloud: require TLS 1.2 as the floor, offer TLS 1.3 where the client supports it, and drop the legacy cipher suites. Only the spelling changes.

$ aws elbv2 modify-listener \
    --listener-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:listener/app/prod-alb/50dc.../f2f7... \
    --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06

{
    "Listeners": [
        {
            "ListenerArn": "arn:aws:elasticloadbalancing:us-east-1:111122223333:listener/app/prod-alb/50dc.../f2f7...",
            "Port": 443,
            "Protocol": "HTTPS",
            "SslPolicy": "ELBSecurityPolicy-TLS13-1-2-2021-06",
            "Certificates": [
                { "CertificateArn": "arn:aws:acm:us-east-1:111122223333:certificate/abcd-1234" }
            ]
        }
    ]
}

# GCP: the RESTRICTED profile keeps only strong ciphers and requires TLS 1.2+.
# (--min-tls-version sets the FLOOR at 1.2; TLS 1.3 is still negotiated with capable clients.)
$ gcloud compute ssl-policies create prod-tls \
    --profile RESTRICTED --min-tls-version 1.2
$ gcloud compute target-https-proxies update prod-https-proxy \
    --ssl-policy prod-tls

Creating SSL policy...done.
Created [https://www.googleapis.com/compute/v1/projects/my-prod/global/sslPolicies/prod-tls].
Updating [prod-https-proxy]...done.
Updated [https://www.googleapis.com/compute/v1/projects/my-prod/global/targetHttpsProxies/prod-https-proxy].

# Azure: a strong predefined policy on Application Gateway (TLS 1.2 floor, TLS 1.3 supported).
$ az network application-gateway ssl-policy set \
    --resource-group prod-rg --gateway-name prod-agw \
    --policy-type Predefined --policy-name AppGwSslPolicy20220101S
$ az network application-gateway ssl-policy show \
    -g prod-rg --gateway-name prod-agw \
    --query "{Policy:policyName, Type:policyType, MinTLS:minProtocolVersion}" -o table

Policy                   Type        MinTLS
-----------------------  ----------  --------
AppGwSslPolicy20220101S  Predefined  TLSv1_2

Three commands, one idea. AWS names its policy, GCP picks a profile plus a minimum version, Azure selects a predefined policy. All three land in the same place: a front door that turns away obsolete protocols before a request ever reaches the mesh.

The Bill For All This

None of this is free, and the trade-off is real enough to plan around. Every sidecar is one more proxy hop on the wire: usually a few milliseconds of added latency and tens of megabytes of memory per pod, plus the control-plane compute and the standing cost of running certificate infrastructure. At a handful of pods you will not notice. At several thousand it is measurable money and a fatter p99 (the latency your slowest 1 percent of requests see). Istio's ambient mode is the current answer. Instead of a translator at every desk, it posts one shared interpreter per floor: a per-node proxy called ztunnel (zero-trust tunnel) that carries mTLS for every pod on that node. Dropping the per-pod sidecar cuts overhead at high density, and gives up some workload isolation in return, since one ztunnel now handles many pods' traffic. Whatever you pick, roll it out by blast radius. Turn STRICT on for the namespaces that would hurt most if they leaked (payments, secrets, anything holding personal data), watch the latency histogram on that one namespace for a few days, and let that evidence set the pace for the next. Flip the whole cluster in a single change and the first latency page will get blamed on the mesh, fairly or not.

Try this

Run kubectl apply -f payments-mtls.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an ALLOW Policy With No Rules Denies Everything. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related