Workload identity, not IP

Cryptographic identity that follows the workload.

Write any return address you like on an envelope. The post office still delivers it, and whoever opens it has no way to check whether you were telling the truth. A source IP address (Internet Protocol address, the numeric label a packet claims to have come from) works the same way: a claim, accepted on faith by everything downstream. Workload zero trust starts by demoting that claim and replacing it with something the sender has to prove. A workload (any running unit of software: a pod, a container, a virtual machine, a serverless function) gets a cryptographic identity it can produce on demand, the way you hand over a passport at a border. Encryption, authorization, audit trails and cross-cluster trust then all key off that identity instead of off a location.

Why an address makes a bad credential

Three separate things go wrong when you authorize on an address, and they break in different ways. It moves. Pods reschedule on every deploy, node drain, autoscale event and crash loop. The address a service holds this morning belongs to something unrelated this afternoon. It is shared. Traffic leaving a node behind source NAT (Network Address Translation, where the kernel rewrites the source address on the way out) arrives wearing the node's address, so forty pods look identical to the receiver, and a cloud NAT gateway collapses a whole subnet the same way. Its honesty is somebody else's decision. Whether a pod can forge a source address depends on your CNI (Container Network Interface, the plugin that wires up pod networking) and how somebody configured it. Some enforce source addresses. Some do not. You should not have to know, and you should not stake authorization on a property whose trustworthiness lives in a config file written by another team.

kubectl -n prod get pods \
  -o custom-columns='NAME:.metadata.name,IP:.status.podIP,SA:.spec.serviceAccountName'
 
kubectl -n prod rollout restart deploy/payments
kubectl -n prod rollout status deploy/payments
 
kubectl -n prod get pods \
  -o custom-columns='NAME:.metadata.name,IP:.status.podIP,SA:.spec.serviceAccountName'

NAME                        IP            SA
checkout-5b8d7c9f4-qm2vn    10.42.3.118   checkout
payments-7c9f6d84b7-2xk4p   10.42.3.117   payments
 
deployment.apps/payments restarted
 
Waiting for deployment "payments" rollout to finish: 1 old replicas are pending termination...
deployment "payments" successfully rolled out
 
NAME                        IP            SA
checkout-5b8d7c9f4-qm2vn    10.42.3.118   checkout
payments-6d4b8f5c9d-lz7wt   10.42.3.121   payments

The service account did not change. The address did. A rule naming 10.42.3.117 is now wrong in the more dangerous direction. Today it blocks the real payments service. Later, when the cluster recycles that address onto some other pod, it silently authorizes a stranger. A rule that says "callers proven to hold the payments identity" survived the restart with nobody touching it.

What the identity is made of

A passport has two parts that people tend to run together: the printed name, and the physical booklet with its signatures and holograms. Workload identity splits the same way. The name is a SPIFFE ID (Secure Production Identity Framework For Everyone, an open standard for naming workloads). It is a URI (Uniform Resource Identifier, the same kind of string as a web address): spiffe://acme.internal/ns/prod/sa/payments. The host portion, acme.internal, is the trust domain, the boundary of one issuing authority, the way a passport belongs to one country. The path says what the workload is, here the payments service account in the prod namespace.

The booklet is an SVID (SPIFFE Verifiable Identity Document), the signed artifact that proves the name. It comes in two shapes. An X.509-SVID is a short-lived certificate carrying the SPIFFE ID, used to authenticate a TLS connection (Transport Layer Security, the encryption behind every https:// page). A JWT-SVID (JSON Web Token, a small signed blob of claims you can put in a header) is a short-lived token you attach to a single request, useful when something in the path terminates TLS and the connection identity is lost. This lesson uses the X.509 form. SPIFFE defines the naming and the documents. SPIRE (the SPIFFE Runtime Environment) is the reference implementation that decides which workload deserves which name, and does the signing.

You register properties, never addresses

A company badge office does not print a badge for whoever asks nicely. It works off a rule: anyone whose personnel record says payments team, building four, gets a badge that says payments. A SPIRE registration entry is that rule, written down. The properties are called selectors: facts about the running workload that the platform can confirm on its own, such as its namespace, its Kubernetes service account, the name of the container, or the exact image it was started from. Every selector on an entry has to match, because they are ANDed together rather than ORed. None of them can be an address.

# a node alias: one name standing in for "any agent that proved it belongs to this cluster"
spire-server entry create \
  -node \
  -spiffeID spiffe://acme.internal/nodes/prod-cluster \
  -selector k8s_psat:cluster:prod-cluster
 
# the workload entry, parented to that alias
spire-server entry create \
  -parentID spiffe://acme.internal/nodes/prod-cluster \
  -spiffeID spiffe://acme.internal/ns/prod/sa/payments \
  -selector k8s:ns:prod \
  -selector k8s:sa:payments \
  -selector k8s:container-name:payments \
  -x509SVIDTTL 1800

Entry ID                : 0c1a7f43-9b25-4e6d-a0f8-51c2d9e73b40
SPIFFE ID               : spiffe://acme.internal/nodes/prod-cluster
Parent ID               : spiffe://acme.internal/spire/server
Revision                : 0
X509-SVID TTL           : default
JWT-SVID TTL            : default
Selector                : k8s_psat:cluster:prod-cluster
 
Entry ID                : 5f8c1e2a-6b0d-4a9f-8c11-2e7d4b9a0f31
SPIFFE ID               : spiffe://acme.internal/ns/prod/sa/payments
Parent ID               : spiffe://acme.internal/nodes/prod-cluster
Revision                : 0
X509-SVID TTL           : 1800
JWT-SVID TTL            : default
Selector                : k8s:ns:prod
Selector                : k8s:sa:payments
Selector                : k8s:container-name:payments

Read the parent chain from the top. The first entry is a node alias, and its own parent is the SPIRE server, because nothing vouches for it except the server's own attestation rules. Its selector is a node fact: k8s_psat is the node attestor that proves a node's agent using a Kubernetes projected service account token, so every agent underneath the alias earned its own identity before it was trusted to vouch for anything else. The second entry hangs the payments workload off that alias, which is why the identity keeps working when the pod lands on a different node. -x509SVIDTTL takes seconds, so this entry issues thirty-minute certificates. Two things to know about that number. SPIRE will not issue an SVID that outlives the certificate authority signing it, so a generous SVID lifetime under a short ca_ttl gets quietly clipped. And a short lifetime buys you the bill at the end of this lesson. If you want identity bound to a specific build rather than to a service account, add -selector k8s:container-image:ghcr.io/acme/payments@sha256:..., and a container running any other image is refused the identity even with the right service account.

How the agent knows who is asking

A bouncer who asks your name learns nothing. A bouncer who reads the wristband already on your arm learns something you cannot edit. The SPIRE agent is the second kind. Every node runs a spire-agent, and workloads reach it over a Unix domain socket (a file on disk that behaves like a network socket, usable only by processes on the same machine). When your process connects, the kernel hands the agent the caller's process ID, user ID and group ID. Your code does not supply those values and cannot forge them. The agent takes the process ID, follows it to the container through the process's cgroup (control group, the kernel's per-container accounting bucket), asks the kubelet (the Kubernetes agent running on every node) which pod owns that container, and turns the answer into selectors. Then it finds the registration entry whose selectors all match and issues an SVID for that entry's name. The request your workload sent carries no name, no token and no address. There is nothing in it to lie about.

spire-agent api fetch x509 \
  -socketPath /run/spire/agent-sockets/spire-agent.sock -write .
 
openssl x509 -in svid.0.pem -noout -subject -dates -ext subjectAltName

Received 1 svid after 6.204852ms
 
SPIFFE ID:              spiffe://acme.internal/ns/prod/sa/payments
SVID Valid After:       2026-07-21 09:00:12 +0000 UTC
SVID Valid Until:       2026-07-21 09:30:12 +0000 UTC
CA #1 Valid After:      2026-07-21 08:41:03 +0000 UTC
CA #1 Valid Until:      2026-07-22 08:41:03 +0000 UTC
 
Writing SVID #0 to file svid.0.pem.
Writing key #0 to file svid.0.key.
Writing bundle #0 to file bundle.0.pem.
 
subject=C = US, O = SPIRE
notBefore=Jul 21 09:00:12 2026 GMT
notAfter=Jul 21 09:30:12 2026 GMT
X509v3 Subject Alternative Name:
    URI:spiffe://acme.internal/ns/prod/sa/payments

Three details there matter more than they look. There is no Common Name. SPIRE's default subject for a leaf certificate is the generic C=US, O=SPIRE, which names nothing in particular, because SPIFFE puts identity in the URI SAN (Subject Alternative Name, the certificate extension listing the names a certificate speaks for). SPIRE fills in a Common Name only when the registration entry declares DNS names with -dns, and this entry did not. A legacy service that authorizes on the Common Name, and plenty still do, is comparing its allowlist against an empty field. The certificate lives thirty minutes because the entry asked for it. SPIRE's own defaults are one hour for X.509-SVIDs (default_x509_svid_ttl) and five minutes for JWT-SVIDs (default_jwt_svid_ttl), and the agent renews at roughly half the lifetime, so nothing waits for expiry. A stolen X.509-SVID is scrap by the end of a coffee break. The private key never travelled. The agent generated it on the node. There is no key material to steal out of your CI system because none was ever put there.

Verify the chain, then read the name

Border control checks the booklet against a reference book of genuine designs before anybody types the name into a computer. Verification here is the same two steps in the same fixed order. Confirm the certificate chain against the trust bundle (the set of certificate authority certificates for the trust domain, distributed to every verifier), then read the SPIFFE ID out of the URI SAN and compare it with what you expected. Doing those in the other order, or skipping the first, is the entire vulnerability. This is also where SPIFFE parts company with the TLS your browser does. A browser checks that the hostname you typed appears in the certificate. A SPIFFE certificate normally carries no DNS name at all, so hostname verification has nothing to work with, and the name check happens against the URI SAN instead.

openssl s_client -connect payments.prod.svc.cluster.local:8443 \
  -cert svid.0.pem -key svid.0.key -CAfile bundle.0.pem \
  -verify_return_error </dev/null 2>/dev/null \
  | tee peer.txt | grep 'Verify return code'
 
# openssl skips the surrounding text and reads the first PEM block in the file
openssl x509 -in peer.txt -noout -ext subjectAltName

Verify return code: 0 (ok)
 
X509v3 Subject Alternative Name:
    URI:spiffe://acme.internal/ns/prod/sa/payments

Nothing in that exchange asked where the peer was. Reschedule it onto a different node in a different subnet and run the same two commands. The answer is identical.

The mesh runs that handshake for every call

Hand-wiring handshakes stops being practical somewhere past the demo. A service mesh gives every workload its own receptionist: a proxy (Envoy) parked beside the application, doing the TLS on both ends of every connection, so your code keeps speaking plain HTTP and never touches a certificate. In a sidecar deployment that proxy is a second container inside the same pod. Istio runs its own certificate authority inside istiod and issues certificates in exactly the SPIFFE shape, in the default trust domain cluster.local. You can read what a running sidecar is holding right now.

istioctl proxy-config secret deploy/payments -n prod
 
istioctl proxy-config secret deploy/payments -n prod -o json \
  | jq -r '.dynamicActiveSecrets[]
           | select(.name == "default")
           | .secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 -d | openssl x509 -noout -ext subjectAltName

RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER              NOT BEFORE
default           Cert Chain     ACTIVE     true           9a3f5c1e77b24d0f8e6a2b4c1d5e7f30     2026-07-22T09:03:41Z   2026-07-21T09:03:41Z
ROOTCA            CA             ACTIVE     true           c41b8e02fd9a476b5c3e10d8a97f2b64     2036-07-16T08:12:55Z   2026-07-19T08:12:55Z
 
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/prod/sa/payments

Same URI SAN, different trust domain. The two date columns on the leaf are twenty-four hours apart, which is Istio's default workload certificate lifetime, and the agent inside the sidecar rotates at half life. The root above it is good for ten years, which is the usual arrangement: a long-lived root signing very short-lived leaves. SPIRE's default hour is tighter than Istio's day. Both beat a certificate somebody minted in 2023 and pasted into a Secret.

What mTLS proves, and what it does not

mTLS (mutual Transport Layer Security, where both ends present certificates instead of only the server, like a courier who shows you their badge before asking for yours) proves something narrow and exact: the party on the other end of *this connection* holds the private key for a certificate that chains to a certificate authority you trust, and that certificate names identity X. Strong, useful, and very easy to over-read.

It does not prove the request *originated* at X. Any hop that terminates TLS and opens a fresh connection presents its own identity, so what you verify at a gateway or an egress proxy is the hop, not the caller standing behind it. Carrying the original identity across that boundary takes a JWT-SVID or a header you have decided, deliberately, to trust. It does not prove *which process* is talking, and the floor is lower than people assume. Istio's default scheme derives identity from the Kubernetes service account, a pod-level fact, so every container in the pod speaks with the same name and so does anyone who can kubectl exec into it. SPIRE can go one level finer, because it attests the specific calling container: k8s:container-name and k8s:container-image really will give the sidecar and the application separate identities. Neither system gets below the container. Whoever can run a process inside it holds that identity. And mTLS does not prove X *may do* what it is asking. Authentication answers who. Authorization is a separate decision, made later, by something else.

Istio draws its own security boundary in the same place, and says so in its own documentation. A client cannot bypass *another* pod's sidecar, because the destination's proxy is what enforces inbound policy. It can bypass its own. The application and the proxy share a network namespace, so a compromised application can remove the redirect rules, or run as the proxy's own user ID, which those rules deliberately skip. Ambient mode moves the proxy out of the pod and onto the node, which relocates that boundary rather than deleting the question. Either way, NetworkPolicy (the Kubernetes object that filters pod-to-pod traffic at the network layer) stays in the design after the mesh lands.

Turning a proven identity into a decision

Two Istio objects finish the job, and they answer different questions. PeerAuthentication is the door: does an unauthenticated connection get through it at all. AuthorizationPolicy is the room list: which proven identity may call which endpoint.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod          # no selector, so it covers the whole namespace
spec:
  mtls:
    mode: STRICT           # reject plaintext: no proven identity, no conversation
                           # (with no PeerAuthentication at all, the default is PERMISSIVE,
                           #  which accepts plaintext and mTLS side by side)
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-checkout
  namespace: prod
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/checkout"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charge"]

A principal in Istio is the SPIFFE ID with the spiffe:// scheme stripped, trust domain included. principals only matches on connections that carried mTLS, because a plaintext request has no verified identity to compare against. That produces one of the sharper traps in the product. An ALLOW policy keyed on principals fails closed against a plaintext caller: no principal, no match, request denied. A DENY policy keyed on principals fails open: no principal, no match, so the rule you wrote to block someone never fires. PERMISSIVE mode is what makes that trap reachable, and it is the default until you say otherwise. Get PeerAuthentication to STRICT first, then trust either shape. The field you are retiring is source.ipBlocks. It still exists and still works, and it is still the wrong basis for a caller's credential.

# allowed caller: the checkout service account
kubectl -n prod exec deploy/checkout -c checkout -- \
  curl -s -o /dev/null -w '%{http_code}\n' -X POST http://payments:8080/v1/charge
 
# same namespace, same node, same subnet, different service account
kubectl -n prod exec deploy/reporting -c reporting -- \
  curl -s -w '\n%{http_code}\n' -X POST http://payments:8080/v1/charge
 
# an echo workload with no AuthorizationPolicy on it, to see what the sidecar
# tells an application about its peer
kubectl -n prod exec deploy/checkout -c checkout -- \
  curl -s http://echo:8080/headers | jq -r '.headers["X-Forwarded-Client-Cert"]'

200
 
RBAC: access denied
403
 
By=spiffe://cluster.local/ns/prod/sa/echo;Hash=3f9a1c7e50b2d84f6a1e9c03b7d25f81e4a60c9d3b18f7526ae0d4c91b3f8a27;Subject="";URI=spiffe://cluster.local/ns/prod/sa/checkout

Those two pods sit in the same namespace, on the same node, inside the same address range. Every IP-based control in the cluster treats them as one thing. The identity-based control does not, and the RBAC: access denied body (Role-Based Access Control, Envoy's name for the filter running the check) is how you confirm the policy is enforcing rather than merely loaded. One caution about that last header. Istio's inbound sidecar *appends* its verified element to whatever x-forwarded-client-cert already arrived instead of replacing it, because the default for meshConfig.defaultConfig.proxyHeaders.forwardedClientCert is APPEND_FORWARD. An application reading the first element is reading a value the caller typed. Read the last one, or set that field to SANITIZE_SET so the proxy overwrites the header with what it actually verified.

The bill that comes with it

You are now running issuance infrastructure, and it sits on the critical path. A certificate authority plus a per-node agent have to stay available, because when agents cannot issue or renew, workloads lose their identity and connections fail closed. Short lifetimes cut both ways: they make theft survivable and they make outages expensive. On a thirty-minute certificate, a control plane outage starts breaking production about half an hour in, as cached certificates expire one by one, and it keeps breaking until issuance comes back. Pick that number on purpose instead of inheriting it from a tutorial.

Rotation load is fleet size divided by the renewal interval, and renewal happens at half the lifetime rather than at expiry, which doubles the naive figure. Ten thousand workloads on thirty-minute certificates renew every fifteen minutes: roughly forty thousand signings an hour, plus ten thousand long-lived Workload API connections held open, because the API pushes fresh SVIDs down a stream that stays connected. Signing throughput and agent memory are capacity inputs, not footnotes. Identity is also not authorization and not encryption; it makes both possible and replaces neither. And IP controls do not retire. NetworkPolicy and firewall rules stay the right tool for coarse segmentation, for shrinking blast radius, and specifically for the gap where a compromised application container bypasses its own sidecar. Keep them. Stop treating them as the caller's credential.

The fastest way to find out how much of your cluster still decides by location is to ask it. Every policy that still names an address range is a rule an attacker satisfies by landing anywhere inside it.

kubectl get authorizationpolicies.security.istio.io -A -o json \
  | jq -r '.items[]
      | select([.. | (.ipBlocks?, .notIpBlocks?, .remoteIpBlocks?, .notRemoteIpBlocks?)
                   | select(. != null)] | length > 0)
      | "\(.metadata.namespace)/\(.metadata.name)"'

prod/payments-cidr-allow
prod/legacy-batch-range
staging/office-vpn-range

Work that list down. The next lesson, Why the perimeter fails, points the same argument outward at the firewall everyone still believes in. The local version of the fix is available today: every rule you move from ipBlocks to principals is one that can no longer be satisfied by landing in the right subnet.

Try this

Run kubectl -n prod rollout restart deploy/payments on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: reading the name without verifying the chain is worse than no check. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related