The zero-trust model

Never trust, always verify; assume breach.

Old office buildings run on one guard and one badge. You show a card at the front desk in the morning, and after that the building is yours: every floor, every corridor, every drawer somebody left open. Nobody checks again, because getting past the lobby was the check. That is the perimeter model, and a great many corporate networks still work exactly this way. Hard shell, soft trusting middle.

The trouble is arithmetic. An attacker has to get past the lobby once. You have to stop them every time. One phished laptop, one exposed pod, one leaked VPN (virtual private network) credential, and everything the interior trusts by default becomes reachable from that single foothold. Cloud and Kubernetes made the shell thinner and the interior far bigger. Workloads spread across clusters and providers, so there is no single edge left to guard, and pod IP (Internet Protocol) addresses churn every time something reschedules, so the addresses you carefully typed into firewall rules stop meaning anything by Tuesday.

Zero trust rebuilds the building so every internal door re-checks your badge, the badge opens only the rooms your job needs, and it quietly expires every hour. Reaching the lobby now buys you nothing. Written as engineering commitments, that is three things: never trust, always verify; assume breach; verify continuously. This lesson takes each one and shows what enforcing it looks like on the wire.

Never trust, always verify

Implicit trust is the thing zero trust deletes. In the perimeter model your location stands in for your identity: being on the corporate LAN (local area network, the office wiring), or inside the cluster's pod network, counts as evidence that you belong. Zero trust cuts that wire. Every request is authenticated and authorized on its own merits, whatever address it arrived from.

Two words get used interchangeably and must not be. Authentication (authn, proving who you are, like showing an ID at the door) establishes an identity. Authorization (authz, deciding what that identity is allowed to do, like which floors the badge actually opens) is a separate decision made afterwards. The perimeter model skipped both for internal traffic. Zero trust demands both, on every request. Google's BeyondCorp was the landmark version of this for humans: it removed the privileged corporate network entirely, so access depends on a verified user and a verified device rather than on being "on the VPN". Workloads get the same treatment. A service proves who it is with cryptography, and the network it happens to sit on grants it nothing.

"Identity is the new perimeter" means the receiver stops asking what address is this from and starts asking what signed name did this connection present. You can watch that happen. Calls inside a service mesh already travel over mTLS. A service mesh is a layer of small proxy containers, one running beside each of your pods, that every packet in and out has to pass through. mTLS is mutual TLS: TLS (Transport Layer Security) is the padlock behind HTTPS, and "mutual" means both ends present a certificate, not only the server. Connect by hand the same way and read what the peer actually proved.

$ openssl s_client -connect payments.prod.svc.cluster.local:8443 \
    -cert svid.pem -key key.pem -CAfile bundle.pem </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -ext subjectAltName -dates

subject=C = US, O = SPIRE, x500UniqueIdentifier = 3f6b0c9d2e5a814f7b03d6c1a9e4f28b
issuer=C = US, O = SPIFFE
X509v3 Subject Alternative Name:
    URI:spiffe://acme.internal/ns/prod/sa/payments
notBefore=Jul 21 09:14:02 2026 GMT
notAfter=Jul 21 10:14:02 2026 GMT

The name lives in the certificate's SAN (Subject Alternative Name) field, written as a URI (Uniform Resource Identifier, the same shape as a web address): spiffe://acme.internal/ns/prod/sa/payments. Not an address. The Subject line above it holds a unique but deliberately meaningless string that SPIFFE verifiers ignore, and the source IP plays no part in the decision at all. What the verifier checks is the signature chain leading back to a root certificate in bundle.pem. Look at the validity window too. About an hour, not a year.

What the handshake proves, and what it does not

A courier turns up at reception wearing a company badge. The badge proves this person works for that courier firm and that the firm issued it. It tells you nothing about whether the parcel is safe, who ordered it, or whether the courier should be allowed into the server room on the fourth floor.

Be exact about mTLS, because fuzzy thinking here produces real incidents. A completed handshake proves that the process at the other end of this TCP (Transmission Control Protocol) connection holds the private key matching a certificate that chains to a root you trust, and that the certificate names one particular SPIFFE ID. SPIFFE, the Secure Production Identity Framework For Everyone, is the open standard that defines those names. Per connection. Per hop. That is the whole claim.

It does not prove the workload is behaving. Get code execution inside the payments pod and your traffic carries the payments identity, correctly signed, indistinguishable from the real thing. It does not prove which end user a call is for: the peer identity names a service, never a human, so a captured or buggy service can act for any customer it can name. That is the classic confused deputy problem, where a trusted component gets talked into misusing privileges it legitimately holds. It does not prove the request is allowed, because authentication has finished and authorization has not started. And it proves nothing past a hop that terminates TLS. A layer 7 gateway (layer 7 meaning it reads the HTTP (HyperText Transfer Protocol) request rather than forwarding raw bytes) decrypts and then opens a fresh connection of its own, so the service behind it sees the gateway's identity and not the original caller's. Carrying a user across that boundary is a separate job, done with [request-level tokens](/courses/zerotrust/zt-l7/).

There is a fifth limit, and it lives outside the mesh entirely, which is exactly why people miss it. A workload identity is bound to selectors: facts about the running process that the identity system checks before it hands over a certificate. In Kubernetes the usual selectors are the namespace and the ServiceAccount, the in-cluster identity a pod runs as. You can read them straight out of the registration entry.

$ kubectl exec -n spire spire-server-0 -c spire-server -- \
    /opt/spire/bin/spire-server entry show \
    -spiffeID spiffe://acme.internal/ns/prod/sa/payments

Found 1 entry
 
Entry ID         : 4b1a9f30-6c72-4b0e-9a3f-2d5e8c71a0b4
SPIFFE ID        : spiffe://acme.internal/ns/prod/sa/payments
Parent ID        : spiffe://acme.internal/spire/agent/k8s_psat/prod-cluster/9c2f1d84
Revision         : 0
X509-SVID TTL    : 3600
JWT-SVID TTL     : default
Selector         : k8s:ns:prod
Selector         : k8s:sa:payments

Two selectors. That is the entire test. So anyone who can create a pod in prod with serviceAccountName: payments gets the payments identity, legitimately, out of your own identity system, and every policy naming that identity will happily agree. Kubernetes RBAC (role-based access control, the rules deciding who may do what in the cluster) is the real root of your workload identity. Find out who holds that power before you treat SPIFFE IDs downstream as gospel.

$ kubectl auth can-i create pods -n prod --as=system:serviceaccount:ci:deployer
$ kubectl auth can-i create pods -n prod --as=system:serviceaccount:prod:reporting
$ kubectl auth can-i --list -n prod --as=system:serviceaccount:ci:deployer \
  | grep '^pods '

yes
no
pods                     []      []      [create delete get list patch update watch]

Making identity mandatory

A door that accepts a badge or a friendly nod is a door that accepts a friendly nod. Proving identity is worth nothing while presenting one stays optional. Istio's PeerAuthentication resource is the setting that decides which of those two doors you have. PERMISSIVE accepts mTLS and plaintext both, which is what you need during a migration. STRICT refuses any connection arriving without a client certificate, at the transport layer, before your application sees a single byte. Scope comes from where the object lives: created in prod it covers that namespace, created in the mesh root namespace (istio-system by default) it covers everything at once.

# security.istio.io/v1 is the current API version. Older blog posts say
# v1beta1, which is the same object under a previous version label.
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod          # namespace scope; in istio-system this is mesh-wide
spec:
  mtls:
    mode: STRICT           # PERMISSIVE while migrating, STRICT once proven

$ kubectl apply -f prod-peerauth.yaml
$ istioctl x describe pod payments-7d9f8c6b5-nq2vt -n prod

peerauthentication.security.istio.io/default created
Pod: payments-7d9f8c6b5-nq2vt
   Pod Revision: default
   Pod Ports: 8080 (payments), 15090 (istio-proxy)
--------------------
Service: payments
   Port: http 8080/HTTP targets pod port 8080
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.prod

Effective PeerAuthentication is the line that matters. Istio merges mesh-wide, namespace-level and workload-level policies plus any per-port overrides, so what you wrote is not reliably what applies, and Applied PeerAuthentication names the object that won the merge. Note what this command does not tell you: it reports mTLS mode only, never your authorization rules.

Do not flip STRICT because the manifest applied cleanly. Flip it because telemetry says nothing is still arriving in plaintext. Every proxy tags its request counters with a connection_security_policy label. On the receiving side the value is mutual_tls or none. On the sending side it is always unknown, because a client cannot see what the far end enforced, which is why the query below filters on reporter="destination". The receiving proxy is the only one that can tell you who is still calling you unauthenticated.

$ kubectl exec -n prod deploy/payments -c istio-proxy -- \
    pilot-agent request GET stats/prometheus \
  | grep '^istio_requests_total.*reporter="destination"' \
  | sed -E 's/.*source_workload="([^"]*)".*connection_security_policy="([^"]*)".*} ([0-9]+)/\1 \2 \3/'
 
$ kubectl get pods -A -o json | jq -r '
    .items[]
    | select(.metadata.namespace | test("^(kube-|istio-|spire)") | not)
    | select([.spec.containers[].name] | index("istio-proxy") | not)
    | "\(.metadata.namespace)/\(.metadata.name)"'

frontend mutual_tls 41273
frontend mutual_tls 118
checkout mutual_tls 8891
unknown none 1204
 
batch/nightly-recon-28912340-h7k9q
monitoring/blackbox-exporter-6c4f7d9b8-xw2rp

frontend mutual_tls 41273
frontend mutual_tls 118
checkout mutual_tls 8891
unknown none 1204
 
batch/nightly-recon-28912340-h7k9q
monitoring/blackbox-exporter-6c4f7d9b8-xw2rp

Those are raw counters and there is one row per label combination, so the same caller appears more than once (here frontend split across two response codes) and you add its rows together. The last row is the reason to run this at all. 1,204 requests arrived with no client certificate, from a caller the mesh cannot even name, because it has no proxy beside it and so has no identity to present. Turn on STRICT now and those 1,204 requests become connection resets on somebody's dashboard at three in the morning. The second command hunts the usual suspects: pods with no istio-proxy container, which in most clusters means a batch job nobody labelled and a monitoring probe.

Assume breach

Ships do not stay afloat by having a hull that never fails. They stay afloat because the hull is divided into sealed compartments, so a hole floods one of them instead of the whole vessel. The second commitment is that posture. Build as though somebody is already inside, because one day somebody will be. The question stops being how do I keep them out and becomes how little can they reach once they are in. Everything reachable from one compromised workload is its blast radius, and shrinking that number is the work.

Measure it before you change anything. Pick a pod that a poisoned dependency could plausibly own, and see what answers it.

$ kubectl exec -n prod deploy/frontend -c frontend -- sh -c '
    for s in payments ledger reporting; do
      printf "%-11s %s\n" "$s" "$(curl -s -o /dev/null -w %{http_code} -m 2 \
        http://$s.prod.svc.cluster.local:8080/v1/status)"
    done'

payments    200
ledger      200
reporting   200

Three for three, and in a real cluster it is usually thirty for thirty. Notice that mTLS did nothing for you here. Every one of those calls was authenticated and encrypted, and every one was permitted, because nothing was checking authorization. Encryption without authorization hands an attacker a private, tamper-proof channel to everything.

A guest list works the opposite way round from a doorman turning away people he dislikes. Nobody gets in unless they are written down. Default-deny is the guest list. An Istio AuthorizationPolicy with an empty spec selects every workload in its namespace and matches no request at all: the default action is ALLOW, and an ALLOW policy that matches nothing permits nothing. Then you write the list back one declared call path at a time, against identity instead of address.

# 1. Deny everything in the namespace. Empty spec = selects every workload,
#    matches no request, so the default ALLOW action permits nothing.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: default-deny
  namespace: prod
spec: {}
---
# 2. Add back one declared call path, scoped to a method and a route.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-callers
  namespace: prod
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            # No "spiffe://" scheme here. Istio expects
            # <trust-domain>/ns/<namespace>/sa/<serviceaccount>
            principals: ["acme.internal/ns/prod/sa/frontend"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charge"]

$ kubectl apply -f prod-authz.yaml
$ kubectl exec -n prod deploy/frontend -c frontend -- sh -c '
    for s in payments ledger reporting; do
      printf "%-11s %s\n" "$s" "$(curl -s -o /dev/null -w %{http_code} -m 2 \
        http://$s.prod.svc.cluster.local:8080/v1/status)"
    done'
$ kubectl exec -n prod deploy/frontend -c frontend -- \
    curl -s -o /dev/null -w '%{http_code}\n' -X POST \
    http://payments.prod.svc.cluster.local:8080/v1/charge

authorizationpolicy.security.istio.io/default-deny created
authorizationpolicy.security.istio.io/payments-callers created
payments    403
ledger      403
reporting   403
200

Read that result carefully. frontend holds a valid, current SVID (SPIFFE Verifiable Identity Document, the certificate form of a SPIFFE ID) and the handshake succeeded every single time. It still got HTTP 403 Forbidden on /v1/status, because the only rule naming it permits POST on /v1/charge and nothing else. Authentication proved who was calling. Authorization declined anyway. A compromised frontend now reaches one method on one path on one service, instead of an entire namespace.

Two details worth keeping. The 403 comes from the destination's proxy, not the caller's, so that is where you go looking when a call fails and you cannot see why. And Istio evaluates CUSTOM policies first, then every DENY policy, then ALLOW: if no ALLOW policy selects a workload at all, everything is permitted. That is precisely why the empty-spec policy exists. It guarantees at least one ALLOW policy always selects every pod in the namespace, so a missing rule turns into a refusal rather than a free pass. Building that list out of real traffic is its own discipline, covered in [identity-based authz](/courses/zerotrust/zt-authz/).

Trust with an expiry date

The third commitment goes after a quieter assumption: that trust, once granted, stays good. A gym membership works for a year. A cinema ticket works for one showing. Traditional systems hand out gym memberships, authenticating you at login and then issuing a session or a certificate valid for months. Zero trust hands out cinema tickets.

Two mechanisms carry that. Credentials are short-lived and rotated automatically: the SVID you inspected earlier had a one-hour window, and the identity infrastructure replaces it in the background well before it lapses, so a certificate an attacker copies out of a pod stops working within the hour and there is no long-lived secret sitting on disk to steal in the first place. It is also why SPIFFE ships no revocation machinery, no CRL (certificate revocation list) and no OCSP (Online Certificate Status Protocol) responder: nothing needs revoking when everything expires on its own. Price in the honest cost before you commit. A signing CA (certificate authority, the service that issues and vouches for certificates) serving thousands of workloads that all rotate hourly becomes a hard runtime dependency with a capacity ceiling, and clock skew between nodes turns from an annoyance into an outage.

The second mechanism is per-request evaluation. Authorization is checked on every call rather than once per session, so a policy you change now takes effect on the next request instead of whenever some long-lived token happens to expire.

$ kubectl delete authorizationpolicy payments-callers -n prod
$ kubectl exec -n prod deploy/frontend -c frontend -- \
    curl -s -o /dev/null -w '%{http_code}\n' -X POST \
    http://payments.prod.svc.cluster.local:8080/v1/charge

authorizationpolicy.security.istio.io "payments-callers" deleted
403

No restart, no reconnection, no waiting for anything to time out. The TCP connection stayed open the whole time and the next request over it was refused, roughly a second after the config push reached the proxy. Compare that with a firewall rule or a session cookie, where a decision you revoked at noon can stay effectively true until the evening.

A staged rollout, not a product

No box on a vendor price list makes you zero trust. It is a set of properties enforced across identity, network and policy, and the order you build them in is forced by dependency rather than preference. You cannot write identity-based authorization until workloads have identities they can prove. You cannot safely default-deny until you have evidence that mTLS already covers every caller. Each stage is a place to stop, measure, and be honest about what is still speaking plaintext.

Everything on that path rests on the first stage, and the first stage is the one people skip. Each workload needs a name it can prove, one that survives rescheduling and has nothing to do with the address it holds this minute. If you want one number to start from today, run that connection_security_policy query against your busiest namespace and count the requests coming back none. That count is your zero-trust backlog, and it is almost always bigger than anyone expects. The [next lesson](/courses/zerotrust/zt-workload-id/) builds the name that makes the rest of it possible.

Try this

Run kubectl auth can-i create pods -n prod --as=system:serviceaccount:ci:deployer on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: whoever can create pods can mint the identity. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related