Continuous verification

Short-lived trust, posture, per-request checks.

Expert30 min · lesson 15 of 15

A keycard is standing trust. Somebody hands it to you once, it opens the door every morning until a human remembers to switch it off, and if you drop it in a car park, whoever picks it up walks in behind you. A conference day pass is the opposite. It carries today's date, it dies at midnight whether or not anyone lifts a finger, and revoking a lost one means declining to print tomorrow's. Continuous verification is the day pass model for machines: hand out trust on a short cadence, re-check it on every use, and let it lapse the moment issuance stops. Every control this course has built rests on that.

Revocation Is a Side Effect of Expiry

Each credential a workload holds carries a TTL (time to live, the validity window burned into the certificate when it is signed). SPIRE (the SPIFFE Runtime Environment, the server-and-agent pair that hands identity to machines; SPIFFE itself stands for Secure Production Identity Framework For Everyone) issues an SVID (SPIFFE Verifiable Identity Document, the short-lived certificate a workload presents to prove it is spiffe://acme.internal/ns/payments/sa/payments) that lives one hour by default. An Istio workload certificate defaults to twenty-four. Keep those windows short and revocation stops being a subsystem you operate. It turns into a property of the clock.

The alternative is machinery nobody enjoys running. A CRL (certificate revocation list, a signed file naming the serial numbers everyone must stop trusting) is the printed list of stolen cards taped up next to the till: useless until it reaches every till, and out of date the moment it arrives. OCSP (Online Certificate Status Protocol, a live lookup asking the issuer whether a certificate is still good) is phoning the bank on every swipe. It puts a network call in the handshake path and a responder in your on-call rotation. Both fail quietly in the same direction. Stale file, unreachable responder, verifier shrugs and continues. Expiry has no such failure mode, because nothing has to be delivered for a certificate to become worthless.

Read your own window before believing any of that. The Workload API is a hatch in the wall of the pod. It is a Unix socket where a workload asks the local SPIRE agent "who am I?" and gets back a freshly signed identity, with nothing written to disk unless you ask for it.

terminal
# Ask the local SPIRE agent for THIS workload's current SVID, and write
# out the PEM files (PEM is the base64 text wrapper certificates are
# normally stored in) so other tools can read them.
spire-agent api fetch x509 \
-socketPath /run/spire/agent-sockets/api.sock \
-write /tmp/svid
output
Received 1 svid after 6.312ms
SPIFFE ID: spiffe://acme.internal/ns/payments/sa/payments
SVID Valid After: 2026-07-21 09:15:03 +0000 UTC
SVID Valid Until: 2026-07-21 10:15:13 +0000 UTC
CA #1 Valid After: 2026-07-21 00:04:11 +0000 UTC
CA #1 Valid Until: 2026-07-22 00:04:21 +0000 UTC
Writing SVID #0 to file /tmp/svid/svid.0.pem.
Writing key #0 to file /tmp/svid/svid.0.key.
Writing bundle #0 to file /tmp/svid/bundle.0.pem.

Three numbers do the work there. The SVID covers an hour and ten seconds, because SPIRE backdates notBefore by ten seconds so a verifier whose clock runs slightly behind does not reject a certificate that was signed a moment ago. The CA (certificate authority, the key that signs everything beneath it) lives twenty-four hours, and nothing it signs can outlive it. And the identity is a URI (Uniform Resource Identifier, the same shape as a web address), not a hostname. Confirm that from the certificate itself rather than from the tool that handed it over, because the field a verifier actually reads is the SAN (Subject Alternative Name, the certificate extension where a SPIFFE ID lives).

terminal
openssl x509 -in /tmp/svid/svid.0.pem -noout -dates -ext subjectAltName
output
notBefore=Jul 21 09:15:03 2026 GMT
notAfter=Jul 21 10:15:13 2026 GMT
X509v3 Subject Alternative Name:
URI:spiffe://acme.internal/ns/payments/sa/payments

That hour is the whole revocation story, with one condition people skip. A certificate expires only if nobody replaces it. So stop staring at the certificate and look at what keeps minting them.

Cutting a Workload Off Means Stopping the Printer

Here is the mistake that turns a contained incident into a long one. A workload is confirmed compromised, somebody says "the certificates are short-lived, it times out within the hour", and nobody touches anything. It never times out. The agent renews the SVID before the old one dies, deliberately, so that healthy workloads never fall off the network. While the registration entry (the rule saying "any workload matching these selectors gets this SPIFFE ID") exists, the badge printer keeps printing.

terminal
# What identity is being minted, for whom, and for how long?
spire-server entry show \
-spiffeID spiffe://acme.internal/ns/payments/sa/payments
output
Found 1 entry
Entry ID : 8f2c9d14-a3b1-4e77-9c02-1d5f2e3a4b5c
SPIFFE ID : spiffe://acme.internal/ns/payments/sa/payments
Parent ID : spiffe://acme.internal/spire/agent/k8s_psat/prod/abc123
Revision : 4
X509-SVID TTL : 3600
JWT-SVID TTL : 300
Selector : k8s:ns:payments
Selector : k8s:sa:payments

X509-SVID TTL : 3600 is the exposure you are buying, in seconds, pinned on this entry rather than inherited from the server default. Parent ID names the one agent allowed to hand this identity out, attested through k8s_psat (projected service account token, the short-lived audience-scoped token Kubernetes gives a pod so the node can prove which cluster and node it is). The two selectors are the facts that agent checks about the calling process first: the pod's namespace, and its Kubernetes service account. Revision counts edits, which is how you prove a policy change actually landed.

terminal
# Revoke = remove the rule that mints the identity.
spire-server entry delete -entryID 8f2c9d14-a3b1-4e77-9c02-1d5f2e3a4b5c
# Then, from inside the pod, prove the Workload API issues nothing.
spire-agent api fetch x509 -socketPath /run/spire/agent-sockets/api.sock
output
Deleted entry with ID: 8f2c9d14-a3b1-4e77-9c02-1d5f2e3a4b5c
rpc error: code = PermissionDenied desc = no identity issued

Two clocks start when that entry disappears. The agent re-syncs entries with the server every five seconds by default (sync_interval in the agent config), so the Workload API stops issuing almost at once. Peers are slower. Every service that already accepted the workload's certificate keeps accepting it until notAfter. Your real exposure is the remaining life of the SVID it happened to be holding, up to an hour here. That number is your revocation time, whatever the policy document claims.

When the node is the problem rather than the pod, the same logic moves up a level. Think of the agent as the branch office that prints the badges: every SPIRE agent holds its own identity, and every workload SVID on that node passes through it. spire-server agent evict throws away an agent's attestation record so it has to prove itself from scratch, which is what a rebuilt or stale node needs. spire-server agent ban does that and refuses to let it back in at all. Ban when you think the machine itself is hostile.

terminal
# Which agents are attested, and until when?
spire-server agent list
# The node itself is suspect: refuse to let it attest again.
spire-server agent ban \
-spiffeID spiffe://acme.internal/spire/agent/k8s_psat/prod/abc123
output
Found 1 attested agent:
SPIFFE ID : spiffe://acme.internal/spire/agent/k8s_psat/prod/abc123
Attestation type : k8s_psat
Expiration time : 2026-07-21 09:52:38 +0000 UTC
Serial number : 316325009148772284801
Can re-attest : true
Agent banned successfully

Rotation You Can Watch

Short TTLs are survivable only because something renews them in time, and identity systems do that at roughly the half-life of the certificate. You collect tomorrow's pass halfway through today, not at one minute to midnight. SPIRE's agent replaces an X.509-SVID once the certificate passes the halfway point of its lifetime, so an hour-long certificate is swapped near the thirty-minute mark. Istio works the same way with different words. pilot-agent (the helper process in the sidecar that fetches certificates on behalf of Envoy, the proxy sitting beside your app) asks for a certificate lasting SECRET_TTL, 24 hours by default, then rotates it once the time remaining falls below SECRET_GRACE_PERIOD_RATIO of that lifetime, 0.5 by default. Half of twenty-four is twelve, so the sidecar changes certificates near hour twelve without dropping a connection. That slack is the point. When issuance breaks, you get hours of warning instead of seconds.

terminal
# Stream every SVID update the agent pushes to this workload.
# The elapsed time printed counts from when the watch started,
# not from when the certificate was signed.
spire-agent api watch -socketPath /run/spire/agent-sockets/api.sock
output
Received 1 svid after 4.83ms
SPIFFE ID: spiffe://acme.internal/ns/payments/sa/payments
SVID Valid After: 2026-07-21 09:15:03 +0000 UTC
SVID Valid Until: 2026-07-21 10:15:13 +0000 UTC
CA #1 Valid After: 2026-07-21 00:04:11 +0000 UTC
CA #1 Valid Until: 2026-07-22 00:04:21 +0000 UTC
Received 1 svid after 25m12.407s
SPIFFE ID: spiffe://acme.internal/ns/payments/sa/payments
SVID Valid After: 2026-07-21 09:45:02 +0000 UTC
SVID Valid Until: 2026-07-21 10:45:12 +0000 UTC
CA #1 Valid After: 2026-07-21 00:04:11 +0000 UTC
CA #1 Valid Until: 2026-07-22 00:04:21 +0000 UTC

Rotation with your own eyes: same identity, a window that moved forward by about thirty minutes, no restart. The elapsed counter reads twenty-five minutes rather than thirty because it started when you ran the command, five minutes after the first certificate was signed. Check the mesh separately, because Envoy holds its certificates in memory and will happily keep serving an old one.

terminal
istioctl proxy-config secret payments-7c9f8b6d4-x2kqp -n payments
output
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true 264066809925741038812 2026-07-22T08:41:12Z 2026-07-21T08:41:12Z
ROOTCA CA ACTIVE true 158072248901337720044 2036-07-15T11:22:47Z 2026-07-18T11:22:47Z

default is the workload certificate, good for its twenty-four hours. ROOTCA is the bundle Envoy uses to verify peers, and that ten-year window is Istio's self-signed root doing what roots do. VALID CERT true means Envoy parsed the chain and the clock agrees it is inside its window right now. Nothing more than that. To see what the certificate actually asserts, pull the leaf out of the sidecar and read it.

terminal
istioctl proxy-config secret payments-7c9f8b6d4-x2kqp -n payments -o json \
| jq -r '.dynamicActiveSecrets[] | select(.name=="default")
| .secret.tlsCertificate.certificateChain.inlineBytes' \
| base64 -d | step certificate inspect -
output
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 264066809925741038812 (0xe4f0a1c9b7d3e25f1c)
Signature Algorithm: SHA256-RSA
Issuer: O=acme.internal
Validity
Not Before: Jul 21 08:41:12 2026 UTC
Not After : Jul 22 08:41:12 2026 UTC
Subject:
Subject Public Key Info:
Public Key Algorithm: RSA
Public-Key: (2048 bit)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
Server Authentication, Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name: critical
URI:spiffe://acme.internal/ns/payments/sa/payments

The Subject line is empty on purpose, which is why the SAN is marked critical: with no subject to fall back on, a verifier is forbidden from ignoring the one field that carries the name. Both commands so far describe this instant, though. Alerting needs counters that accumulate, and Envoy keeps them for SDS (Secret Discovery Service, the channel over which the proxy receives certificates and keys from istiod, the Istio control plane process that also runs the mesh certificate authority).

terminal
kubectl exec -n payments payments-7c9f8b6d4-x2kqp -c istio-proxy -- \
pilot-agent request GET stats | grep '^sds\.'
output
sds.ROOTCA.init_fetch_timeout: 0
sds.ROOTCA.update_attempt: 2
sds.ROOTCA.update_failure: 0
sds.ROOTCA.update_rejected: 0
sds.ROOTCA.update_success: 2
sds.ROOTCA.update_time: 1784556758000
sds.default.init_fetch_timeout: 0
sds.default.update_attempt: 4
sds.default.update_failure: 0
sds.default.update_rejected: 0
sds.default.update_success: 4
sds.default.update_time: 1784623272000

update_success should tick up once per rotation per secret. Four on sds.default is a pod that has been running long enough to rotate three times since it started, twelve hours apart. update_time is a millisecond timestamp, and that one lands on 08:41:12 today, matching the NOT BEFORE you just read. The dangerous shape is update_failure climbing while update_time stands still. The proxy is running on a certificate it can no longer replace. Nothing breaks. Traffic stays green all the way to notAfter, and then every mutual TLS connection on that workload fails inside the same second. Alert on update_failure and on the age of update_time. Wait for connection errors and the outage has already started.

Short TTLs punish clock drift
A one-hour certificate lives and dies by notBefore and notAfter, two absolute UTC timestamps that verifiers check with no tolerance built in. If the signer's clock and the verifier's clock differ by two minutes, that is over three percent of the window gone, and you get 'certificate is not yet valid' on freshly minted SVIDs or a dead one briefly accepted. SPIRE backdates notBefore by ten seconds to soak up small drift, which buys you seconds, not minutes. Run chrony or ntpd (daemons that keep a machine's clock in step with a time server, using NTP, the network time protocol) on every node, alarm when drift crosses one second, and never set a TTL below your worst measured skew plus your issuance latency.

Choosing the TTL Is a Capacity Decision

Shorter is not automatically better. Every SVID is a signature performed on the SPIRE server, every mesh certificate is a signing request through istiod, and every federated cloud credential is a call to an STS (Security Token Service, the endpoint that trades a workload identity for a short-lived cloud credential) with a published rate limit. Ten thousand workloads on one-minute certificates rotate at the thirty-second half-life, which is over three hundred signatures per second forever, plus the cache churn each one causes downstream.

Set the floor from measurement, not taste. A TTL shorter than issuance latency plus clock skew plus one retry guarantees outages, because a workload reaches notAfter before its replacement lands. Measure the p99 (the 99th percentile, the slow tail rather than the average) of issuance at the SPIRE server and at istiod, add your worst observed drift, and refuse to go under it.

/run/spire/config/server.conf
server {
trust_domain = "acme.internal"
# Lifetime of the signing key that mints every SVID below.
ca_ttl = "24h"
# Defaults handed to workloads that do not override them.
default_x509_svid_ttl = "1h" # proof of key possession
default_jwt_svid_ttl = "5m" # a bearer token, so keep it tiny
# Keep default_x509_svid_ttl at or under ca_ttl / 6 (here, 4h).
# ...bind_address, data_dir, datastore, ca_subject, plugins omitted...
}

Per-entry overrides are the useful middle ground. Leave the fleet at an hour and give the handful of identities that move money a much shorter leash, with -x509SVIDTTL and -jwtSVIDTTL.

terminal
# Both TTL flags take seconds. The old -ttl flag is deprecated;
# use these two and do not mix the styles on one entry.
spire-server entry create \
-spiffeID spiffe://acme.internal/ns/payments/sa/treasury \
-parentID spiffe://acme.internal/spire/agent/k8s_psat/prod/abc123 \
-selector k8s:ns:payments \
-selector k8s:sa:treasury \
-x509SVIDTTL 900 \
-jwtSVIDTTL 120
output
Entry ID : 5c1e7a80-2b44-4f0e-8c3d-91a7f6b2e004
SPIFFE ID : spiffe://acme.internal/ns/payments/sa/treasury
Parent ID : spiffe://acme.internal/spire/agent/k8s_psat/prod/abc123
Revision : 0
X509-SVID TTL : 900
JWT-SVID TTL : 120
Selector : k8s:ns:payments
Selector : k8s:sa:treasury
Keep the SVID TTL under one sixth of ca_ttl
An SVID can never outlive the key that signed it, so SPIRE caps a certificate's notAfter at the signing key's own expiry. Push default_x509_svid_ttl above roughly one sixth of ca_ttl and SPIRE warns at startup, then quietly hands out SVIDs shorter than you asked for as the CA nears its own rotation. The symptom is baffling: workloads that rotate on schedule all day suddenly rotate twice as fast for an hour, and anything with a hard-coded assumption about certificate lifetime breaks inside that window. Raise ca_ttl rather than squeezing the ratio.

What mTLS Proves, and What It Does Not

Two people at a door, each showing a photo card and each proving they own it. That is mTLS (mutual Transport Layer Security, where both ends present a certificate and demonstrate they hold the matching private key). It establishes one narrow fact: at handshake time, the peer held a private key for that SPIFFE ID, and its certificate chained to a root you trust. Strong, and small. It says nothing about whether the process is running the image you approved, whether it was compromised twenty minutes after it started, whether this particular request is allowed, or whether the person whose money is moving still has a session. A stolen key inside a properly attested pod produces a flawless handshake.

So check more, and check it on every request. Three signals ride the same call. The connection carries workload identity from mTLS. The request carries a token: a user's JWT (JSON Web Token, a signed bundle of claims) from your identity provider, or a JWT-SVID (the SPIFFE token format, for paths that cross something which terminates TLS and destroys the mTLS identity along with it). And posture, the caller's current health and configuration, gets evaluated before the request reaches application code.

One resource is where teams get this wrong. RequestAuthentication validates a token when one is present and waves the request through untouched when the caller sends none. It is a bouncer who checks IDs but never asks for one. The field that makes a token mandatory is requestPrincipals, and it lives inside an AuthorizationPolicy.

payments/per-request.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: payments
spec:
mtls:
mode: STRICT # no plaintext peer, so there is always a principal
---
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
name: ledger-jwt
namespace: payments
spec:
selector:
matchLabels:
app: ledger
jwtRules:
- issuer: "https://sso.acme.internal"
# JWKS = JSON Web Key Set, the public keys used to check signatures.
jwksUri: "https://sso.acme.internal/.well-known/jwks.json"
# Validates a token IF one arrives. A request with no token at all
# passes straight through. This resource requires nothing.
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: ledger-transfers
namespace: payments
spec:
selector:
matchLabels:
app: ledger
action: ALLOW
rules:
- from:
- source:
# Workload identity from the mTLS handshake. Istio drops the
# scheme: <trust domain>/ns/<namespace>/sa/<service account>
principals: ["acme.internal/ns/payments/sa/payments"]
# THIS is what makes a user token mandatory: <issuer>/<subject>
requestPrincipals: ["https://sso.acme.internal/*"]
to:
- operation:
methods: ["POST"]
paths: ["/v1/transfers"]
when:
- key: request.auth.claims[scope]
values: ["transfers.write"]

Read that last rule as one sentence: the payments workload, over mTLS, carrying a live user token from our single sign-on provider whose scope claim includes transfers.write, may POST to /v1/transfers. Break any link and the call dies. A user token stolen from a browser and replayed from an unauthorized service fails the principals check. A fully compromised payments pod with no user token fails requestPrincipals. Two things to watch. Once any ALLOW policy selects the ledger workload, everything else arriving at ledger is denied, so add the routes you still need before you ship this. And plenty of providers pack scope into a single space-separated string rather than a list, in which case an exact match on one word inside it will not fire. Decode a real token from your own provider before you trust that condition, and switch to a list-valued claim if yours is a string.

Where mTLS cannot survive the path, SPIRE issues the token flavour instead. Fetch one and see what it costs you.

terminal
spire-agent api fetch jwt \
-audience spiffe://acme.internal/ns/payments/sa/ledger \
-socketPath /run/spire/agent-sockets/api.sock
output
token(spiffe://acme.internal/ns/payments/sa/payments):
eyJhbGciOiJFUzI1NiIsImtpZCI6IlBrOHhOa1JIYW1oM2EiLCJ0eXAiOiJKV1QifQ.eyJhdWQ...
bundle(spiffe://acme.internal):
{
"keys": [
{
"use": "jwt-svid",
"kty": "EC",
"crv": "P-256",
"kid": "Pk8xNkRHamh3a",
"x": "fA9J8m...",
"y": "kP2mD1..."
}
]
}
terminal
spire-agent api validate jwt \
-audience spiffe://acme.internal/ns/payments/sa/ledger \
-svid eyJhbGciOiJFUzI1NiIsImtpZCI6IlBrOHhOa1JIYW1oM2EiLCJ0eXAiOiJKV1QifQ.eyJhdWQ...
output
SVID is valid.
SPIFFE ID : spiffe://acme.internal/ns/payments/sa/payments
Claims : {"aud":["spiffe://acme.internal/ns/payments/sa/ledger"],"exp":1784625603,"iat":1784625303,"sub":"spiffe://acme.internal/ns/payments/sa/payments"}

exp minus iat is 300 seconds, straight from default_jwt_svid_ttl. It is small for a reason. A JWT-SVID is a bearer token, like a cinema ticket: whoever holds it can use it, no private key required. The -audience value is the other half of the defence, and it is the screen number printed on that ticket. It names the intended receiver, and a receiver that fails to reject tokens whose aud is not itself has built a credential any peer can pocket and replay onward. Prefer X.509-SVIDs wherever the path allows.

Posture is the third signal, and it needs somewhere to run. Istio's CUSTOM action hands the decision to an outside service over Envoy's ext-authz hook (external authorization, the point where the proxy pauses a request to ask another process yes or no), and Istio evaluates CUSTOM ahead of any DENY or ALLOW rule. Point it at OPA (Open Policy Agent, a policy engine that answers allow or deny from rules written in a language called Rego).

ext-authz.yaml
# Part 1: istiod install values, NOT a cluster resource.
meshConfig:
extensionProviders:
- name: opa-ext-authz # referenced by provider.name below
envoyExtAuthzGrpc:
service: opa.opa-system.svc.cluster.local
port: 9191
timeout: 0.2s
failOpen: false # OPA unreachable => request denied
---
# Part 2: the policy that routes sensitive calls through OPA.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: ledger-posture
namespace: payments
spec:
selector:
matchLabels:
app: ledger
action: CUSTOM
provider:
name: opa-ext-authz # must match the extensionProvider name
rules:
- to:
- operation:
methods: ["POST"]
paths: ["/v1/transfers"]
policy/authz.rego
package envoy.authz
# OPA's Envoy plugin reads the decision from data.envoy.authz.allow
default allow := false
# The mTLS peer identity Envoy verified during the handshake, e.g.
# "spiffe://acme.internal/ns/payments/sa/payments"
principal := input.attributes.source.principal
req := input.attributes.request.http
allow if {
principal == "spiffe://acme.internal/ns/payments/sa/payments"
req.method == "POST"
startswith(req.path, "/v1/transfers")
posture_ok
}
# Posture counts only while it is fresh. Stale signal means deny.
posture_ok if {
p := data.posture[principal]
time.now_ns() - p.checked_at_ns < 300000000000 # 5 minutes, in ns
p.image_digest in data.approved_digests
p.critical_cves == 0
}

failOpen: false is the line that decides what an outage means. False is the default and it should stay there. Flip it to true and losing OPA turns your most sensitive route into an open one, which is a cheap win for anybody who can knock over a single pod. Fail closed and an OPA outage becomes a loud incident instead of a silent bypass. Load-test the authorizer before you rely on it, because it now sits in the request path of every transfer and that 0.2 second timeout is not much room.

Posture per request is not free, and the Rego above admits it. data.posture is only as fresh as whatever pushes it into OPA, which is why the freshness check exists: deny when the signal is older than five minutes rather than trusting a scan from last Tuesday. Every check adds latency to every call, and every signal is a pipeline somebody maintains. Most teams settle on identity and scope checks everywhere, full posture evaluation on the routes that move money or data, and a short cache in between. Classify routes deliberately, because unclassified ones drift into the cheap tier and stay there.

The Gap Between Revoked and Disconnected

Expiry only revokes if connections actually end. The guard checks your badge at the door, then the door gets propped open. Envoy verifies the peer certificate during the TLS handshake and does not re-verify it afterwards. A long-lived gRPC (a streaming remote-call protocol built on HTTP/2) stream opened at 09:00 with a certificate that expires at 10:00 is still flowing at 11:00, and deleting the registration entry does not tear it down. For HTTP traffic, authorization rules are re-evaluated per request, so a new AuthorizationPolicy bites on the next request over an old connection, though the identity being judged was captured back at the handshake. For plain TCP, the check happens once per connection and a policy change waits for that connection to close.

Cap connection age, so the handshake happens again on a schedule you chose rather than one an attacker chose.

payments/ledger-destinationrule.yaml
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: ledger
namespace: payments
spec:
host: ledger.payments.svc.cluster.local
trafficPolicy:
connectionPool:
tcp:
maxConnectionDuration: 30m # close, re-handshake, re-verify
http:
idleTimeout: 5m # reap connections nobody is using
terminal
# Client-side sidecar: are connections actually being recycled?
kubectl exec -n payments deploy/payments -c istio-proxy -- \
pilot-agent request GET stats | grep upstream_cx_max_duration_reached
output
cluster.outbound|8080||ledger.payments.svc.cluster.local.upstream_cx_max_duration_reached: 47

A rising counter means connections are being recycled and identities re-verified at the interval you set. A counter stuck at zero after you applied the rule means the rule is not in force: wrong host string, wrong namespace, or another DestinationRule already claiming that host and quietly winning. Check the counter, not the manifest.

Three clocks, each covering what the slower one cannot
Issuance clock (minutes to hours)
Registration entry
delete it and nothing new is minted
default_x509_svid_ttl 1h
agent rotates near the 30-minute half-life
ca_ttl 24h
keep the SVID TTL at or under ca_ttl / 6
Connection clock (per handshake)
mTLS handshake
proves key possession for one SPIFFE ID, once
PeerAuthentication STRICT
no plaintext peer, so a principal always exists
maxConnectionDuration 30m
forces a fresh handshake with a fresh certificate
Request clock (every call)
AuthorizationPolicy
principals + requestPrincipals + claim checks
JWT-SVID
5-minute bearer token, bound to one audience
ext-authz to OPA
posture, with failOpen: false
Expiry revokes nothing while issuance continues, and an open connection outlives the certificate that opened it. Continuous verification means all three clocks run, and you can read each one from a live workload.

Do three things this week and the diagram becomes an operational property. Measure p99 issuance latency at the SPIRE server and at istiod, then write your TTL floor as that number plus your worst clock drift plus one retry. Add alerts on sds.default.update_failure and on the age of sds.default.update_time, because a certificate that cannot be replaced looks perfectly healthy until the second it does not. Then run the drill on a real workload in a real environment: delete its registration entry, start a stopwatch, and record how long peers keep accepting the dead identity. Whatever the stopwatch says is your revocation time, and it belongs in the incident runbook in those words.

Quick check
01A workload is confirmed compromised. Its SVIDs have a one-hour TTL and the SPIRE agent rotates them automatically. What cuts off its access fastest and most reliably?
Incorrect — Short-lived credential systems usually distribute no CRL at all, and a list that lands after the certificate has expired changes nothing.
Correct — stopping issuance is the revocation, and the held certificate dies within the remaining window.
Incorrect — The agent renews at half-life, so a workload that keeps running never expires and waiting revokes nothing.
Incorrect — That invalidates every workload rather than one, and SPIRE deliberately keeps the old root in the trust bundle during rotation anyway.
02You apply a RequestAuthentication to the ledger workload naming your SSO issuer and JWKS URI. A caller sends a request carrying no token at all. What happens?
Incorrect — It rejects tokens that are present and invalid; a missing token triggers nothing at all.
Incorrect — No denial happens unless an AuthorizationPolicy actually requires a request principal.
Incorrect — PeerAuthentication governs transport authentication between workloads and has no bearing on whether a user token was supplied.
Correct — RequestAuthentication validates, AuthorizationPolicy requires.
03A sidecar reports sds.default.update_success: 1, sds.default.update_failure: 9, and an sds.default.update_time that has not moved in 22 hours. istioctl proxy-config secret shows the default cert with VALID CERT true and NOT AFTER about two hours away. Application traffic is healthy. What is going on?
Correct — a 24-hour certificate should have rotated ten hours ago, and a stalled update_time with rising failures is an outage already scheduled for notAfter.
Incorrect — VALID CERT reports on the certificate held in memory right now and says nothing about the ability to obtain the next one.
Incorrect — SDS counters are namespaced per secret, and sds.default is the workload certificate itself.
Incorrect — Skew surfaces as 'certificate is not yet valid' or expired errors on the data path, not as failed SDS updates for a cert Envoy currently accepts.

Try this

Run openssl x509 -in /tmp/svid/svid.0.pem -noout -dates -ext subjectAltName on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: short TTLs punish clock drift. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related