Request-level auth (JWT)

Validate tokens; authorize on verified claims.

A courier walks up to a bank's loading dock wearing a company badge. The guard checks it, the door opens. Nothing on that badge says whose money is in the envelope, or how much of it may move. For that, the guard reads the signed work order inside. Two documents, two questions, two separate signatures. Mutual TLS (mTLS, where both ends of a connection prove who they are with certificates, instead of only the server proving it) is the badge. A JSON Web Token (JWT, a small signed document carried inside the request itself) is the work order. Earlier lessons handed every workload a SPIFFE identity (Secure Production Identity Framework for Everyone, the open standard for naming workloads) and wrapped east-west traffic (service-to-service calls inside the cluster) in [mTLS](/courses/zerotrust/zt-mtls/). So the mesh already knows which *service* sits at the other end of a connection. This lesson is the second signature: proving, per request, on whose behalf the call is being made, and authorizing on claims you have actually checked.

What mTLS proves, and what it cannot

An mTLS handshake proves one narrow thing, and it proves it beautifully. At the instant the connection was established, the peer held the private key for a certificate that the mesh certificate authority issued to the SPIFFE ID cluster.local/ns/prod/sa/frontend, and that certificate was inside its validity window. Read that sentence again and notice its shape. It is a statement about a *connection*, and about the *immediate peer* on the other end of it.

Four things it never says. It does not say which end user a request belongs to, because a single pooled HTTP/2 connection between frontend and orders-api carries requests for thousands of different people over one certificate. It does not say what *caused* the request. It does not say the calling process is healthy, because an attacker with a shell inside that pod talks to the same Unix socket and is handed the same identity. And it is hop by hop, meaning it only ever describes the last leg of the journey: an ingress gateway that terminates TLS and opens a fresh connection inward presents its own identity to the next service, not the original client's.

Add those up and you get a classic confused deputy problem, where a trusted component is tricked into using its own authority on an attacker's behalf. Say your [AuthorizationPolicy](/courses/zerotrust/zt-authz/) lets the frontend send DELETE /orders/*. Somebody finds a server-side request forgery bug in the frontend (SSRF, where a user-supplied URL makes the server fetch something on the attacker's behalf). Every request that bug produces leaves the frontend's own sidecar, carrying the frontend's own certificate, and matches your rule perfectly. mTLS does not fail here. It was never asked the question. The only thing that closes the gap is a second credential riding *inside* the request, one the bug cannot mint.

What a JWT actually carries

Think of a JWT as a postcard in a tamper-evident sleeve. Anyone who handles it can read it. Nobody can alter it without visibly breaking the seal. Three chunks of base64url text (an alphabet-and-digits encoding that survives being pasted into a URL or an HTTP header) separated by dots: a header naming the signing algorithm and the key id (kid), a payload of claims, and a signature covering the first two.

Claims are assertions. iss names the issuer that minted the token, sub the subject (the user), aud the audience (which service this token is *for*), exp the moment it dies, nbf the moment it starts working ("not before"), plus application claims like roles or groups. The issuer signs with a private key and publishes the matching public keys as a JWKS (JSON Web Key Set, a JSON document listing public keys indexed by kid) at a well-known URL. Verification is asymmetric, so the mesh never touches the issuer's private key and the issuer never sees your mesh traffic. Inspect a token by hand before you write a single line of policy against it.

# Decode without verifying. Every claim is plain JSON to anyone holding the token.
echo "$USER_JWT" | step crypto jwt inspect --insecure
 
# Now verify for real: signature against the issuer's published keys, plus iss/aud/exp.
curl -sS https://idp.acme.com/.well-known/jwks.json -o jwks.json
echo "$USER_JWT" | step crypto jwt verify \
  --iss https://idp.acme.com --aud orders-api --alg RS256 --jwks jwks.json \
  | jq -r '"ok: " + .payload.sub'
 
# The same token, offered to a service it was never minted for.
echo "$USER_JWT" | step crypto jwt verify \
  --iss https://idp.acme.com --aud billing-api --alg RS256 --jwks jwks.json
echo "exit=$?"

{
  "header": {
    "alg": "RS256",
    "kid": "8f2c1a",
    "typ": "JWT"
  },
  "payload": {
    "aud": "orders-api",
    "exp": 1753101000,
    "iat": 1753100400,
    "iss": "https://idp.acme.com",
    "roles": [
      "orders.reader"
    ],
    "scope": "openid orders.read",
    "sub": "u-4815"
  },
  "signature": "Yx0Qk3rW9nT4hK1s..."
}
ok: u-4815
validation failed: invalid audience claim (aud)
exit=1

Two lessons hide in that output. The signature proves the token is authentic and unmodified, and nothing else. Whether *this* service should honour it is a separate question, which is exactly why the third command fails on a token that is cryptographically perfect. And every claim is readable by whoever holds the token, so a JWT is a credential you present, never a place to keep a secret. If you have ever seen an API key stuffed into a custom claim, you have seen an API key published.

Validating tokens at the sidecar

You could verify tokens inside every service. You would then be maintaining a JWKS cache, a clock-skew policy, and an algorithm allow-list in five languages, and on the day one team forgets to pin alg you have a hole shaped like alg: none. Every pod in the mesh already runs a second container next to the application. That sidecar is an Envoy proxy, and it sees every byte going in and out. Put the check there once. Two Istio objects do it, and they do very different jobs, which is where almost every mistake starts.

# 1) HOW to check a token. On its own, this requires nothing.
apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: orders-jwt
  namespace: prod
spec:
  selector:
    matchLabels:
      app: orders-api
  jwtRules:
    - issuer: "https://idp.acme.com"
      jwksUri: "https://idp.acme.com/.well-known/jwks.json"
      audiences: ["orders-api"]     # omit this and ANY token this issuer signed is accepted
      forwardOriginalToken: true    # default false: the sidecar strips the header after checking
      outputClaimToHeaders:
        - header: "x-auth-sub"      # hand the app a verified claim it never has to parse itself
          claim: "sub"
---
# 2) WHETHER a token is required, and which identity may do what with it.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-require-jwt
  namespace: prod
spec:
  selector:
    matchLabels:
      app: orders-api
  action: ALLOW                     # once ANY allow policy selects a workload,
  rules:                            # every request that matches no rule on it is denied
    # Reads: the frontend workload, carrying any valid token from our issuer.
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/frontend"]  # mTLS identity, spiffe:// stripped
            requestPrincipals: ["https://idp.acme.com/*"]      # "<iss>/<sub>", any subject
      to:
        - operation:
            methods: ["GET"]
            paths: ["/orders/*"]
    # Deletes: same workload, same issuer, and an explicit role claim on top.
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/frontend"]
            requestPrincipals: ["https://idp.acme.com/*"]
      to:
        - operation:
            methods: ["DELETE"]
            paths: ["/orders/*"]
      when:
        - key: request.auth.claims[roles]   # an ARRAY claim, matched element by element
          values: ["orders.admin"]

RequestAuthentication teaches the sidecar how to check a token. If one is present, Envoy's jwt_authn filter verifies the signature against the issuer's keys and checks iss, aud, exp and nbf. A token that fails is rejected with 401 on the spot, before any authorization rule is evaluated and long before your code runs. A token that passes has its claims published into the request's metadata, where policy reads them as request.auth.claims[roles] (nested claims use request.auth.claims[a][b]) and reads its identity as request.auth.principal, which is iss and sub joined by a slash: https://idp.acme.com/u-4815. The AuthorizationPolicy is where the two identity dimensions finally meet in one rule. The frontend *service*, carrying a valid *user* token, may read orders, and may delete one only when that user's roles claim contains orders.admin. Neither layer alone is enough. A stolen token cannot be replayed from an unauthorized workload, and a compromised workload gets nowhere without a token. Pair both objects with a PeerAuthentication in STRICT mode so plaintext traffic cannot skip the badge check in the first place.

Prove the policy actually bites

A policy that is applied is not a policy that is working. Fire one call per branch from a pod that carries the right mTLS identity, and change only the token.

# Every probe runs inside the frontend pod, so its sidecar supplies the mTLS identity
# cluster.local/ns/prod/sa/frontend. Only the token varies.
call() {
  kubectl exec -n prod deploy/frontend -c frontend -- \
    curl -s -w ' <- %{http_code}\n' "$@" \
    http://orders-api.prod.svc.cluster.local/orders/42
}
 
call                                                  # no Authorization header at all
call -H 'Authorization: Bearer not-a-token'           # not even three sections
call -H "Authorization: Bearer $BILLING_JWT"          # real token, minted for another API
call -H "Authorization: Bearer $USER_JWT"             # the right token
call -X DELETE -H "Authorization: Bearer $USER_JWT"   # right token, role claim too weak

RBAC: access denied <- 403
Jwt is not in the form of Header.Payload.Signature with two dots and 3 sections <- 401
Audiences in Jwt are not allowed <- 401
{"id":42,"customer":"u-4815","status":"shipped"} <- 200
RBAC: access denied <- 403

Read those two status codes as a triage tool, because they point at completely different teams. 401 came from the authentication filter: the token itself was bad, and Envoy names the exact reason in the body. 403 with RBAC: access denied came from the authorization filter (RBAC is role-based access control, Envoy's rule engine), which means the token was fine, or absent, and no rule allowed the call. If a release starts throwing 401s you go look at the identity provider and its keys. If it throws 403s you go look at your own policy. Chasing the wrong one burns an hour of an incident. Envoy keeps a running tally of both decisions, which is the cheapest proof that the filters are wired in at all.

# Counters on the destination sidecar, read from the Envoy admin interface.
# These are cumulative since the proxy started, so read them before and after a test run.
kubectl exec -n prod deploy/orders-api -c istio-proxy -- \
  curl -s localhost:15000/stats \
  | grep -E 'jwt_authn\.(allowed|denied)|rbac\.(allowed|denied)'

http.inbound_0.0.0.0_8080.jwt_authn.allowed: 3
http.inbound_0.0.0.0_8080.jwt_authn.denied: 2
http.inbound_0.0.0.0_8080.rbac.allowed: 1
http.inbound_0.0.0.0_8080.rbac.denied: 2

Those deltas reconcile exactly with the five probes, and that reconciliation is the point. jwt_authn.allowed: 3 counts the calls whose token was acceptable *or absent* (the untokened one, the good GET, the good DELETE), while denied: 2 counts the garbage token and the wrong-audience token. Then rbac.denied: 2 counts the untokened call and the under-privileged DELETE, leaving rbac.allowed: 1 for the single call that got through. If the jwt_authn counters do not exist on the pod at all, your RequestAuthentication selector is not matching the workload and nothing whatsoever is being verified, no matter how green kubectl get requestauthentication looks.

Where the signing keys really come from

This is the part that catches people mid-incident. The sidecar does not call your identity provider. istiod, the Istio control plane, fetches the JWKS from jwksUri, caches it, and inlines the keys into every selected proxy's configuration as a static key set. So the network path that must work is istiod to the identity provider, not thirty thousand pods to the identity provider. Get that backwards in an egress rule or a NetworkPolicy and JWT authentication fails in a way that looks like a token problem for the first twenty minutes. There is an opt-in istiod setting, PILOT_JWT_ENABLE_REMOTE_JWKS, that pushes the fetch out to Envoy instead, but it is off by default, so assume the istiod path until you have checked. You can see exactly which keys a given proxy is holding right now.

# Which signing keys does this sidecar actually hold at this moment?
# Sidecar inbound traffic all lands on the virtualInbound listener, port 15006.
istioctl proxy-config listener deploy/orders-api -n prod --port 15006 -o json \
  | jq -r '[.. | objects | select(has("localJwks")) | .localJwks.inlineString] | first' \
  | jq -r '.keys[] | [.kid, .alg, .kty] | @tsv'

8f2c1a	RS256	RSA
d41e77	RS256	RSA

Two key ids means the rotation overlap is healthy. istiod refreshes on PILOT_JWT_PUB_KEY_REFRESH_INTERVAL, 20 minutes by default, and it holds on to the last copy it successfully fetched for days if the issuer stops answering, so a short outage at the identity provider does not immediately take your traffic down. The safe rotation order falls straight out of that refresh number. Publish the new public key in the JWKS first, re-run the command above until both kid values appear, and only then let the identity provider start signing with the new key. Do it in the other order and every token signed with the new key, right up until the next refresh lands, is rejected with Jwks doesn't have key to match kid or alg from Jwt, while you stare at an identity provider that reports itself perfectly healthy.

JWT-SVIDs, when the connection cannot carry identity

Sometimes mTLS cannot survive the path. A managed layer-7 load balancer terminates TLS and starts a new connection of its own. A job lands on a queue and is processed forty minutes later, with no connection left to inspect. A call crosses into a cloud provider's own API. The connection identity dies at the first of those hops, but the *request* still needs to name its caller. That is what a JWT-SVID (SPIFFE Verifiable Identity Document in JWT form) is for: a workload identity that SPIRE mints as a token, which a workload can hand to something far away. You register the workload once, then it asks the local agent for tokens over a Unix socket.

# Register the workload, and cap how long its JWT identity documents live.
# -jwtSVIDTTL is in seconds; k8s_psat is the projected service account token attestor.
spire-server entry create \
  -spiffeID  spiffe://cluster.local/ns/prod/sa/report-exporter \
  -parentID  spiffe://cluster.local/spire/agent/k8s_psat/prod-cluster/7f4e2a10-9b3d-4c11-8f02-1a5d6e7c9b44 \
  -selector  k8s:ns:prod \
  -selector  k8s:sa:report-exporter \
  -jwtSVIDTTL 300
 
# From inside that workload, ask the Workload API for a token for ONE audience.
# This also prints the trust bundle used to verify it; trimmed from the output below.
spire-agent api fetch jwt \
  -audience orders-api \
  -socketPath /run/spire/sockets/agent.sock

Entry ID         : 4b1f9a3c-6d21-4e8f-9a77-2c0b5e1d8f30
SPIFFE ID        : spiffe://cluster.local/ns/prod/sa/report-exporter
Parent ID        : spiffe://cluster.local/spire/agent/k8s_psat/prod-cluster/7f4e2a10-9b3d-4c11-8f02-1a5d6e7c9b44
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : 300
Selector         : k8s:ns:prod
Selector         : k8s:sa:report-exporter
 
token(spiffe://cluster.local/ns/prod/sa/report-exporter):
	eyJhbGciOiJFUzI1NiIsImtpZCI6IlR2WnBRMk5rNGFSbTh4TGQiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOlsib3JkZXJzLWFwaSJdLCJleHAiOjE3NTMxMDA3MDAsImlhdCI6MTc1MzEwMDQwMCwic3ViIjoic3BpZmZlOi8vY2x1c3Rlci5sb2NhbC9ucy9wcm9kL3NhL3JlcG9ydC1leHBvcnRlciJ9.t5BwPxmRZlzpZDSkmXtvAOh3jtG7p077kHCeiZlJn9a-4k2rgBYL2e3FovbRkY4wUmzOgvnIPm8-Tq5xEDFFrA

Decode the middle segment and you get {"aud":["orders-api"],"exp":1753100700,"iat":1753100400,"sub":"spiffe://cluster.local/ns/prod/sa/report-exporter"}. The header says ES256, because SPIRE signs with an elliptic-curve P-256 key by default rather than RSA. Now notice what is missing: there is no iss claim. Envoy picks a jwtRule by matching the token's iss, Istio requires that field on every rule, and request.auth.principal is assembled from iss and sub. A stock JWT-SVID matches nothing and comes back as Jwt issuer is not configured, so RequestAuthentication is the wrong tool for it. Validate these inside the workload instead, against the trust bundle the local agent already serves, or hand them to an external authorizer. If a verifier lives outside the cluster entirely, the SPIFFE OIDC Discovery Provider publishes the trust domain's JWT public keys over HTTPS so it can fetch them.

// go-spiffe v2: validate a JWT-SVID against the bundle served by the local agent.
// The socket path comes from the SPIFFE_ENDPOINT_SOCKET environment variable.
source, err := workloadapi.NewJWTSource(ctx)
if err != nil {
	return err
}
defer source.Close()
 
// Audience is not optional: a token minted for another service must not pass here.
svid, err := jwtsvid.ParseAndValidate(token, source, []string{"orders-api"})
if err != nil {
	return fmt.Errorf("rejecting call: %w", err) // bad signature, wrong audience, or expired
}
log.Printf("caller is %s", svid.ID) // spiffe://cluster.local/ns/prod/sa/report-exporter

Be exact about the trade-off you are taking on. A JWT-SVID is a bearer token with no binding to the channel it travels over, so whoever holds it can use it from anywhere until it expires. The SPIFFE specification says as much and steers you toward X.509-SVIDs and mTLS wherever the path allows them. Keep the lifetimes short (-jwtSVIDTTL 300 above is five minutes, and [short-lived SVIDs](/courses/zerotrust/zt-svid/) are the whole design), and treat every log line, trace attribute, or cache that touches one as a credential store.

What this still does not fix

A JWT is valid until it expires and the sidecar has no way to learn otherwise. No revocation list, no callback to the issuer, nothing. When a token leaks, your exposure window is exactly its remaining lifetime, which means token lifetime *is* your revocation SLA (service level agreement, the promise you make about how fast something takes effect). Ten minutes, with a refresh token behind it, is a sane default. If you truly need instant revocation, or a decision richer than when clauses can express, move the check out of the proxy: an AuthorizationPolicy with action: CUSTOM forwards each matching request to an external authorizer such as OPA (Open Policy Agent, a general-purpose policy engine you write rules for), declared in the mesh config under extensionProviders. That buys live lookups and real logic. It costs a network round trip on every single request, plus a brand new hard dependency in the hot path. Price it honestly before you reach for it.

Three smaller sharp edges. Envoy allows 60 seconds of clock skew on exp and nbf, and RequestAuthentication has no field to change it, so a node whose clock has drifted further than that rejects perfectly good tokens with Jwt is expired if it runs fast, or Jwt not yet valid if it runs slow. Fix the clock; an EnvoyFilter patch is the only other lever and you do not want it. Second, forwardOriginalToken defaults to false, which means the sidecar strips the Authorization header once it has validated the token, a sensible default right up until an application that wanted to read a claim quietly starts seeing nothing. Third, outputClaimToHeaders writes x-auth-sub only after a token has actually been validated, so your application must treat that header's absence as "reject", never as "anonymous is fine". The AuthorizationPolicy demanding requestPrincipals is the only thing that makes its presence a guarantee.

Before any of this touches a live namespace, run the same curl matrix from a pod that sits deliberately *outside* your allow rule and confirm you get 403 rather than 200. One policy pair is easy to hold in your head. A fleet carrying hundreds of RequestAuthentication and AuthorizationPolicy objects across dozens of namespaces is not, and a single mistyped requestPrincipals prefix can black-hole a production API in seconds with no warning at all. [Policy as code and rollout](/courses/zerotrust/zt-policy/) turns these objects into version-controlled artifacts with a dry-run stage in front of them, so identity rules ship as carefully as the identities they govern.

Try this

Run echo "$USER_JWT" | step crypto jwt inspect --insecure on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a RequestAuthentication on its own requires no token at all. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related