The mesh CA & trust chain

Issuance, custom roots, protecting signing keys.

Somewhere in your cluster sits one process that can mint a valid identity for any service you run, including services that do not exist yet. That is the mesh CA (certificate authority: the thing whose signature makes other parties believe a certificate). A coin mint is the right mental picture. There is a master die locked in a vault, a working die bolted into the press, and the coins themselves, which circulate for a day and then stop being money. Steal a coin and you can spend it until it expires. Steal the working die and you can strike coins until someone changes the press. Steal the master die and you can build new presses forever, and nobody downstream can tell. Every design choice in this lesson is about keeping those three things at three different distances from an attacker.

Root, intermediate, leaf

The root CA, also called the trust anchor, is a self-signed certificate every proxy in the mesh believes without further proof. Self-signed means its subject and its issuer are the same name: it vouches for itself, and you accept that because you put it there deliberately. Its private key is the master die. An intermediate CA is a certificate the root signs, granting it permission to sign further certificates. The leaf is what a single workload presents on the wire, valid for hours. Verification runs upward: a peer checks the leaf's signature against the intermediate, the intermediate's against the root, and stops the moment it reaches a root it already trusts.

Here is the part most write-ups get wrong. A default Istio install has no intermediate at all. On first boot istiod (the Istio control-plane process) generates a self-signed root, stores it in a Secret named istio-ca-secret in istio-system, and signs workload leaves directly with it. The chain is two links, not three, and the key that can forge every identity in your cluster is sitting in the cluster database as text. Fine for a demo. A bad place for a master die.

# What signing material does this cluster actually have?
kubectl -n istio-system get secret istio-ca-secret cacerts --ignore-not-found
 
# The crown jewel is one kubectl away from anyone with permission to read it
kubectl -n istio-system get secret istio-ca-secret \
  -o jsonpath='{.data.ca-key\.pem}' | base64 -d | head -1
 
# So find out who can read it before an attacker does
kubectl auth can-i get secrets -n istio-system \
  --as system:serviceaccount:prod:api

NAME              TYPE               DATA   AGE
istio-ca-secret   istio.io/ca-root   5      41d
 
-----BEGIN RSA PRIVATE KEY-----
 
no

No cacerts Secret, so this mesh is running on the root istiod made for itself. DATA 5 is the five files istiod keeps in there, and ca-key.pem is one of them. The head -1 is the whole point: no hardware boundary, no approval step, no ceremony, no second pair of eyes. Kubernetes Secrets are base64 text, and base64 is an encoding, not encryption. It hides nothing from anyone who can read the object. Turning on encryption at rest protects the bytes sitting in etcd (the database Kubernetes keeps all its objects in), which matters if someone walks off with a backup, and it does nothing about the person holding RBAC (role-based access control: the Kubernetes permission system) to get that Secret, because the API server decrypts it on the way out. Whoever that is can impersonate every workload you own, and mTLS (mutual TLS, a handshake where both ends present a certificate instead of only the server) will keep working perfectly for them. Nothing in the mesh will look wrong, because from the mesh's point of view nothing is wrong.

How a workload earns its certificate

Nobody hands these out by hand, and there is no shared password anywhere in the flow. It works like the badge desk in an office lobby. You show a pass your employer issued, the desk phones your employer to confirm it is real, and only then does it print a badge with your name on it. In the mesh, the badge desk is istiod and the employer is the Kubernetes API server.

When a pod starts, the istio-agent process inside the sidecar (the extra container that runs beside your application and takes over its network traffic) generates a private key inside the pod and never transmits it. It builds a CSR (certificate signing request: the public half of that key, plus a request for a name) and calls istiod on port 15012. As proof of identity it attaches a projected Kubernetes service-account token with audience istio-ca and a twelve-hour expiry (expirationSeconds: 43200). istiod hands that token to the Kubernetes API server's TokenReview endpoint, which answers with the namespace and service account it was minted for. From those two facts istiod builds the SPIFFE ID (SPIFFE is the Secure Production Identity Framework For Everyone; a SPIFFE ID is a URI that names a workload, like spiffe://cluster.local/ns/prod/sa/api), signs a leaf carrying it, and returns it over the gRPC stream. One detail people get backwards: the agent, not istiod, is the SDS server (Secret Discovery Service: the channel Envoy uses to receive keys and certificates while it is running). Envoy, the proxy container handling the pod's traffic, reads from a Unix socket inside its own pod and picks up new material without restarting.

Be precise about what that certificate proves, because a lot of zero-trust marketing is sloppy here. A successful mTLS handshake proves exactly one thing: the peer holds the private key for a certificate signed by a CA in your trust bundle (the set of roots that proxy accepts), and that certificate names a SPIFFE ID. It does not prove the code in that pod is the code you shipped. It does not prove the pod is still uncompromised, because the leaf stays valid for its full lifetime and nothing revokes it. It does not prove the caller is allowed to make this request. And the identity is only ever as strong as your control over who can run a pod under that service account. If a developer can start anything in prod with serviceAccountName: api, they get the api identity, properly signed, and every policy that trusts api waves them through.

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: api-callers
  namespace: prod
spec:
  selector:
    matchLabels:
      app: api
  action: ALLOW
  rules:
    - from:
        - source:
            # note: no spiffe:// scheme here
            principals: ["cluster.local/ns/prod/sa/checkout"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/v1/orders/*"]

Two objects, two different jobs. PeerAuthentication in STRICT mode tells the proxies in prod to refuse plaintext, which is what makes a verified peer identity exist at all. AuthorizationPolicy then decides what that identity may do. Watch the shape of principals: Istio writes the SPIFFE ID without the spiffe:// scheme, so spiffe://cluster.local/ns/prod/sa/checkout becomes cluster.local/ns/prod/sa/checkout. Get that wrong and the rule quietly matches nothing. Watch the default too. Once any ALLOW policy selects a workload, requests matching none of its rules are denied. Until you write the first one, every authenticated workload in the mesh can reach every other one.

Read the chain your proxy is actually holding

Do not take the mesh's word for any of this. istioctl proxy-config secret reads the live Envoy configuration out of the proxy's admin interface, so what you see is what that proxy will present, and what it will validate peers against, right now.

istioctl proxy-config secret deploy/api -n prod

RESOURCE NAME   TYPE         STATUS   VALID CERT   SERIAL NUMBER                             NOT AFTER             NOT BEFORE
default         Cert Chain   ACTIVE   true         104839372884019283746501928374650192837   2026-07-22T09:11:42Z  2026-07-21T09:11:42Z
ROOTCA          CA           ACTIVE   true         218374650192837465019283746501928374650   2036-06-07T08:22:31Z  2026-06-10T08:22:31Z

Two resources. default is this workload's leaf, good for 24 hours (SECRET_TTL, an istio-agent setting, defaults to 24h; TTL is time to live, meaning how long the thing stays valid). ROOTCA is the trust bundle Envoy validates every peer against, good for ten years, and its NOT BEFORE is the day somebody first installed Istio here, which lines up with the 41-day-old Secret from earlier. The agent does not wait for expiry to renew. It asks for a fresh leaf once the remaining lifetime falls below SECRET_GRACE_PERIOD_RATIO of the total, which defaults to 0.5, with a little random jitter so a whole deployment does not renew in the same second. A 24-hour leaf is therefore replaced around the twelve-hour mark, over the SDS socket that is already open, with no pod restart and no dropped connection.

Work out what that gives you when istiod goes down, because the answer is not one number. New pods break instantly: no CA, no certificate, no handshake, and the sidecar never reaches ready. Pods already running keep serving traffic until their current leaf genuinely expires, which is up to 24 hours away, although they start retrying and logging failures from the twelve-hour mark. So a dead control plane costs you twelve to twenty-four hours of grace on existing traffic and zero minutes on anything that needs to start.

istioctl proxy-config secret deploy/api -n prod -o json > sds.json
 
# jq is a filter for JSON; these two lines pull the PEM text out of the dump
jq -r '.dynamicActiveSecrets[]|select(.name=="default").secret.tlsCertificate.certificateChain.inlineBytes' sds.json | base64 -d > chain.pem
jq -r '.dynamicActiveSecrets[]|select(.name=="ROOTCA").secret.validationContext.trustedCa.inlineBytes'      sds.json | base64 -d > root.pem
 
# The private key is not in the dump: Envoy redacts it before the admin API prints it
jq -r '.dynamicActiveSecrets[]|select(.name=="default").secret.tlsCertificate.privateKey.inlineBytes' sds.json | base64 -d
 
# Does the leaf really chain to the root this proxy trusts?
openssl verify -CAfile root.pem -untrusted chain.pem chain.pem
 
# Identity lives in the URI SAN, not in the subject
openssl x509 -in chain.pem -noout -subject -issuer -dates -ext subjectAltName

[redacted]
 
chain.pem: OK
 
subject=
issuer=O = cluster.local
notBefore=Jul 21 09:11:42 2026 GMT
notAfter=Jul 22 09:11:42 2026 GMT
X509v3 Subject Alternative Name: critical
    URI:spiffe://cluster.local/ns/prod/sa/api

The subject line is empty, and that is deliberate. A web certificate binds identity to a hostname, in the common name or a DNS SAN (subject alternative name: the field listing the names a certificate is good for). An Istio certificate leaves the subject blank and puts the identity in a URI SAN. Because the subject is empty, RFC 5280 requires that SAN to be marked critical, which is exactly what you see, and a critical extension is one a verifier must either understand or reject the certificate over. Policy therefore keys off the SPIFFE ID and nothing else. Look at the issuer too: O = cluster.local is the same name as the root, confirming that on a default install the root signs leaves directly. Now notice what is absent. There is no revocation anywhere in the picture. Istio publishes no CRL (certificate revocation list: the roll of certificates a CA has taken back) for workload certificates, and the leaves carry no pointer to one. Envoy will check a CRL if you hand it one yourself, but istiod does not produce one. If a leaf leaks, your choices are to wait out the TTL or rotate the signing key, which invalidates everything at once.

Move the master die out of the cluster

The fix is a plug-in CA. You generate the root yourself, keep its private key offline in an HSM (hardware security module: a tamper-resistant box that signs on request and never releases the key) or in Vault, and hand istiod nothing but a short-lived intermediate. Three things improve at once. The key that can forge anything is no longer in your cluster. The mesh chains into a PKI (public key infrastructure: the tree of CAs and certificates your organisation already runs) that auditors and legacy systems can follow. And two clusters given intermediates cut from the same root can validate each other's workloads, which is the groundwork the federation lesson builds on.

# 1. The offline root. In production this key is generated inside an HSM and
#    never written to a laptop; --no-password --insecure is a lab shortcut.
step certificate create "Example Org Root CA" root-cert.pem root-key.pem \
  --profile root-ca --not-after 87600h --no-password --insecure
 
# 2. The per-cluster intermediate istiod signs leaves with. One year, not ten.
step certificate create "Istio CA - cluster.local" ca-cert.pem ca-key.pem \
  --profile intermediate-ca --not-after 8760h \
  --ca root-cert.pem --ca-key root-key.pem --no-password --insecure
 
# 3. istiod reads exactly these four filenames from a Secret named 'cacerts'
cat ca-cert.pem root-cert.pem > cert-chain.pem
kubectl create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem --from-file=ca-key.pem \
  --from-file=root-cert.pem --from-file=cert-chain.pem
 
kubectl rollout restart deploy/istiod -n istio-system

Your certificate has been saved in root-cert.pem.
Your private key has been saved in root-key.pem.
Your certificate has been saved in ca-cert.pem.
Your private key has been saved in ca-key.pem.
secret/cacerts created
deployment.apps/istiod restarted

Those filenames are not suggestions. istiod mounts that Secret at /etc/cacerts and looks for exactly four things: ca-cert.pem (the intermediate it signs with), ca-key.pem (that intermediate's private key), root-cert.pem (the anchor it distributes to every proxy), and cert-chain.pem (the intermediate followed by the root, so a verifier can walk the whole path). Misname one and the mount comes up short, istiod falls back to self-signing, and the only sign is a log line nobody reads. istiod does watch those files and can reload in place, but on a cluster where the Secret did not exist at install time, restart it and give yourself a timestamp to correlate against. Either way, verify rather than assume.

# Force one workload to re-issue against the new signing key
kubectl -n prod rollout restart deploy/api && kubectl -n prod rollout status deploy/api
 
# Print every certificate in the chain that proxy now presents
istioctl proxy-config secret deploy/api -n prod -o json \
 | jq -r '.dynamicActiveSecrets[]|select(.name=="default").secret.tlsCertificate.certificateChain.inlineBytes' \
 | base64 -d > chain.pem
openssl crl2pkcs7 -nocrl -certfile chain.pem | openssl pkcs7 -print_certs -noout
 
# istiod republishes the anchor into every namespace; confirm workloads see it
kubectl -n prod get cm istio-ca-root-cert \
  -o jsonpath='{.data.root-cert\.pem}' | openssl x509 -noout -subject -dates

deployment.apps/api restarted
deployment "api" successfully rolled out
 
subject=
issuer=CN = Istio CA - cluster.local
 
subject=CN = Istio CA - cluster.local
issuer=CN = Example Org Root CA
 
subject=CN = Example Org Root CA
issuer=CN = Example Org Root CA
 
subject=CN = Example Org Root CA
notBefore=Jul 21 10:02:11 2026 GMT
notAfter=Jul 18 10:02:11 2036 GMT

Three links now, and the top one is a certificate whose private key is not in the cluster at all. The istio-ca-root-cert ConfigMap is worth knowing on its own: istiod writes the current anchor into every namespace, which is how a starting proxy and the various webhook clients bootstrap trust before they hold anything of their own. If that ConfigMap still shows the old subject after a change, your rollout did not take.

Linkerd: same shape, a different landmine

Linkerd splits the same job across two pieces of material. A trust anchor you supply lives in the linkerd-identity-trust-roots ConfigMap, and an issuer certificate plus its key lives in the linkerd-identity-issuer Secret, which the identity controller uses to sign 24-hour workload certificates. The trap is lifetime. Whether you take the certificates linkerd install generates for you or mint your own following the docs, the issuer is usually the short-lived piece, valid for one year, and nothing renews it unless you wire in cert-manager (a Kubernetes controller that issues and rotates certificates on a schedule). Once it expires, the identity controller keeps signing and every peer starts rejecting the result, so within a single issuance lifetime the whole data plane stops handshaking. linkerd check --proxy exists largely to shout about this, and it starts shouting 60 days out.

linkerd check --proxy

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2026-08-30T10:14:22Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor
 
linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

That warning line is a dated outage notice. Run the check on a schedule and alert on the warning text, not only on the exit code, because a ‼ is not a failure and will not turn the exit code red for you. The failure mode is total, and the fix takes longer than the warning window if you start late.

Hand the CA to SPIRE

Istio can leave the signing business entirely. Point the sidecars at a SPIRE agent (SPIRE is the reference implementation of SPIFFE) and Envoy fetches its certificates from SPIRE's Workload API socket over the same SDS mechanism, so nothing about the wire format changes. What changes is who decides. SPIRE issues only to workloads that match a registration entry, which is a guest list at the door: a rule mapping platform-attested selectors, facts SPIRE checks for itself rather than takes your word for, to a SPIFFE ID. What it issues against that entry is an SVID (SPIFFE Verifiable Identity Document), either an X.509 certificate for mTLS or a signed JWT token for the places where you cannot use one. SPIRE can also root itself in an UpstreamAuthority plugin backed by Vault, AWS Private CA, or a cloud KMS (key management service), which means the signing key never lands in a Kubernetes Secret in the first place.

# The agent's own identity, attested by the k8s_psat node attestor
# (PSAT = projected service account token)
spire-server entry create \
  -node \
  -spiffeID spiffe://cluster.local/ns/spire/sa/spire-agent \
  -selector k8s_psat:cluster:prod-cluster \
  -selector k8s_psat:agent_ns:spire \
  -selector k8s_psat:agent_sa:spire-agent
 
# The workload entry, parented to that agent alias
spire-server entry create \
  -parentID spiffe://cluster.local/ns/spire/sa/spire-agent \
  -spiffeID spiffe://cluster.local/ns/prod/sa/api \
  -selector k8s:ns:prod \
  -selector k8s:sa:api \
  -selector k8s:pod-label:spiffe.io/spire-managed-identity:true \
  -dns api.prod.svc
 
# On the node, confirm the agent is attested and serving the Workload API
spire-agent healthcheck

Entry ID         : 7c05f3a2-1d64-4f0b-9a3e-2b18d5c40e71
SPIFFE ID        : spiffe://cluster.local/ns/spire/sa/spire-agent
Parent ID        : spiffe://cluster.local/spire/server
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s_psat:agent_ns:spire
Selector         : k8s_psat:agent_sa:spire-agent
Selector         : k8s_psat:cluster:prod-cluster
 
Entry ID         : 8f2c1a4e-6b0d-4c2e-9a11-2f7c9d4e01aa
SPIFFE ID        : spiffe://cluster.local/ns/prod/sa/api
Parent ID        : spiffe://cluster.local/ns/spire/sa/spire-agent
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:ns:prod
Selector         : k8s:pod-label:spiffe.io/spire-managed-identity:true
Selector         : k8s:sa:api
DNS name         : api.prod.svc
 
Agent is healthy.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: prod
spec:
  template:
    metadata:
      labels:
        app: api
        # satisfies the k8s:pod-label selector on the SPIRE entry
        spiffe.io/spire-managed-identity: "true"
      annotations:
        # 'spire' mounts the SPIFFE CSI volume holding the Workload API socket
        inject.istio.io/templates: "sidecar,spire"
    spec:
      serviceAccountName: api
      containers:
        - name: api
          image: registry.example.com/api:1.9.3

Two lines do the work. The label satisfies the k8s:pod-label selector on the registration entry, so SPIRE hands this pod the api identity and no other. The annotation adds Istio's spire injection template, which mounts the SPIFFE CSI (container storage interface) driver volume so Envoy can reach the agent's socket. Roll it out and read the issuer, because that is the only visible difference.

istioctl proxy-config secret deploy/api -n prod -o json \
 | jq -r '.dynamicActiveSecrets[]|select(.name=="default").secret.tlsCertificate.certificateChain.inlineBytes' \
 | base64 -d | openssl x509 -noout -subject -issuer -ext subjectAltName

subject=C = US, O = SPIRE
issuer=C = US, O = SPIFFE
X509v3 Subject Alternative Name:
    DNS:api.prod.svc, URI:spiffe://cluster.local/ns/prod/sa/api

Same SPIFFE ID, different signer. Your AuthorizationPolicy resources do not change, your applications do not change, and the handshake looks identical on the wire. Two details differ from the Istio-signed version, and both are cosmetic. SPIRE fills in a subject where Istio leaves it blank, and that subject carries no identity meaning at all: every X.509-SVID in the trust domain gets the same O = SPIRE, so policy still keys off the URI SAN and nothing else. And because the subject is not empty here, RFC 5280 no longer forces the SAN to be critical, so it is not. Only the office that stamps the document moved.

What this costs you

The bill arrives in three parts. Availability first: the mesh CA is now a hard dependency for starting anything at all. Short leaves make a stolen certificate cheap and a control-plane outage expensive, and you cannot have both. Somewhere between a few hours and a day is where most teams land. Custody second: an offline root means a real key ceremony, an HSM budget, and people who know where the backup lives and have proved it still restores. Rehearsal third: root rotation is the operation nobody has ever performed, attempted for the first time under pressure at 3am. Run it in staging on a cadence, using the union-bundle sequence, so the first production one is boring.

Two things are worth doing this week whatever you decide about the rest. Run kubectl auth can-i get secrets -n istio-system --as system:serviceaccount:<ns>:<sa> against every service account with any business near that namespace, and strip the ones that answer yes. Then put an expiry alert on whichever CA material you actually have, istio-ca-secret or cacerts or the Linkerd issuer, and fire it 60 days out rather than 7. The most common CA incident in production is not theft. It is a certificate nobody was watching quietly reaching its notAfter date on a Saturday.

Try this

Run kubectl -n istio-system get secret istio-ca-secret cacerts --ignore-not-found on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: swap the root without an overlap and the whole mesh drops at once. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related