SPIRE & attestation

Node + workload attestation, the Workload API.

A pod starts. It holds no key, no token, no password, and within a few milliseconds it has to prove to another service that it really is the payments API. The tempting fix is to bake in a bootstrap credential: an API key, a certificate, a token in a Kubernetes Secret. Now that credential has to be created, delivered, stored, rotated, and kept out of your Git history and your container images. Steal it and you *are* the payments service. That is the secret-zero problem, the credential you need before you can get credentials, and it quietly defeats every scheme that begins with "inject a key".

SPIRE, the SPIFFE Runtime Environment, refuses to hand out that first secret at all. (SPIFFE, the Secure Production Identity Framework For Everyone, is the standard; SPIRE is the production implementation of it.) It works like the badge desk on your first day at a job. Nobody asks you for a password, because you do not have one yet. The desk checks facts it can confirm without your help: HR's record says someone with your name starts today, on the fourth floor, in the finance team, and the manager standing next to you is on the list of people allowed to sign someone in. Verified facts, matched against a rule written down in advance. SPIRE calls that attestation, and it happens twice: once for the machine, then once for the process running on it.

Two moving parts and nothing shared

spire-server is the certificate authority (the party that signs certificates and vouches for them) and the rulebook. It signs identities and stores the registration entries that decide which workload gets which name. Behind it sits a database: SQLite for a laptop, replicated PostgreSQL or MySQL for anything real. spire-agent runs once per machine, as a Kubernetes DaemonSet (one pod on every node) or a systemd unit on a virtual machine. It attests the workloads on its own node and hands each one its identity through the Workload API, a gRPC service (gRPC is a remote-call protocol carried over HTTP/2) exposed on a Unix-domain socket, which is a special file that only processes on the same host can open. The identity document itself is an SVID, a SPIFFE Verifiable Identity Document, which is either an X.509 certificate or a signed token. No shared secret is ever shipped to a workload. Every decision comes from facts the platform can check by itself.

Stage one: proving the machine

The server has to trust the agent before the agent can vouch for anything. Skip that step and a rogue agent on an attacker's laptop mints whatever identity it likes. Node attestation is the agent presenting evidence about the machine it runs on, produced by something other than the agent itself. Think of a guard checking the delivery van's paperwork against the courier company rather than against what the driver claims. On AWS that evidence is the signed instance identity document the platform publishes about the virtual machine; on GCP an instance identity token; on Azure a managed-identity token. On bare metal you have a credential rooted in the TPM (Trusted Platform Module, the tamper-resistant chip soldered onto the board) through the tpm_devid attestor, proof that you hold a private key the server already knows (x509pop, short for proof of possession), or a one-time join token you carry over by hand. On Kubernetes it is a projected service account token, which SPIRE calls PSAT.

A projected token is a short-lived JWT (JSON Web Token: a bundle of claims with a signature over them) that the kubelet, the Kubernetes agent running pods on each node, requests from the API server on behalf of one specific pod, stamped with one specific audience, and rewrites into the pod's filesystem before it expires. The SPIRE server checks it with a TokenReview call back to the Kubernetes API, which answers: this token is valid for audience spire-server, and it belongs to service account spire-agent in namespace spire, in pod spire-agent-4rj9d. The server then reads that pod to learn which node it sits on, so it needs permission to create TokenReviews and to read pods and nodes. SPIRE's older k8s_sat attestor used the legacy service account token, which never expired, named no audience, and could not be tied to a node, so a copy lifted from any pod stayed replayable. It was deprecated years ago and removed outright in SPIRE 1.11.0. Current releases ship k8s_psat and nothing else.

agent {
    data_dir          = "/run/spire"
    log_level         = "INFO"
    server_address    = "spire-server.spire.svc.cluster.local"
    server_port       = "8081"
    socket_path       = "/run/spire/agent-sockets/spire-agent.sock"
    trust_domain      = "acme.internal"
    trust_bundle_path = "/run/spire/bundle/bundle.crt"
}
 
plugins {
    # How this node proves what it is. No secret is embedded here.
    NodeAttestor "k8s_psat" {
        plugin_data {
            cluster = "acme-cluster"
            # token_path defaults to /var/run/secrets/tokens/spire-agent
        }
    }
 
    # How this agent identifies the processes that call the Workload API.
    WorkloadAttestor "k8s" {
        plugin_data {
            # Defaults are sane here: kubelet_secure_port 10250 and
            # kubelet_ca_path /run/secrets/kubernetes.io/serviceaccount/ca.crt
        }
    }
 
    WorkloadAttestor "unix" {
        plugin_data {
            # Off by default. Without it you get no unix:path and no unix:sha256.
            discover_workload_path = true
        }
    }
 
    KeyManager "memory" {
        plugin_data {}
    }
}

Three settings there earn their keep. token_path defaults to /var/run/secrets/tokens/spire-agent, which is exactly where the DaemonSet below projects the token, so those two files have to agree. trust_bundle_path is the server's root certificate, the specimen signature the agent keeps on file so it can recognise the real server; without it the agent will not start a conversation. There is an insecure_bootstrap = true switch that skips the check and trusts whatever answers on first contact, which is fine for a ten-minute demo and a real hole anywhere else, because first contact is precisely when an attacker would like to be the server. And KeyManager "memory" keeps the agent's own private key in RAM only, so a restarted agent proves itself from scratch instead of reusing an old key.

# excerpt: the kubelet mints and refreshes the PSAT for us
spec:
  template:
    spec:
      hostPID: true          # the agent must see caller PIDs in the host namespace
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccountName: spire-agent
      containers:
        - name: spire-agent
          image: ghcr.io/spiffe/spire-agent:1.15.2
          args: ["-config", "/run/spire/config/agent.conf"]
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-agent-socket
              mountPath: /run/spire/agent-sockets
            - name: spire-token
              mountPath: /var/run/secrets/tokens
      volumes:
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/agent-sockets
            type: DirectoryOrCreate
        - name: spire-token
          projected:
            sources:
              - serviceAccountToken:
                  path: spire-agent          # -> /var/run/secrets/tokens/spire-agent
                  audience: spire-server     # useless against any other API
                  expirationSeconds: 7200

hostPID: true is not decoration. The agent has to see calling processes in the host's process namespace, otherwise it cannot map a process ID to a container later on. The serviceAccountToken volume is the whole PSAT mechanism in six lines: the kubelet mints the token, keeps it fresh, and nobody stores a copy anywhere. On the server side you name the service accounts allowed to attest as agents, which is the control that stops any random pod in the cluster from claiming to be one.

plugins {
    NodeAttestor "k8s_psat" {
        plugin_data {
            clusters = {
                "acme-cluster" = {
                    # Only this namespace:serviceaccount may attest as an agent.
                    service_account_allow_list = ["spire:spire-agent"]
                    # ["spire-server"] is already the default; spelled out on purpose.
                    audience                   = ["spire-server"]
                    allowed_node_label_keys    = ["tier"]
                }
            }
        }
    }
}

Start the agent, then confirm the server actually accepted it. If this list comes back empty, nothing downstream works and no workload on that node can get an identity.

kubectl -n spire exec deploy/spire-server -- \
  /opt/spire/bin/spire-server agent list

Found 1 attested agent:
 
SPIFFE ID         : spiffe://acme.internal/spire/agent/k8s_psat/acme-cluster/8f3c1a94-6d02-4d1e-9a55-0b7c2f1d3e88
Attestation type  : k8s_psat
Expiration time   : 2026-07-21 18:22:41 +0000 UTC
Serial number     : 187262158695625563943432523535754060389
Can re-attest     : true
Agent version     : 1.15.2

Read that identity from left to right. k8s_psat is the attestor that produced it, acme-cluster is the cluster name from the config, and the last segment is the Kubernetes Node object's UID (unique ID), so the agent's name is pinned to one machine. It sits under the reserved /spire/agent/... path, which no workload can ever be given, because SPIRE rejects registration entries under /spire. Can re-attest: true means the agent may present fresh PSAT evidence when its own certificate runs out, which is what lets a node reboot at 3am without a human. When you retire a node, spire-server agent evict -spiffeID <agent id> drops the record and lets it attest again if it comes back; spire-server agent ban -spiffeID <agent id> makes sure it cannot.

Node aliases: stop hard-coding agent IDs

Pointing every workload entry at one agent's SPIFFE ID works right up to the moment the autoscaler adds a node. Then your entry is parented to a machine the pod is not on, and the pod gets nothing. A node alias is a staff parking permit rather than a permit for one number plate: a second name for a *set* of nodes, chosen by node selectors instead of by identity. Create it once, parent your workload entries onto it, and new machines inherit the arrangement the moment their agent attests.

# One name for every node whose agent attested through this cluster's PSAT
spire-server entry create \
  -node \
  -spiffeID spiffe://acme.internal/k8s/acme-cluster/worker \
  -selector k8s_psat:cluster:acme-cluster \
  -selector k8s_psat:agent_ns:spire \
  -selector k8s_psat:agent_sa:spire-agent

Entry ID                : 2f31f5f2-3f77-4b1f-8a9e-b6a3a17c2f10
SPIFFE ID               : spiffe://acme.internal/k8s/acme-cluster/worker
Parent ID               : spiffe://acme.internal/spire/server
Revision                : 0
X509-SVID TTL           : default
JWT-SVID TTL            : default
Selector                : k8s_psat:agent_ns:spire
Selector                : k8s_psat:agent_sa:spire-agent
Selector                : k8s_psat:cluster:acme-cluster

Look at the parent of the alias itself: spiffe://acme.internal/spire/server. Alias entries hang off the server, not off any agent. The trade-off is real, though. A broad alias means the payments identity can be issued on any node in the cluster. If payments may only run on your card-data machines, add a node label selector such as k8s_psat:agent_node_label:tier:pci, and note that the server only emits label selectors for keys you listed in allowed_node_label_keys. Forget that and your carefully written entry matches nothing at all, silently.

Registration entries are the rules

An entry says one thing: a workload matching all of these selectors, under this parent, receives this SPIFFE ID. It is the line on the door list, and the guard reads it literally. Selectors are facts the agent verifies locally, never claims the workload makes about itself. On Kubernetes you get k8s:ns, k8s:sa, k8s:pod-name, k8s:pod-uid, k8s:pod-label:<key>:<value>, k8s:pod-owner, k8s:container-name, k8s:container-image and k8s:node-name, among others. On plain Linux the unix attestor always gives you unix:uid, unix:gid and their name equivalents, and it adds unix:path and unix:sha256 (the hash of the binary on disk) only when you set discover_workload_path = true, which is off by default and catches people out. None of these is a password, and none of them lives inside your image.

spire-server entry create \
  -parentID spiffe://acme.internal/k8s/acme-cluster/worker \
  -spiffeID spiffe://acme.internal/ns/prod/sa/payments \
  -selector k8s:ns:prod \
  -selector k8s:sa:payments \
  -selector k8s:container-name:payments \
  -x509SVIDTTL 1800 \
  -jwtSVIDTTL 120

Entry ID                : 4d9e2f7a-6b18-4c2a-9f01-2ad7c5e9b310
SPIFFE ID               : spiffe://acme.internal/ns/prod/sa/payments
Parent ID               : spiffe://acme.internal/k8s/acme-cluster/worker
Revision                : 0
X509-SVID TTL           : 1800
JWT-SVID TTL            : 120
Selector                : k8s:container-name:payments
Selector                : k8s:ns:prod
Selector                : k8s:sa:payments

-x509SVIDTTL 1800 gives the certificate thirty minutes of validity, half the shipped default. -jwtSVIDTTL 120 gives tokens two minutes, because a token that travels through gateways and proxies deserves a shorter life than a certificate that never leaves the machine. TTL means time to live, how long the document counts as valid before it is refused. Leave both flags off and the server's default_x509_svid_ttl (1h) and default_jwt_svid_ttl (5m) apply, and the entry prints default where the numbers would be. One clamp to remember: SPIRE will not issue an SVID that outlives the CA certificate signing it, so late in a CA's 24-hour life a generous entry TTL comes back trimmed, with a warning in the server log rather than an error in your terminal.

Stage two: the Workload API

The Workload API is a service window set into the wall of the node. Any process on that machine can knock. It shows nothing, hands over nothing, and asks a single question: what is my identity? Everything the window needs, it learns by inspecting the caller instead of listening to it.

Here is the machinery. When a process connects to the socket, the agent asks the kernel who is on the other end (SO_PEERCRED, a socket option that returns the caller's process ID, user ID and group ID). Those numbers come from the kernel rather than from the message, so the caller cannot lie about them. Workload attestor plugins then turn that PID into selectors. The unix plugin reads /proc/<pid> for the user and group, plus the binary path and its hash if you enabled that. The k8s plugin works out which container owns the PID, from the process's cgroup on cgroup v1 hosts and from /proc/<pid>/mountinfo on cgroup v2 hosts (that second path is what use_new_container_locator does, and it defaults to true in current releases), then asks the local kubelet which pod owns that container, which yields the namespace, service account, container name and image.

The agent matches those selectors against the entries cached for its own node. Every entry whose selectors all match produces an SVID: the agent generates the key pair on the node, sends a certificate signing request (CSR) to the server, and returns the signed certificate, the private key and the trust bundle back over the socket. Two things there are easy to miss. The agent's cache is keyed by registration entry, so two pods on the same node that match the same entry receive the same certificate and the same private key; the identity belongs to the entry, not to the process. And the call is a stream rather than a one-shot request, so the agent pushes a replacement down the same connection before the current SVID expires. Client libraries find the socket through the SPIFFE_ENDPOINT_SOCKET environment variable and send workload.spiffe.io: true as a gRPC header on every request. The workload never wrote a CSR, never held a bootstrap secret, and never had to keep a key on disk.

# Run inside the payments container (the SPIRE CLI is in the image for this walkthrough).
# Nothing is presented: no token, no key, no password.
spire-agent api fetch x509 \
  -socketPath /run/spire/agent-sockets/spire-agent.sock \
  -write /run/spire/svids

Received 1 svid after 6.245ms
 
SPIFFE ID:              spiffe://acme.internal/ns/prod/sa/payments
SVID Valid After:       2026-07-21 17:20:41 +0000 UTC
SVID Valid Until:       2026-07-21 17:50:51 +0000 UTC
CA #1 Valid After:      2026-07-21 08:00:00 +0000 UTC
CA #1 Valid Until:      2026-07-22 08:00:00 +0000 UTC
 
Writing SVID #0 to file /run/spire/svids/svid.0.pem.
Writing key #0 to file /run/spire/svids/svid.0.key.
Writing bundle #0 to file /run/spire/svids/bundle.0.pem.

When a workload gets nothing

You copy the recipe for a second service, ledger, and it does not work. There is a single error for every kind of failure here, which is either merciful or infuriating depending on the hour.

# Same command, from inside the ledger pod
spire-agent api fetch x509 -socketPath /run/spire/agent-sockets/spire-agent.sock

rpc error: code = PermissionDenied desc = no identity issued

no identity issued means the agent found no entry whose selectors all matched this caller. Three things cause it: the entry is parented to a node the pod is not running on, a selector value does not match reality, or the pod is not shaped the way you assumed. So put the rule and the facts side by side.

# The rule
spire-server entry show -spiffeID spiffe://acme.internal/ns/prod/sa/ledger
 
# The facts
kubectl -n prod get pod ledger-7d9f6c8b5-xk2wq \
  -o jsonpath='sa: {.spec.serviceAccountName}{"\n"}{range .spec.containers[*]}container: {.name}{"\n"}{end}'

Found 1 entry
Entry ID                : b71c4c0e-9a2d-4a55-8f0e-1c6b7d2e5a94
SPIFFE ID               : spiffe://acme.internal/ns/prod/sa/ledger
Parent ID               : spiffe://acme.internal/k8s/acme-cluster/worker
Revision                : 0
X509-SVID TTL           : 1800
JWT-SVID TTL            : 120
Selector                : k8s:container-name:ledger
Selector                : k8s:ns:prod
Selector                : k8s:sa:ledger
 
sa: ledger
container: ledger-api
container: istio-proxy

There it is. The entry demands k8s:container-name:ledger, the container is called ledger-api, and one failing selector out of three fails the whole match. Repair it with spire-server entry update -entryID b71c4c0e-..., and remember that update is a full replacement rather than a patch: every selector and flag you want to keep has to appear again in the command, or it disappears and the revision number climbs. Agents re-sync their entries from the server every few seconds (sync_interval under the agent's experimental block, 5 seconds by default), so the workload picks up its identity on the next call with no restart.

What attestation proves, and what mTLS does not

Attestation proves platform-observable facts about a process at the instant it asked: which node it runs on, which pod and namespace, which service account, which container and image. It does not prove that the code inside is the code you reviewed, that the image carries no backdoor, or that the process was not taken over an hour ago. If an attacker gets execution inside the payments container, they can open the socket and receive the payments SVID, because from the platform's point of view they *are* payments.

Carrying that identity onto the wire with mTLS (mutual TLS, where both ends of a connection present a certificate instead of only the server) proves one narrow thing very well: whoever is on the other end holds the private key for that SVID, and that SVID was signed by a CA in your trust bundle. Everything else is out of scope. mTLS does not say which human or customer the request is for, whether this caller is allowed to perform this particular operation, whether the caller is compromised, or what happens to the data afterwards. Those are authorization questions and they live in policy: in Istio, an AuthorizationPolicy (API group security.istio.io/v1) matching source.principals such as acme.internal/ns/prod/sa/frontend, backed by a PeerAuthentication with mtls.mode: STRICT so plaintext connections are refused outright. Identity is the input to that policy, never a replacement for it.

Verify the document you got

An SVID is an ordinary X.509 certificate, so openssl reads it. Two things have to be true. The URI SAN (Subject Alternative Name, the certificate field that carries alternative names such as DNS names and URIs) equals the SPIFFE ID you expect, and it is the only URI SAN, because a SPIFFE identity lives in that field and nowhere else. SPIRE fills the Subject with a fixed placeholder instead, C=US, O=SPIRE on workload certificates and C=US, O=SPIFFE on the CA, which is exactly why reading the Common Name tells you nothing worth having. And the validity window should be short.

openssl x509 -in /run/spire/svids/svid.0.pem -noout -ext subjectAltName
openssl x509 -in /run/spire/svids/svid.0.pem -noout -subject
openssl x509 -in /run/spire/svids/svid.0.pem -noout -dates

X509v3 Subject Alternative Name:
    URI:spiffe://acme.internal/ns/prod/sa/payments
subject=C = US, O = SPIRE
notBefore=Jul 21 17:20:41 2026 GMT
notAfter=Jul 21 17:50:51 2026 GMT

Running it for real

In production the server runs as a small cluster of replicas over a shared SQL datastore, because it is your certificate authority. While it is unreachable, existing SVIDs keep working until they expire, and nothing new can attest. That is the honest price of short lifetimes: a server outage becomes an availability incident on a stopwatch, and availability_target in the agent config is the knob that buys you a longer cushion. Agents scale with node count rather than pod count, since each one attests only its local workloads. Inside a service mesh you rarely call the Workload API by hand, because Envoy pulls certificates from the agent over SDS (Secret Discovery Service, the way Envoy asks for keys and certificates at runtime) on that same socket, and SPIRE can act as the mesh's CA so one trust domain covers several clusters and clouds. The recurring cost is the datastore plus the discipline of keeping entries accurate. Stale, over-broad entries are security debt that nothing alerts on.

After every node rollout, compare spire-server agent list against your node count. A missing agent is not a slow degradation you can look at next sprint: every workload on that machine is running with no identity at all right now, and the ones still holding a certificate are on a thirty-minute clock. What happens as that clock runs down, and what breaks when rotation stalls, is the next lesson, SVIDs & rotation.

Try this

Run /opt/spire/bin/spire-server agent list on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: loose selectors hand out the wrong identity. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related