Trust-domain federation

Exchange bundles; multi-cluster, multi-cloud, B2B.

A German passport gets waved through at a Tokyo border desk because the two governments swapped the public certificates they use to sign passports. No printing plates changed hands. No signing key left its vault. Each country still issues its own documents, in its own building, on its own terms, and the other side only learned how to recognise a genuine one. Trust-domain federation is that same arrangement, rebuilt for software.

Three words carry this lesson, so here they are in plain terms first. A trust domain is one identity namespace with one certificate authority (CA, the service that signs certificates) standing behind it: a cluster, a cloud account, a whole company. Names inside it look like spiffe://acme.internal/ns/prod/sa/checkout. An SVID (SPIFFE Verifiable Identity Document, where SPIFFE is the Secure Production Identity Framework For Everyone) is the short-lived credential a workload presents to prove one of those names really belongs to it. A trust bundle is the paperwork the border desk needs: the public root certificates a domain signs its X.509 SVIDs with (X.509 being the ordinary certificate format, here carrying a SPIFFE name), plus the public keys it signs its JWT-SVIDs with (JWT, JSON Web Token, a signed blob of text rather than a certificate). Everything in a bundle is public. Nothing in it is a secret the signer has to guard. Federation is the automated, repeating exchange of those bundles between domains that otherwise stay completely independent, each running its own SPIRE server (SPIRE, the SPIFFE Runtime Environment, is the software that issues and rotates SVIDs) and its own CA.

What Crosses the Boundary, and What Never Does

When acme.internal and partner.io federate, acme imports partner's bundle and partner imports acme's. That is the entire exchange. Acme's private signing key stays inside acme, on acme's hardware, under acme's control. If partner.io is breached tomorrow, the attacker inherits partner's power to mint partner names and nothing else. They cannot forge spiffe://acme.internal/ns/prod/sa/payments, because they have never touched the key that signs it.

The tempting shortcut is one shared root across both organisations, or one flattened trust domain spanning every cluster. Less config, fewer moving parts, and it does work. It also means one compromised CA forges identities on both sides at once, which erases the boundary the whole zero-trust design exists to draw. Federation keeps two CAs that distrust each other by default and teaches each one to recognise the other's signature. Same connectivity, very different blast radius.

Bundles carry a version stamp, the way a printed parts catalogue carries an edition number, so a consumer can tell a fresh copy from a stale one. SPIRE stamps each bundle with a spiffe_sequence that increments whenever the contents change, and a spiffe_refresh_hint in seconds telling peers how often to come back. Several things have to line up before one workload can make a mutual-TLS call (mTLS, where both sides present a certificate instead of only the server) across the line.

The Bundle Endpoint, Because a File Copy Rots

Roots rotate. A bundle somebody emailed you in March is worthless the week partner's CA turns over, and every cross-domain call starts failing at certificate verification. So federation cannot be a one-time copy. Each domain runs a bundle endpoint: a small HTTPS server publishing that domain's current public bundle, which peers poll on a schedule. It is the country publishing its passport-signing certificates on a page every border desk refreshes automatically, instead of mailing a disc once and hoping.

Two authentication profiles exist for that endpoint, and the choice matters more than it looks. https_web protects it with an ordinary web certificate from a public CA, the same kind your browser already trusts, so a peer needs no prior state to fetch safely. https_spiffe protects it with the endpoint's own SVID, which drops any dependency on the public certificate ecosystem but creates a chicken-and-egg problem: you cannot validate that SVID until you already hold the domain's bundle.

server {
    trust_domain          = "acme.internal"
    bind_address          = "0.0.0.0"
    bind_port             = "8081"
    data_dir              = "/run/spire/data"
    ca_ttl                = "24h"
    default_x509_svid_ttl = "1h"
 
    federation {
        # Publish OUR public bundle so partner.io can fetch it
        bundle_endpoint {
            address      = "0.0.0.0"
            port         = 8443
            refresh_hint = "5m"
            profile "https_spiffe" {}
        }
 
        # Fetch partner.io's public bundle on a schedule
        federates_with "partner.io" {
            bundle_endpoint_url = "https://spire.partner.io:8443"
            bundle_endpoint_profile "https_spiffe" {
                endpoint_spiffe_id = "spiffe://partner.io/spire/server"
            }
        }
    }
}
 
# plugins { ... } datastore, node attestors, key manager omitted

bundle_endpoint publishes, federates_with consumes, and the two are independent. A domain can publish without consuming anything, which is normal when you are the service provider and your customers are the ones fetching. refresh_hint is advice you hand peers, not something you enforce; a peer that ignores it goes stale on its own schedule and you will never know. The line that earns its keep is endpoint_spiffe_id. It pins which identity is allowed to answer at that URL, so a hijacked DNS record (DNS, the Domain Name System, the phone book that turns names into addresses) or a stolen IP cannot feed you a forged bundle unless the impostor also holds a partner-signed SVID for that exact name. If you pick https_web instead, swap the publisher block for profile "https_web" with either an acme section for automatic certificates or a serving_cert_file section pointing at a cert you manage.

Bootstrapping the First Exchange

With https_spiffe, the very first bundle has to arrive by some route other than the endpoint. A file over a channel both sides already trust, a fingerprint read aloud on a call, a runbook step with two names on it. This is the one manual moment in the whole design, and the one worth slowing down for, because everything downstream inherits its correctness. Hand someone the wrong bundle here and you have federated with an impostor, and every automated check afterwards passes with a clean green tick.

# What we publish: acme's own public bundle, in SPIFFE format
# (a JWKS, JSON Web Key Set, which is a list of public keys as JSON)
spire-server bundle show -format spiffe \
  -socketPath /run/spire/server/private/api.sock

{
    "keys": [
        {
            "use": "x509-svid",
            "kty": "EC",
            "crv": "P-256",
            "x": "fA9J8mQ1tR7bZ0xYq3nL5hVd2gKcS8pWuE6oT1aXjMk",
            "y": "kP2mD1r7ZsN4vB9cQ0eH6tL3yXfA8uJgW5iR2dOnTbY",
            "x5c": [
                "MIIB9zCCAZ2gAwIBAgIQb3nR1kYc9sVpQ2mLxTfEZDAKBggqhkjOPQQDAjA...pQ=="
            ]
        },
        {
            "use": "jwt-svid",
            "kty": "EC",
            "crv": "P-256",
            "x": "ZQ8v1nR4tCmB7yUwK0dLxH3sVfE9gPjA2oTiN5bXcRq",
            "y": "T1sK9dGuP4hMzX0cVbE7rY2fL8nQaJwO6iRt3ZpBySk",
            "kid": "C6vs25welZOx6WksNYfbMfiw9l6Rd3iA"
        }
    ],
    "spiffe_sequence": 7,
    "spiffe_refresh_hint": 300
}

Two kinds of key come out. The x509-svid entry is a root that validates certificates during an mTLS handshake, with the actual certificate carried in x5c as base64 text. The jwt-svid entry is a public key for validating JWT-SVIDs, so federation covers token-carrying calls as well as connection-level ones. spiffe_sequence: 7 says this bundle has been revised seven times, and that number is how a peer spots a copy that has quietly stopped moving. Note what is missing: no private key, no d field, nothing you would mind reading out in a public meeting.

# Partner's bundle arrived out of band and was verified by a human on both ends
spire-server bundle set -format spiffe \
  -id spiffe://partner.io \
  -path /tmp/partner.bundle.json
 
# Every federated bundle we now hold. `bundle list` shows OTHER trust domains
# only; our own bundle lives behind `bundle show`.
spire-server bundle list -format pem

bundle set.
****************************************
* spiffe://partner.io
****************************************
-----BEGIN CERTIFICATE-----
MIIBvDCCAWKgAwIBAgIRAJ8kR2mVpQ1xN7bLcTfEZ0YwCgYIKoZIzj0EAwIwHjEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBlNQSUZGRTAeFw0yNjA3MjEwMjU5NTBaFw0y
NjA3MjIwMzAwMDBaMB4xCzAJBgNVBAYTAlVTMQ8wDQYDVQQKEwZTUElGRkUwWTAT
-----END CERTIFICATE-----

That import is what makes the first automated fetch possible. From here SPIRE polls https://spire.partner.io:8443, verifies the endpoint's SVID against the copy you pasted in, and swaps in a new bundle whenever partner publishes a higher sequence. The same relationship is also available through the API instead of the config file. spire-server federation create, federation list, federation show, federation update and federation delete add or drop a peer with no server restart, -trustDomainBundlePath on the create command does the bootstrap and the relationship in one step, and spire-server federation refresh -id spiffe://partner.io forces a poll right now when you are trying to work out whether the endpoint is answering at all.

Deciding Which Workloads Learn the Peer's Roots

A visa is not issued to a country. It is issued to a person. Registration entries do that job here: holding partner's bundle on the SPIRE server does not mean every pod in your cluster gets a copy. A workload receives a peer's roots only if its registration entry carries -federatesWith for that trust domain. Leave the flag off and that workload's handshake to a partner service dies at certificate verification, even though the bundle is sitting right there on the server.

That is deliberate, and it is the cheapest containment control in this lesson. Cross-domain trust stays scoped to the two or three services that genuinely need it, instead of being taught to every workload in the cluster. When a random batch job is compromised, it cannot dial a partner service at all, because it was never handed the roots that would let it verify one.

spire-server entry create \
  -spiffeID  spiffe://acme.internal/ns/prod/sa/checkout \
  -parentID  spiffe://acme.internal/spire/agent/k8s_psat/prod-cluster/abcd1234 \
  -selector  k8s:ns:prod \
  -selector  k8s:sa:checkout \
  -federatesWith spiffe://partner.io \
  -x509SVIDTTL 3600

Entry ID                : 8c4f2b19-6d0e-4a37-9c58-1f7a2e5b3d64
SPIFFE ID               : spiffe://acme.internal/ns/prod/sa/checkout
Parent ID               : spiffe://acme.internal/spire/agent/k8s_psat/prod-cluster/abcd1234
Revision                : 0
X509-SVID TTL           : 3600
JWT-SVID TTL            : default
Selector                : k8s:ns:prod
Selector                : k8s:sa:checkout
FederatesWith           : spiffe://partner.io

# From inside the checkout pod: what did the Workload API actually hand us?
spire-agent api fetch x509 -socketPath /run/spire/sockets/agent.sock

Received 1 svid after 6.1327ms
 
SPIFFE ID:		spiffe://acme.internal/ns/prod/sa/checkout
SVID Valid After:	2026-07-21 09:12:03 +0000 UTC
SVID Valid Until:	2026-07-21 10:12:13 +0000 UTC
CA #1 Valid After:	2026-07-21 06:59:50 +0000 UTC
CA #1 Valid Until:	2026-07-22 07:00:00 +0000 UTC
[spiffe://partner.io] CA #1 Valid After:	2026-07-21 02:59:50 +0000 UTC
[spiffe://partner.io] CA #1 Valid Until:	2026-07-22 03:00:00 +0000 UTC

The bracketed lines are the payoff. Those are partner.io's roots, delivered to this one workload over the SPIFFE Workload API because its entry says FederatesWith. Notice the arithmetic while you are here: the SVID lives one hour, matching -x509SVIDTTL 3600, and the CA lives twenty-four hours, matching ca_ttl. Remove the flag and the partner lines vanish on the agent's next sync, without waiting for any certificate to expire, because the Workload API is a stream the agent pushes updates onto rather than something the workload polls.

Proving the Handshake Really Works

Config that reads correctly still has to survive a real connection. Drive one by hand from a debug pod, presenting your own SVID and trusting only the partner root the agent handed you, so nothing in the operating system's default trust store can quietly rescue a broken setup. The -write flag drops everything the Workload API returned onto disk, and -silent keeps stdout clean so you can see the filenames on their own.

mkdir -p /tmp/id
spire-agent api fetch x509 \
  -socketPath /run/spire/sockets/agent.sock \
  -write /tmp/id -silent
 
ls -1 /tmp/id

bundle.0.pem
federated_bundle.0.0.pem
svid.0.key
svid.0.pem

bundle.0.pem is your own trust domain's roots. federated_bundle.0.0.pem is partner's, written only because this entry federates. That second file is the one to point at, because using it proves the roots reached the workload rather than proving they exist somewhere on a server the pod cannot even reach.

# Real mTLS to the partner service, trusting ONLY partner's root
openssl s_client -connect orders.partner.io:8443 \
  -cert /tmp/id/svid.0.pem \
  -key  /tmp/id/svid.0.key \
  -CAfile /tmp/id/federated_bundle.0.0.pem \
  </dev/null 2>/dev/null > /tmp/handshake.txt
 
grep -m1 'Verify return code' /tmp/handshake.txt
sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' /tmp/handshake.txt \
  | openssl x509 -noout -ext subjectAltName

    Verify return code: 0 (ok)
X509v3 Subject Alternative Name: 
    URI:spiffe://partner.io/ns/prod/sa/orders

Verify return code: 0 (ok) means partner's SVID chained to the root you were handed, so the federation plumbing is alive. The SAN URI (Subject Alternative Name, the certificate field that carries the identity) tells you which workload answered, which is often a surprise the first time you look. Two things to keep straight. OpenSSL did no hostname check here, and that is correct for SPIFFE, where the URI in the SAN is the identity and the DNS name is scaffolding. And if your SPIRE server sits under an upstream CA, svid.0.pem holds intermediate certificates as well as the leaf, and OpenSSL sends only the first one from -cert, so add -cert_chain or the far end will reject you for reasons that have nothing to do with federation. If instead you get Verify return code: 20 (unable to get local issuer certificate) or 21 (unable to verify the first certificate), your side is missing or holding a stale copy of partner's bundle. That is a refresh problem, and no amount of editing authorization policy will fix it.

The Handshake Proved a Name and Nothing Else

A clean verify proves exactly three things and then stops. The peer holds the private key matching the certificate it presented. That certificate chains to a root you decided to trust. The name written in its SAN is spiffe://partner.io/ns/prod/sa/orders. That is the whole list. It says nothing about whether that workload is compromised, whether the code inside it is the code you reviewed, whether this particular call should be permitted, or anything at all about the request body. And SPIFFE has no revocation list, so a key lifted out of that pod keeps verifying cleanly until the certificate expires. The one-hour lifetime is your revocation mechanism, which is why treating it as a tuning knob to relax is a bad trade.

The two credential types differ in a way worth holding onto. An X.509 SVID is proof of possession: the peer has to use the private key live during the handshake, so a copied certificate on its own buys an attacker nothing. A JWT-SVID is a bearer token, closer to a cinema ticket, where whoever holds it can present it. That is why JWT-SVIDs are bound to an audience and kept very short-lived. Both key types travel in the same bundle, so federating once covers connection-level calls and token-carrying calls together.

Federation quietly widens the set of identities that can complete a handshake with you. Yesterday only acme names verified. Today every valid partner.io SVID does. If partner runs two hundred services and one of them gets popped, that attacker's SVID passes your verification perfectly, because it is genuine. Authentication is behaving exactly as designed. The thing that stops the call is [identity-based authorization](/courses/zerotrust/zt-authz/), written against the specific identity you agreed to talk to.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-allow-partner-checkout
  namespace: prod
spec:
  selector:
    matchLabels:
      app: orders
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["partner.io/ns/prod/sa/checkout"]
    to:
    - operation:
        methods: ["POST"]
        paths: ["/v1/orders"]

A principal in Istio is the peer's SPIFFE ID with the spiffe:// scheme stripped, which is why a foreign trust domain slots straight in as partner.io/ns/prod/sa/checkout. The Istio rule doing the real work here is the one that is never written down: once any ALLOW policy selects a workload, every request to that workload is denied unless it matches at least one rule. Partner's other 199 identities get RBAC: access denied (RBAC, role-based access control) from the sidecar while holding perfectly valid certificates. Keep PeerAuthentication in STRICT mode alongside it so nothing reaches the workload in plaintext and skips the identity check entirely. If your mesh issues its own certificates rather than sourcing them from SPIRE, teach it about the peer's roots through meshConfig.caCertificates, which takes either an inline pem blob or a spiffeBundleUrl pointing at the peer's bundle endpoint. That fetcher expects a bundle endpoint served with a publicly trusted certificate, so it pairs with the https_web profile, not https_spiffe.

Three Deployment Shapes and What They Cost

The same primitive gives you three arrangements. Multi-cluster: each cluster is its own trust domain with its own SPIRE server, so cross-cluster calls are identity-checked rather than IP-checked, and taking over one cluster's CA does not mint names in the others. Multi-cloud: workloads on AWS, GCP and Azure each attest with their own platform's node attestor, then federate, so you stop writing bespoke cross-cloud credential glue for every new pairing. Cross-organisation (B2B, business to business): two companies let workloads call each other with mutual identity, instead of a shared API key sitting in a vault at both ends, or a flat site-to-site VPN that hands over the entire network.

The bill comes in three parts. Relationships grow as pairs, so N domains that all talk to each other need N(N-1)/2 of them, each with its own bootstrap, URL, policy set and pager rotation. Past a handful, pick one profile, automate the bootstrap in the same pipeline that provisions the SPIRE server, and hold one source of truth for who federates with whom. Offboarding is manual on both sides: spire-server bundle delete -id spiffe://partner.io -mode dissociate drops the bundle and strips the association from every entry that referenced it, where the default -mode restrict refuses to delete a bundle still in use and -mode delete removes those entries outright. And with no global revocation list anywhere in SPIFFE, the fastest way to cut a partner off during an incident is to delete the bundle and let their certificates stop verifying at your side, then tighten [egress](/courses/zerotrust/zt-egress/) behind it.

# Is our copy of partner's bundle still tracking theirs?
# jq is a small command-line tool for pulling fields out of JSON.
spire-server bundle list -id spiffe://partner.io -format spiffe \
  | jq '{seq: .spiffe_sequence,
         hint: .spiffe_refresh_hint,
         x509_roots: [.keys[] | select(.use == "x509-svid")] | length}'

{
  "seq": 12,
  "hint": 300,
  "x509_roots": 2
}

Two x509_roots means partner is mid-rotation and you already hold the incoming root, which is exactly where you want to be sitting. One root and a seq frozen for a week, while partner insists they rotated on Tuesday, means your poll is dead and you are running on borrowed time. Graph that sequence number per federated domain, alert when it stops moving for longer than a few refresh hints, and you will find the broken endpoint days before it takes production down.

Try this

Run spire-server bundle list -format pem on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: https_spiffe bootstraps once, then depends on reachability forever. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related