Private PKI & short-lived certificates

Root/intermediate CAs, 24-hour leaf certs, cert-manager, CRL/OCSP.

Someone copies your TLS private key (Transport Layer Security, the encryption behind every https address) off a build server. The certificate that key belongs to is good for another two years, so the stolen copy is good for another two years too. Certificates do not care who is holding the key, only that the maths checks out. Running your own CA (Certificate Authority, the thing that signs certificates and vouches for the name on them) changes that arithmetic. Issue certificates that expire in hours and a stolen one is scrap by tomorrow morning. Vault's PKI engine (Public Key Infrastructure, the machinery of certificates and the authorities that sign them) makes issuing thousands of them cheap enough that rotation stops being a fire drill and becomes background noise.

Keep the root offline, let the intermediate do the work

A healthy PKI has a root CA that signs exactly one thing, an intermediate CA, and then goes back in the safe. The intermediate handles every certificate you actually issue day to day. If someone steals the intermediate, you revoke it, sign a replacement, and every client carries on trusting the same root it always trusted. Vault mirrors that shape directly: a root PKI mount (or a root that lives outside Vault entirely) signs a CSR (Certificate Signing Request, a formal "please vouch for this key and this name" message) from an intermediate mount, and your workloads only ever talk to the intermediate.

Ideally the root's private key never touches the online Vault at all. The intermediate is the piece you harden, audit, and replace on a schedule measured in years. Leaf certificates, the ones your services actually present on the wire, rotate in hours. Treat the root signing as a ceremony in the literal sense: witnesses in the room, every step written down as it happens, key material stored somewhere you would notice had been opened.

vault secrets enable -path=pki pki
vault secrets tune -max-lease-ttl=87600h pki
vault write -field=certificate pki/root/generate/internal common_name="Acme Root CA" ttl=87600h > root_ca.crt
 
vault secrets enable -path=pki_int pki
vault write -field=csr pki_int/intermediate/generate/internal common_name="Acme Issuing CA" > int.csr

Success! Enabled the pki secrets engine at pki/
Success! Tuned the pki secrets engine
 
-----BEGIN CERTIFICATE-----
... root CA written to root_ca.crt ...

vault write -field=certificate pki/root/sign-intermediate [email protected] format=pem_bundle ttl=43800h > int.pem
vault write pki_int/intermediate/set-signed [email protected]
vault write pki_int/config/urls issuing_certificates="https://vault.internal/v1/pki_int/ca" crl_distribution_points="https://vault.internal/v1/pki_int/crl"

-----BEGIN CERTIFICATE-----
... intermediate bundle in int.pem ...
Success! Data written to: pki_int/intermediate/set-signed
Success! Data written to: pki_int/config/urls

Short-lived leaf certificates that renew themselves

Once a role exists, issuing a certificate is a single call that hands back three things at once: the certificate, its private key, and the chain up to the CA. All of it is disposable. A service asks for a fresh certificate when it starts, then asks again well before the old one runs out. Because these certificates only live a few hours, you almost never reach for a revocation list. Expiry does that job for you, on time, every time, with nobody filing a ticket.

This is the whole mTLS story (mutual TLS, where the client proves who it is to the server and the server proves who it is right back) for service meshes and internal APIs. Every service presents a short-lived certificate tied to its identity, and the trust flows from the intermediate they all share. Inside Kubernetes, cert-manager's Vault issuer runs that loop for you, so a pod is always holding a certificate with most of its life still ahead of it.

vault write pki_int/roles/svc allowed_domains="svc.acme.internal" allow_subdomains=true max_ttl=24h key_type=ec key_bits=256 server_flag=true client_flag=true
vault write -format=json pki_int/issue/svc common_name="payments.svc.acme.internal" ttl=24h | jq ".data.expiration,.data.serial_number"

Success! Data written to: pki_int/roles/svc
 
1721894400
"7a:3f:..."
# cert + private_key + issuing_ca returned in same response

apiVersion: cert-manager.io/v1
kind: Certificate
metadata: { name: payments-tls, namespace: prod }
spec:
  secretName: payments-tls
  duration: 24h
  renewBefore: 8h            # renew with a third of life left
  commonName: payments.svc.acme.internal
  issuerRef: { name: vault-int, kind: ClusterIssuer }
# the pod always has a valid, hours-old cert; no human in the loop.

Long-lived versus short-lived: what you are actually trading

Revocation is the backstop, not the plan

You still need revocation, because a certificate can be stolen an hour after issue, or a role can be wrong in a way you only spot on Tuesday. Vault keeps a CRL (Certificate Revocation List, a published list of certificates you have declared dead) and can answer OCSP queries (Online Certificate Status Protocol, where a client asks a server "is this one still good?" in real time). Clients that check either will reject a revoked certificate before it expires. The catch is that most clients soft-fail. If the responder is unreachable, they shrug and connect anyway. So wire revocation up, then let short lifetimes do the real security work.

Certificates that face the public internet live in a different trust store with the same habits. ACME (Automatic Certificate Management Environment, the protocol Let's Encrypt made famous) lets a machine prove it controls a name and collect a certificate with no human in the chain, and Vault speaks it too. Internal mTLS runs off your private intermediate with far more aggressive lifetimes. Two trust stores, one automation pattern.

vault write pki_int/revoke serial_number="7a:3f:..."
vault read -field=crl pki_int/cert/crl | openssl crl -inform DER -text -noout | head -20

Success! Data written to: pki_int/revoke
 
Revoked Certificates:
    Serial Number: 7A3F...
        Revocation Date: Jul 24 10:00:00 2026 GMT

A loose role is a licence to impersonate

A PKI role decides which names a caller is allowed to ask for. Open that up too far and you have handed out permission to be anyone inside your trust boundary. Whoever can reach the issuing path can mint a certificate for payments, for the admin console, for the login page. Pin allowed_domains to names the role legitimately owns, leave allow_any_name off, restrict which key usages the certificate carries, and cap max_ttl per role. A role meant for one service should never be able to sign intermediates or client certificates it has no business signing.

Re-read your roles every time a service onboards. The wide-open role somebody created for a two-week pilot is still sitting there a year later, and by then it is a back door with a change ticket attached to it. Split roles by tier: edge ingress gets real DNS names, internal traffic gets svc.internal names, and humans get a role of their own with a shorter max_ttl and stricter usage flags.

Public trust and the service mesh

For anything a browser or a partner touches, ACME pulls the same trick against publicly trusted CAs: Let's Encrypt, a commercial CA, or Vault's own ACME server when the names are internal. The short-lifetime discipline carries over unchanged. cert-manager with an ACME issuer renews long before expiry, and the spreadsheet where somebody tracked multi-year expiry dates quietly stops being load-bearing. That spreadsheet was never right anyway.

Inside the mesh, the intermediate CA certificate gets distributed once as a trust bundle, and after that workloads only ever request leaf certificates. Istio and Linkerd both plug into cert-manager or a Vault agent, so the identity a service uses for mTLS rotates on the same clock as its database credentials. One PKI story then covers traffic arriving at the cluster and traffic moving between services inside it.

kubectl get certificate -A | grep -v True
kubectl describe certificate payments-tls -n prod | grep -A2 "Events:"
cert-manager renews before renewBefore threshold

NAMESPACE   NAME          READY   SECRET
prod        payments-tls  True    payments-tls
 
Normal  Issuing    Certificate issued successfully
# no manual renewal tickets — automation owns the lifecycle

Ship the intermediate CA certificate to workloads and ingress controllers through a single trust bundle ConfigMap (a Kubernetes object that holds plain configuration data and can be mounted into pods as files). Replace the intermediate later and you update that one bundle instead of hunting down every consumer. cert-manager already copies the chain into the Secret it creates, but mesh-wide trust stores usually need their own delivery path.

Put notBefore and notAfter, the two timestamps that bound a certificate's validity, into metrics the same way you track secret expiry. A certificate that has burned 80% of its life with no renewal event pending should page somebody. Short lifetimes only protect you while the automation is actually running. The classic failure is a team saying "it expires in 24 hours, we are fine" while the renewal job has been crashing quietly since last Thursday.

When an auditor asks for a certificate inventory, generate it from cert-manager Certificate resources and Vault's audit record of issuance paths. A list assembled by hand after the last incident proves only that somebody once made a list. Generated inventory is evidence the short-lived PKI is running; a hand-kept one is evidence you intend to automate at some point.

Put allowed_domains and max_ttl on the onboarding checklist for every new service. PKI roles are configuration you revisit, not something you set during a hackathon and never look at again once real traffic shows up.

Make the cert-manager readiness check fail when a Certificate sits at Ready=False for more than an hour. If readiness only watches the app container, the first person to notice an expiry outage is a user and the second is whoever they complain to.

vault read pki_int/cert/ca
vault write pki_int/issue/svc common_name="api.svc.acme.internal" ttl=12h -format=json | jq ".lease_id,.data.expiration"

-----BEGIN CERTIFICATE-----
... issuing CA cert ...
 
"database/creds/..."
1721856000
# lease_id on issued certs — revoke early if key material suspected leaked

Vault PKI earns its keep when you want short-lived certificates and have no enterprise CA team to stand up overnight. Keep the root offline, or at minimum behind a namespace almost nobody can reach. Issue everything from the intermediate. Set lifetimes in hours or days rather than years. Be honest about what that buys you: short lifetimes shrink the window an attacker gets, they do not remove the trust problem, because everything still hangs off an intermediate that can sign whatever its roles permit. A leaf certificate valid for a year is the standing-privilege problem again, wearing a different file extension.

Push issuance out to where the workload runs: Vault agent templates, cert-manager with the Vault issuer, or a CSI driver (Container Storage Interface, the plugin system Kubernetes uses to mount external stores as files inside a pod). The moment humans start pasting PEM blocks (Privacy Enhanced Mail, the BEGIN CERTIFICATE text format) into tickets, intermediates get over-issued and private keys end up in somebody's chat history.

Pick your revocation story out loud and write down which one you chose: CRL, OCSP, or short lifetimes. For internal mTLS, short lifetimes usually beat the other two on both cost and reliability, because a stolen certificate dies on its own before the incident bridge has finished filling up. Browsers and partners are a separate conversation. Plan public trust and ACME on their own terms rather than bolting them onto internal Vault PKI.

Try this

Build a throwaway root, give it a role, and issue a two-hour certificate. Then read what comes back, so you know precisely what a client sees when it inspects one of these.

vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki
vault write -field=certificate pki/root/generate/internal \
  common_name="Lab Root" ttl=87600h > ca.crt
vault write pki/roles/mtls allow_any_name=true max_ttl=24h
vault write pki/issue/mtls common_name=payments.svc ttl=2h

Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----...
issuing_ca          -----BEGIN CERTIFICATE-----...
private_key         -----BEGIN RSA PRIVATE KEY-----...
serial_number       39:3e:78:...
expiration          1721894400
# openssl x509 -in leaf.crt -noout -dates
# notAfter = ~2 hours from issue

Takeaway

Vault PKI holds together when three things are true: the intermediate does all the issuing, roles are narrow enough that nobody can request a name they do not own, and private keys travel from the issuance response into a running process without stopping in a ticket or a git repository. TTL (time to live, how long a certificate stays valid) is the same standing-privilege argument you already had about passwords, restated in certificate form.

Next: list the services still running year-long certificates, move one of them onto automated Vault issuance this week, and write down the steps for rotating the intermediate before the day it expires becomes an outage nobody planned for.

Related