Secure introduction: solving secret-zero
AppRole, response wrapping, and platform-native workload identity.
The payments pod boots, reaches for Vault, and stops dead. It has no token yet, and Vault does not talk to strangers. One team fixed that by pasting a long-lived Vault token into a Kubernetes Secret called vault-token. It worked fine until a namespace admin ran kubectl get secret -o yaml and walked off with the keys to half the cluster. That first credential, the one you need before any other credential can be fetched, is called secret-zero. There are three ways to hand it over, weakest to strongest: ship a shared secret (AppRole with a SecretID delivered somehow), hand over a one-time envelope (response wrapping), or use an identity the platform already gave the workload (a Kubernetes service account token, a cloud IAM role, where IAM means identity and access management, or a SPIFFE ID, from Secure Production Identity Framework For Everyone, a standard way to give a workload a name). Most real systems use a mix. The goal never changes. The workload proves who it is with something it was already holding, and gets back a short-lived token scoped to one job.
AppRole: two halves, and only one is safe to publish
AppRole cuts the bootstrap credential in two. The RoleID is the username half. It is not secret, so you can bake it into a Helm value or a config file without losing sleep. The SecretID is the password half. The workload sends both, Vault checks them, and hands back a token. This beats one static token because you can rotate a SecretID, cap how many times it may be used, and pin it to a CIDR range (Classless Inter-Domain Routing, shorthand for a block of IP addresses) so it only works from your build network. One question stays open. How did the SecretID reach the machine in the first place? Answer that badly and you have relocated secret-zero rather than solved it. What makes AppRole defensible is delivering the SecretID out of band and making it single-use, which is exactly what response wrapping does.
Reach for AppRole when the workload has no platform identity to offer: an old virtual machine, a CI runner (continuous integration, the system that builds and tests your code) sitting outside the cluster, a batch job on bare metal. Even then, treat minting a SecretID as a privileged act. A trusted orchestrator calls that endpoint. A script on somebody's laptop, checked into git, does not.
vault auth enable approlevault write auth/approle/role/payments \token_policies=payments-read token_ttl=20m token_max_ttl=1h \secret_id_ttl=10m secret_id_num_uses=1vault read auth/approle/role/payments/role-id
Success! Enabled approle auth methodSuccess! Data written to: auth/approle/role/paymentsrole_id a1b2c3d4-e5f6-7890-abcd-ef1234567890# RoleID is non-secret — safe in Helm values or config maps
vault write -wrap-ttl=90s -f auth/approle/role/payments/secret-idVAULT_TOKEN=$WRAP vault unwrapvault write auth/approle/login role_id=$ROLE_ID secret_id=7e2f...
wrapping_token: hvs.CAESI... # single-use, 90s to livesecret_id: 7e2f...token: hvs.CAE... ttl: 20m policies: ["payments-read"]
Response wrapping: a sealed envelope that shows if it was opened
Response wrapping drops the secret into a temporary one-use locker (Vault calls it a cubbyhole) and hands back a wrapping token instead of the secret itself. Whoever holds that token can open the locker exactly once, inside a short TTL (time to live, the window before something expires). The clever part is tamper-evidence. If the workload you meant to give it to tries to unwrap and Vault replies that the token is already spent, somebody else got there first. The delivery mechanism doubles as a burglar alarm.
That turns a question you normally cannot answer, did anyone read this secret on its way over, into an alert you can page on. It is how CI platforms introduce a workload to Vault without a SecretID ever landing in a build log. Keep the wrap TTL to 60 or 120 seconds and num_uses at one. A wrapping token that outlives the handoff window has quietly become standing privilege.
Platform identity: send no secret at all
The strongest answer ships nothing secret to the workload. A Kubernetes pod already has a signed service account JWT (JSON Web Token, a small signed piece of text that states who the bearer is) mounted inside it. An EC2 instance (Elastic Compute Cloud, an AWS virtual machine) already carries an instance role. A properly bootstrapped node already holds a SPIFFE SVID (SPIFFE Verifiable Identity Document, a short-lived certificate naming the workload). You configure Vault to trust that platform and verify the identity token against it. The Kubernetes auth method checks the token with the cluster's own TokenReview endpoint, which answers yes, I issued that, and it belongs to this service account. The AWS auth method verifies a signed identity request instead. Nothing was delivered, so there is no secret-zero left to steal.
You set the cluster trust up once. Point the Kubernetes auth method at the cluster's CA (certificate authority, the thing whose signature everyone agrees to trust) and its TokenReview endpoint, then bind each Vault role to named service accounts in named namespaces. After that, every pod login is an audited exchange. No SecretID inventory to keep, no wrapping ceremony to run, no token file on disk beyond the service account token Kubernetes mounts for you anyway.
vault auth enable kubernetesvault write auth/kubernetes/config \kubernetes_host=https://kubernetes.default.svc \kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crtvault write auth/kubernetes/role/payments \bound_service_account_names=payments \bound_service_account_namespaces=prod \policies=payments-read ttl=20m
Success! Enabled kubernetes auth methodSuccess! Data written to: auth/kubernetes/configSuccess! Data written to: auth/kubernetes/role/payments
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)vault write auth/kubernetes/login role=payments jwt=$JWTvault token lookup -format=json | jq ".data.ttl,.data.policies"
token: hvs.CAE... # scoped to payments-read, 20 min, auto-renewable1200["payments-read"]
Cloud IAM auth: let the cloud vouch for the machine
A workload running on a cloud virtual machine should log in as the identity the cloud already stamped on it. Vault's AWS auth method verifies a signed GetCallerIdentity call. Google Cloud auth validates service account JWTs against Google's JWKS (JSON Web Key Set, the public keys Google publishes so anyone can check its signatures). An EC2 instance with an instance profile proves it is that instance. A Google Compute Engine instance with an attached service account proves the same thing. No access keys buried in user-data, no SecretIDs stuffed into launch templates.
Bind roles to exact IAM role ARNs (Amazon Resource Names, the full unique identifier of an AWS resource) or exact Google service account emails. Wildcards do not belong here. A role bound to arn:aws:iam::*:role/* will authenticate every instance in the account, the same failure as bound_service_account_names=* on the Kubernetes side. Cloud auth pays off most when your fleet already runs with instance roles. Pair it with dynamic secrets so even the cloud credential Vault itself holds gets rotated behind the scenes.
Bindings are the real access control
Platform identity is only as tight as the binding you wrote. Pointing a Vault role at the shared default service account is the classic slip, because every pod that never asked for a service account is already running as default, so that role now authenticates most of the namespace. Name the service account you actually mean. Name the namespace it lives in. Then read the policy that role hands out and confirm it cannot be used to widen itself. A role that can edit auth mounts or write policies is a root token wearing a costume.
Review auth method configs the way you review firewall rules. Every bound field you loosen widens the set of callers that can mint a token, nothing breaks, and so nobody notices. Check entity aliases while you are in there. A service that logs in through Kubernetes on Monday and through AWS on Tuesday should still resolve to one Vault entity, or revoking it later will only catch half of it.
Picking a bootstrap path per workload
Legacy virtual machines and old apps with no platform identity get AppRole plus wrapping, handed out by a trusted orchestrator. Pods get Kubernetes auth. Cloud instances get cloud IAM auth. Services that already carry SPIFFE identities get JWT auth with SPIRE (the SPIFFE Runtime Environment, the agent that issues and rotates those identities) as the trusted issuer. The auth method changes, the shape does not: prove who you are with something the platform vouches for, walk away with a short-lived scoped token.
Write the chosen path for each service into your architecture decision record. When a team asks whether they can put a Vault token in a Secret and move on, you want a document to point at instead of an argument. A static token in etcd (the database behind your Kubernetes cluster) is secret-zero moved, not secret-zero solved. What decides the answer is where the workload runs, never which integration is fastest to copy off a blog post.
Let the Vault Agent do the handshake
The Vault Agent runs the same bootstrap steps a person would run, unwrapping a SecretID or calling Kubernetes auth, then keeps the resulting token renewed and drops it when the process stops. An init container that logs in once and writes a token to a tmpfs file (a filesystem held in memory that never touches disk) beats a Secret sitting in etcd, but it loses to a sidecar that keeps renewing, because the init container's token expires while the pod is still serving traffic. Prefer the sidecar or the CSI path (Container Storage Interface, the standard plug-in point for mounting things into pods) whenever the workload lives long enough for a TTL to run out.
Treat token TTL and renewal failures as application health, not Vault trivia. A pod that loses its token halfway through a request rarely fails cleanly. It crash-loops. Alert on login errors from the agent with the same seriousness you give a database connection alarm.
vault agent -config=/etc/vault/agent.hcl# agent.hcl uses kubernetes auth + auto_auth; template renders secretsvault token lookup -format=json | jq ".data.ttl,.data.renewable"
==> Vault agent started! Log data will stream in below:3600true# agent renews before TTL hits zero — no human token management
If SecretIDs still get handed over in a chat message or pasted into a ticket, wrap them. Response wrapping turns someone pasted a secret in Slack into a sealed envelope with a short fuse and a loud failure when the wrong person opens it first. Keep AppRole only where platform auth genuinely cannot reach, and put those leftovers on a list to migrate to Kubernetes, IAM, or JWT auth the moment the platform can carry them.
Watch the login metrics on the agent or sidecar. A spike in failed logins is usually a broken binding, a renamed service account, or a JWT audience that no longer matches, rather than Vault being down. Repair the identity path. Do not widen a policy to make the error go quiet.
Try this
Run wrapping and Kubernetes auth end to end in a lab. You are looking for three outcomes: an unwrap that works exactly once, a tight service account binding that logs in cleanly, and a wrong service account that gets refused.
vault write -wrap-ttl=60s -f auth/approle/role/payments/secret-idvault unwrap <wrapping-token>vault write auth/kubernetes/role/payments \bound_service_account_names=payments bound_service_account_namespaces=prod \token_ttl=15m token_policies=payments-readkubectl exec -n prod deploy/payments -- \sh -c 'JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token); vault write auth/kubernetes/login role=payments jwt=$JWT'
Key Value--- -----wrapping_token: hvs.CAESI...wrapping_token_ttl: 1m# unwrap once:secret_id s.8f3a...secret_id_num_uses 1Key Value--- -----token hvs.CAESI...token_policies ["default" "payments-read"]token_duration 15m# wrong SA name -> permission denied at login
Takeaway
Secret-zero goes away when a workload trades an identity it already holds for a token, not when you hide a longer-lived token somewhere prettier. Reach for platform auth first. Where AppRole is the only option left, wrap the SecretID, bind the role to one service account in one namespace, and keep token TTLs in the minutes with renewal switched on.
Next, go and count them. List every workload still mounting a durable Vault token out of a Kubernetes Secret, then move each one onto Kubernetes auth or an agent auto_auth stanza.