Vault on Kubernetes, deeply

The agent injector, the Secrets Store CSI driver, and External Secrets.

Run kubectl get secret -o yaml on a production namespace and there it is: the database password, wrapped in base64 that any web page will decode for you in a second. Base64 is an envelope, not a lock. A Kubernetes Secret is readable by anyone holding get on that namespace, it sits in etcd (the database where the API server keeps every object in your cluster), and it reaches your app as a file or an environment variable. Kubernetes is where most secrets actually get consumed, and it is also where the default story is thinnest. The advanced move is to keep no value in the cluster at all and have something fetch a short-lived one from an external manager when the pod starts. Three patterns grew up around that idea: the Vault Agent sidecar, the CSI driver, and the External Secrets Operator.

Pattern 1: the Vault Agent Injector sidecar

A mutating webhook is a doorman with a clipboard who is allowed to edit your request on the way in. The Vault Agent Injector is that doorman. You put a few annotations on a pod, he notices them, and before the pod ever runs he adds a second container to it. That container is the Vault Agent. It logs in to Vault with the pod's own service account token (Kubernetes auth, which is how you get past secret-zero, the chicken-and-egg problem of who hands a workload its very first credential), pulls what its role allows, and writes the values as files onto a small in-memory volume both containers share. Your app opens a file. It never speaks to Vault and never holds a Vault token.

The agent stays alive for the life of the pod. As a lease nears expiry it renews it, and when the value changes it rewrites the file. Templates shape that file into whatever the app already knows how to read, a dotenv file, a YAML fragment, Java properties. Dynamic database credentials are where this really pays. The sidecar renews the lease and rewrites the connection file before the database user underneath expires, so the app never notices the credential churning below it.

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "payments"
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/payments-ro"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "database/creds/payments-ro" -}}
      DB_USER={{ .Data.username }}
      DB_PASS={{ .Data.password }}
      {{- end -}}
# rendered to /vault/secrets/db (tmpfs); app reads the file.

kubectl apply -f payments-pod.yaml
kubectl get pod payments-0 -n prod -o jsonpath="{.spec.containers[*].name}"
kubectl exec -n prod payments-0 -c payments -- cat /vault/secrets/db

pod/payments-0 created
payments vault-agent
DB_USER=v-kubernetes-payments-ro-x7Qb...
DB_PASS=A1b2C3...

Pattern 2: the Secrets Store CSI driver

The second pattern reuses plumbing the cluster already has. CSI, the Container Storage Interface, is the standard way any storage system plugs into Kubernetes and mounts a volume into a pod. The Secrets Store CSI driver pretends to be storage. Instead of a disk it hands the pod a directory of files pulled from Vault, AWS Secrets Manager, Google Secret Manager or Azure Key Vault, fetched when the pod starts and rotated afterwards if you enable that. The interesting part is the optional sync. You can tell the driver to also copy a value into a real Kubernetes Secret when something genuinely needs one, which makes etcd materialization a bridge you open deliberately, one workload at a time, instead of the default sprawl.

Reach for this when you want one mounting mechanism everywhere and would rather not run an extra container in every pod. A SecretProviderClass object declares which Vault path or which cloud secret to fetch. The pod spec then stays the same across EKS, GKE and AKS (the managed Kubernetes services from Amazon, Google and Microsoft). Only the provider class changes.

kubectl apply -f secret-provider-class-vault.yaml
kubectl get secretproviderclass vault-db -n prod -o yaml | grep -A2 "vault"
kubectl describe pod payments-csi-0 -n prod | grep -A3 "Mounts:"

secretproviderclass.secrets-store.csi.x-k8s.io/vault-db created
  vaultAddress: "https://vault.internal:8200"
  roleName: "payments"
  Mounts:
    /mnt/secrets from secrets-store (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access

Pattern 3: the External Secrets Operator

ESO (the External Secrets Operator) takes the opposite bet. You write an ExternalSecret object saying where the real value lives, and a controller inside the cluster fetches it on a timer and builds an ordinary Kubernetes Secret from the result. Everything downstream keeps working the way it always did, because what your workloads consume is a plain Secret. The source of truth stays outside the cluster. Rotate the value in Vault or in AWS and ESO copies the new one down at the next refresh.

The bill arrives in etcd. That materialized Secret really does live there between refreshes, which makes this the most compatible pattern and the one least able to claim the value never touches cluster storage. Two things keep it survivable. Shorten refreshInterval for your highest-value secrets so a stolen etcd snapshot goes stale fast, and pair it with encryption at rest so the snapshot is unreadable to begin with. Compatibility is bought with an exposure window, so know how wide yours is.

kubectl apply -f external-secret.yaml
kubectl get externalsecret payments-db -n prod
kubectl get secret payments-db -n prod -o jsonpath="{.metadata.annotations}"

externalsecret.external-secrets.io/payments-db created
NAME          STORE      REFRESH INTERVAL   STATUS         READY
payments-db   vault-prod 1h                 SecretSynced   True
# materialized Secret exists in etcd — know your exposure window

Choosing a pattern per workload

Match the pattern to what the workload needs. A dynamic Vault lease that has to be renewed on a schedule points at the Agent Injector, which speaks Vault natively and renews in place. The same file mount needed in three clouds points at CSI. A controller or Helm chart that only knows how to read a Kubernetes Secret points at ESO, on the shortest refresh interval it will tolerate. None of the three replaces etcd encryption, tight RBAC (role-based access control, the rules for who may read what), or switching off service account tokens that nothing uses.

Write that decision into your service template so no team has to relitigate it. New microservices get the injector. Third-party charts with a hardcoded secretKeyRef get ESO on a fifteen-minute refresh. Batch jobs on CronJobs get a CSI mount that fetches once at start. When a platform team standardizes on one pattern for everything, it is almost always ESO, because ESO is the easiest to adopt, and the quiet result is that every secret in the company lands in etcd anyway.

Cluster hygiene underneath every pattern

All three patterns sit on top of the cluster, and a soft cluster leaks around them. Turn on encryption at rest for Secrets with a real KMS provider (a key management service, the hardware-backed home for the key that encrypts everything else). Kubernetes also ships an identity provider that encrypts nothing, so check which one you actually have configured. Cut Secret read access down to the workloads that need it. Switch off auto-mounted service account tokens for pods that never call the API. And hold on to this one: any secret rendered into a running pod can be read by anyone able to exec into that pod. A shell in a container is a secret-read permission, even for someone with no rights over Secret objects at all.

So audit who holds create pods and exec pods with the same suspicion you apply to get secrets, in every namespace running something sensitive. External delivery shrinks what an attacker finds in etcd. It does nothing about a cluster admin running kubectl exec and cat. Tight RBAC goes on top of external delivery, never in place of it.

kubectl auth can-i get secrets --as=system:serviceaccount:prod:payments -n prod
kubectl auth can-i create pods --as=developer-alice -n prod
# EncryptionConfiguration is an apiserver config file (--encryption-provider-config),
# not a kubectl API object. On kubeadm, inspect the static-pod manifest / enc file:
grep -n encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml
# or for EKS/GKE/AKS: check the managed secrets-encryption / CMEK setting in the console/CLI

no
yes
# payments SA cannot read Secret objects directly — good
# developer can create pods that mount secrets — review create RBAC

When the injector webhook breaks (and what upgrades do to it)

Back to the doorman. If he goes home, the pods that never needed him walk straight in, and the pods whose whole secret delivery depends on him get turned away at the door. That is the injector's failure mode, and it is deliberate. The webhook runs at pod admission with failurePolicy set to Fail, so annotated pods stop scheduling while it is unavailable. Run more than one replica. Alert on admission failures. Test cluster upgrades in staging with injection switched on, because a webhook that stops answering looks like a deployment outage for a good hour before anyone thinks to blame secrets.

Vault and injector chart upgrades deserve the same care. Annotation keys get renamed between chart versions, template syntax shifts, and Kubernetes auth role names are a hard coupling between your Vault config and your pod specs. Put a smoke-test pod in the post-upgrade checklist, one that does nothing but read /vault/secrets/* and complain loudly when the file comes back empty.

kubectl get mutatingwebhookconfiguration vault-agent-injector-cfg -o yaml | grep -A3 failurePolicy
kubectl logs -n vault deploy/vault-agent-injector --tail=5

failurePolicy: Fail
# pods requiring injection fail if webhook unavailable — monitor this
 
Successfully injected vault-agent into pod/payments-0

NetworkPolicy belongs in this conversation, because the Vault API and the cloud metadata endpoint are part of the secrets path. A pod that must reach Vault and has no business on the public internet sits in a different trust zone from one that needs both, and the policy should say so out loud. Sidecars complicate it. The agent shares the pod's network namespace, so your egress rules have to allow its calls to Vault even though the application container never makes them.

If you enabled secretObjects sync on a SecretProviderClass, the Secret it creates joins your threat model, and so does everyone who can read it. You traded etcd exposure for compatibility on purpose, so manage it on purpose. Label the synced Secrets so one query finds all of them, and scope get and list to the single service account that consumes each, rather than to the whole developer group in that namespace.

Track webhook admission latency next to Vault availability, as a service level objective you actually get paged on. A slow injector does not fail loudly. It stalls pod scheduling across the cluster at the exact moment you are scaling up to meet a traffic spike.

kubectl get secret -n prod -l secrets-store.csi.k8s.io/managed=true
kubectl describe secretproviderclass vault-db -n prod | grep -A5 secretObjects

NAME           TYPE     DATA   AGE
payments-db    Opaque   2      1h
# synced Secret exists — verify RBAC and encryption at rest
 
secretObjects:
  - secretName: payments-db
    type: Opaque

Put ESO and the Vault CSI provider side by side when someone asks which to use for an app you did not write. They pull the same value out of Vault and then part ways. ESO gives you a Kubernetes Secret you can mount as environment variables, convenient, and back to a value anyone with namespace read access can see. CSI gives you a file on tmpfs (a filesystem that lives only in the node's memory and dies with the pod), which leaves nothing in cluster storage and irritates every application that insists on environment variables.

Refresh intervals are a tuning problem with two bad ends. Too long, and a rotated secret leaves half your pods holding a credential that no longer works. Too short, and you hammer Vault and the API server for a value that rarely changes. Line the interval up with the Vault TTL (time to live, how long the credential stays valid) and with what the app actually does when a file or env var changes. A Secret that updates in the API but is never re-read by the process holding the old copy is theater.

RBAC on Secret and ExternalSecret objects is usually the last thing tightened and often the first thing that hands an attacker the whole set. Read access on Secrets in a namespace generally means read access to the values themselves, and read access on ExternalSecret tells someone precisely which Vault paths are worth going after. Namespace boundaries are part of the secrets design here, not only a tenancy convenience.

Try this

Look at how secrets are actually landing in one namespace. Compare an ESO-managed Secret with a CSI mount, then prove that a service account from a different namespace cannot read either of them.

kubectl get externalsecret -n payments
kubectl get secret payments-db -n payments -o jsonpath='{.metadata.annotations}' ; echo
kubectl exec -n payments deploy/payments -- ls -l /mnt/secrets
kubectl auth can-i get secrets -n payments --as=system:serviceaccount:dev:default

NAME            STORE         STATUS   READY
payments-db     vault-backend SecretSynced True
 
{"force-sync":"2026-07-24T01:00:00Z","reconcile-strategy":"Merge"}
 
total 4
-r-------- 1 root root 24 Jul 24 01:00 db-password
 
no
# default SA in another namespace must not read payments Secrets

Takeaway

Kubernetes is where a fifteen-minute Vault credential meets etcd and a process environment, both of which outlive it. Let the pod prove who it is with its own service account, deliver the value as a file that dies with the pod wherever the app will accept one, and count Secret RBAC as part of the blast radius rather than a ticket for next quarter.

Next: pick one namespace, find the long-lived vault-token Secret somebody created there two years ago, and move that workload onto Kubernetes auth with ESO or CSI before you delete it.

Related