Vault on Kubernetes, deeply

The agent injector, the Secrets Store CSI driver, and External Secrets.

Kubernetes is where most secrets actually get consumed, and it is also where the default story is weakest — a Kubernetes Secret is base64-encoded, readable by anyone with get on the namespace, and mounted into pods as files or env vars. Advanced practice replaces "a Secret sitting in etcd" with "a short-lived secret delivered to the pod at runtime from an external manager." There are three mature integration patterns, and choosing between them is a real design decision.

Pattern 1 — the Vault Agent Injector (sidecar)

The injector is a mutating webhook: annotate a pod and it injects a Vault Agent init/sidecar container that authenticates with the pod’s service account (Kubernetes auth, solving secret-zero), fetches secrets, and renders them to a shared in-memory volume as files. The app reads a file; it never talks to Vault directly and never holds a Vault token. The agent renews leases and re-renders on rotation, and templates let you shape the output into whatever config format the app expects.

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "payments"                 # K8s-auth role
    vault.hashicorp.com/agent-inject-secret-db: "database/creds/payments-ro"
    vault.hashicorp.com/agent-inject-template-db: |
      {{- with secret "database/creds/payments-ro" -}}
      DB_USER={{ .Data.username }}
      DB_PASS={{ .Data.password }}
      {{- end -}}
# rendered to /vault/secrets/db (tmpfs); the app just reads the file.

Pattern 2 — the Secrets Store CSI driver

The CSI driver mounts external secrets as a volume through the standard container storage interface, with providers for Vault, AWS, GCP, and Azure. The pod sees files under a mount path; the secret is fetched at pod start and (optionally) rotated. Its distinguishing feature is that it can also sync into a native Kubernetes Secret when something genuinely needs one (an image pull secret, a controller that only reads Secrets) — a controlled bridge rather than the default sprawl. It is a good fit when you want a uniform mount mechanism across clouds.

Pattern 3 — External Secrets Operator (ESO)

ESO takes the opposite stance: it reconciles an ExternalSecret custom resource by pulling from the external manager and creating a normal Kubernetes Secret, refreshed on an interval. You keep working with plain Secrets (maximum compatibility), and the source of truth stays external — rotate in Vault or AWS and ESO refreshes the cluster copy. The tradeoff is that the materialized Secret does live in etcd between refreshes, so this is the most compatible but least "never touches etcd" of the three.