Vault & secrets interview questions
Prep Vault interviews from KV basics to production ops: auth, policies, dynamic secrets, Kubernetes, leases, seal/unseal, and the secret-zero problem.
Beginner → Intermediate → Advanced → Expert — answers are written the way you’d say them in a real interview. Advanced and Expert go deeper with war-story detail, architecture diagrams, and the follow-up an interviewer often asks next.
What problem does Vault solve?Beginner
The short version: centralized secrets with identity-based access, leasing, rotation, and audit — so credentials aren't scattered in config files, env vars, and CI, and every access is authenticated and logged.
vault kv put secret/app db_pass=s3cr3t vault kv get -field=db_pass secret/app
Everything in Vault is a path — what does that mean?Beginner
Engines and auth methods are mounted at API paths — secret/, database/, auth/kubernetes/. Clients read and write those paths; policies grant capabilities per path. The API and the ACL model share the same namespace.
vault secrets list vault kv get secret/app
KV v1 vs KV v2?Beginner
Both store static key/value data; v2 adds versioning, soft delete, undelete, and check-and-set. I'd prefer v2 for new mounts unless you specifically need the simpler v1 semantics.
vault kv put secret/app pass=v2 vault kv get -version=1 secret/app vault kv rollback -version=1 secret/app
How do auth methods and tokens relate?Beginner
An auth method — Kubernetes, AppRole, OIDC, cloud IAM — verifies an identity and issues a Vault token bound to policies and a TTL. Clients present that token on later requests, not a shared password.
vault login -method=userpass username=alice # token inherits policies + TTL from the auth role
What is seal and unseal?Intermediate
Vault starts sealed: storage encryption keys aren't in memory. Unseal reconstructs the master key — Shamir shares or auto-unseal via KMS/HSM — so Vault can decrypt storage and serve requests.
Sealed Vault refuses almost all operations; stealing the disk without unseal keys or KMS access is useless. Shamir needs a human quorum after restart; auto-unseal trades that ops burden for trust in the KMS. Lost unseal material is a cluster-loss scenario — protect it and rehearse recovery.
vault operator unseal # enter share 1 vault operator unseal # enter share 2 vault status # Sealed: false
How do Vault policies work?Intermediate
HCL policies grant capabilities — create, read, update, delete, list, sudo — on paths, deny-by-default. Auth roles attach policies to issued tokens. Least-privilege path scoping is the whole ACL model.
path "secret/data/app/*" {
capabilities = ["read"]
}
path "database/creds/app" {
capabilities = ["read"]
}What capabilities does a policy path entry control?Intermediate
create, read, update, delete, list, and sometimes sudo or deny. list is separate from read — missing list can hide keys even when read on a known path works. deny overrides allow when both match.
path "secret/metadata/app/*" { capabilities = ["list"] }
path "secret/data/app/*" { capabilities = ["read"] }What is a Vault token accessor?Intermediate
It's a handle that lets you look up or revoke a token without knowing the token value itself — handy for operators and audit workflows when the raw token has to stay secret.
vault token lookup -accessor <accessor> vault token revoke -accessor <accessor>
Identity entities and groups — why do they matter?Advanced
Vault can map many auth aliases — OIDC user, GitHub, AppRole — to one entity and assign group policies. That keeps ACL consistent when the same human uses different login methods.
Without entities, each auth method issues tokens with only that method's policies — duplicated and inconsistent. Entities unify identity; groups attach shared policies like team-platform. Useful for human SSO plus a break-glass AppRole mapped to the same entity. Machines usually stay on dedicated roles without human entity merging. Review entity merges carefully — a wrong merge elevates access.
vault write identity/lookup/entity name=alice vault read identity/group/name/platform
Interviewer often follows with: How do you stop an OIDC user from inheriting a machine AppRole's policies?
What are dynamic secrets?Intermediate
Vault generates credentials on demand with a lease and TTL — for example a short-lived database user — and revokes them at expiry so no long-lived shared password sits in config.
Each checkout is unique and tied to an identity in the audit log. A leak is time-bounded by TTL; operators can revoke a lease tree instantly during an incident. Needs a secrets engine that can create and delete users or keys in the backend.
vault read database/creds/readonly # username, password, lease_id, lease_duration
Static vs dynamic secrets — when do you use each?Intermediate
I'd use dynamic wherever the backend supports it — DB, cloud IAM, SSH. Use static KV for third-party API keys Vault can't mint, then rotate them on a schedule and version them in KV v2.
What does the transit secrets engine do?Intermediate
I'd call it encryption as a service: apps send plaintext and get ciphertext back without holding the key. Transit also signs, verifies, and HMACs; key rotation and rewrap stay inside Vault.
vault write transit/encrypt/orders \ plaintext=$(echo -n 'hi' | base64) vault write -f transit/keys/orders/rotate
You need field-level encryption in an app without teaching every service to manage AES keys. How do you use transit safely?Advanced
I'd mount transit, create a named key, grant apps encrypt/decrypt only on that key path, and keep ciphertext in the app database. Keys never leave Vault; I rotate and rewrap on a schedule.
Transit is cryptography as a service, not a KV store — you store ciphertext elsewhere. Policies should split encrypt-only vs decrypt: many writers, few readers. Use convergent encryption only when you understand the dedup/leak tradeoffs. Rotate keys with rewrap so old ciphertext stays readable. For high volume, use batch APIs and be careful about caching tokens. Transit doesn't replace TLS or disk encryption of the DB.
path "transit/encrypt/orders" { capabilities = ["update"] }
path "transit/decrypt/orders" { capabilities = ["update"] }
# prefer separate roles: producers encrypt-only, readers decryptInterviewer often follows with: What happens to old ciphertext after you rotate the transit key?
What are leases, TTLs, and revocation?Intermediate
Dynamic secrets and tokens carry a lease with a TTL; clients renew before expiry or Vault revokes. Operators can revoke one lease or a prefix instantly — that's a big advantage over static shared secrets.
vault lease revoke -prefix database/creds/readonly vault token revoke -self
How does Vault act as a PKI/CA?Intermediate
The PKI engine issues short-lived X.509 certs from a configured CA — so services get automated mTLS that rotates frequently instead of long-lived files sitting on disk.
vault write pki/issue/web \ common_name=web.example.com ttl=72h
Dynamic DB credentials are leaking into application logs. How do you respond and harden?Expert
I'd revoke the lease prefix immediately, rotate the database root/config connection if needed, fix the app logging, shorten TTLs, and use transient env or memory-only injection so creds aren't written to disk or stdout.
Revoke first, investigate second. Audit logs show which identity checked out the lease. Root rotation for the database engine updates the admin credential Vault uses without redistributing it to apps. Prevent recurrence: structured logging redaction, agent/CSI file modes with tight permissions, no echo in CI, and TTL short enough that a leaked password dies quickly. Prefer per-role creation statements with least privilege. Post-incident: confirm no persistent clone of the user remains in the DB.
vault lease revoke -prefix database/creds/app vault write database/roles/app \ default_ttl=15m max_ttl=1h \ [email protected]
Interviewer often follows with: How do you rotate the database engine's root credentials?
How does Kubernetes auth to Vault work?Advanced
The workload presents its ServiceAccount JWT; Vault validates it with TokenReview and maps the SA and namespace to a role and policies. No static Vault token needs to live in the Pod — that's the whole point.
This solves secret zero inside the cluster: the platform already issues the SA token. Configure Vault with a token reviewer JWT or the official auth method with audiences. Bound SA names and namespaces must be tight — a wildcard SA binding is an escape hatch. Prefer projected, audience-bound tokens over long-lived SA secrets. Agent injector or CSI then uses the resulting Vault token to fetch secrets and renew leases.
Pod SA JWT → Vault TokenReview → role/policies → short-lived Vault token.
vault write auth/kubernetes/role/app \ bound_service_account_names=app \ bound_service_account_namespaces=prod \ policies=app ttl=1h
Interviewer often follows with: What goes wrong if bound_service_account_names is "*"?
Agent injector vs Secrets Store CSI vs External Secrets Operator?Advanced
Injector adds a sidecar that writes and renews secrets to files; CSI mounts secrets as volumes without a sidecar; ESO syncs external secrets into native Kubernetes Secrets. I'd choose based on whether apps read files or expect Secret objects.
Injector: minimal app change, auto-renew, extra container. CSI: clean mount API, renew depends on driver and version. ESO: easiest for apps already using envFrom or secretKeyRef, but it materializes secrets into etcd — pair with encryption at rest and tight RBAC. Many platforms use ESO for non-sensitive config sync and injector or CSI for high-value dynamic creds.
ESO pulls from Vault (or cloud SM) into a Kubernetes Secret the Pod already consumes.
annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "app" vault.hashicorp.com/agent-inject-secret-db: "database/creds/app"
Interviewer often follows with: If ESO syncs into a Secret, what still protects etcd?
What is the secret-zero problem?Advanced
To fetch secrets you need a bootstrap credential — it's an infinite regress. I'd solve it with platform identity — Kubernetes SA, cloud instance identity — that Vault can verify so nothing static is pre-planted.
Planting a long-lived Vault token in a Secret or AMI just moves the problem. Pattern: workload proves who it is via the platform; Vault trusts the platform's attestation; short-lived Vault token follows. For CI, OIDC or carefully delivered wrapped AppRole secret-ids. For VMs, AWS IAM / GCP / Azure auth methods. Document the trust anchors — if the platform IdP is wrong, Vault will mint wrongly.
# no VAULT_TOKEN in the Pod manifest # role bound to SA app in namespace prod
Interviewer often follows with: How do you bootstrap the first Vault admin without secret-zero theater?
When would you use AppRole instead of Kubernetes auth?Intermediate
For automated clients outside Kubernetes — CI jobs, VMs, appliances — that authenticate with a RoleID plus a SecretID delivered out-of-band. I'd prefer platform identity when it exists.
vault write auth/approle/role/ci \ token_policies=ci token_ttl=20m vault read auth/approle/role/ci/role-id vault write -f auth/approle/role/ci/secret-id
AppRole vs Kubernetes auth for a Jenkins agent that deploys to prod — how do you decide?Expert
If the agent runs in Kubernetes, I'd use Kubernetes auth bound to that SA. If it's a classic VM agent, AppRole or cloud IAM auth — with wrapped, single-use secret-ids and short token TTLs, never a long-lived root token in Jenkins credentials.
Jenkins Vault Plugin often defaults to a stored token — that's secret zero. Better: AppRole where Jenkins only stores role-id; secret-id is injected per job via a trusted broker, or JWT/OIDC auth if available. Limit policies to the deploy paths needed. Separate roles for PR builders — no prod — vs main. Audit every login. Rotate secret-ids and revoke on agent compromise. Prefer moving the deploy step to a pipeline with OIDC to cloud and Vault JWT auth over a standing agent when you can.
vault write -f -wrap-ttl=60s \ auth/approle/role/ci/secret-id # deliver wrapping token to job; unwrap once
Interviewer often follows with: Why wrap a secret-id instead of writing it to the Jenkins credential store?
How does an app get a secret with the Vault Agent injector?Beginner
You annotate the Pod with role and secret path; the injector adds an init or sidecar agent that authenticates, writes the secret to a shared volume, and renews leases. The app just reads the file.
vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "app" vault.hashicorp.com/agent-inject-secret-db: "secret/data/app"
Why enable audit devices, and what do they capture?Advanced
Audit devices log every request and response — secrets HMAC'd — to file, syslog, or socket. If Vault can't write audit logs it refuses requests. Auditing is mandatory, not best-effort.
Ship audits to a SIEM; alert on auth failures, policy changes, and unusual path reads. HMAC means raw secrets aren't in the log, but path and identity still enable forensics. Use at least one durable audit device; dual devices avoid a single broken sink blocking the cluster. Protect log integrity — WORM or object lock — so an attacker with Vault admin can't silently erase history on the only copy.
vault audit enable file file_path=/var/log/vault_audit.log vault audit list
Interviewer often follows with: What happens if the audit disk fills up?
How does Vault provide HA and DR?Expert
For HA I'd run Integrated Storage — Raft — across an odd number of voting nodes with one leader. Back up Raft snapshots and rehearse restore; Enterprise adds performance/DR replication across sites. And protect the unseal or auto-unseal path through the disaster.
HA keeps serving through node loss; DR is about losing the region. Snapshots must be offline and tested — a restore yields a sealed Vault. Document who holds Shamir shares or how auto-unseal KMS is recovered. Don't treat replication enabled as a backup substitute without restore drills. Monitor raft peer health and leadership flaps.
vault operator raft snapshot save vault.snap vault operator raft snapshot restore vault.snap
Interviewer often follows with: Where do you store Shamir shares relative to the Vault cluster?
How do you handle rotation and break-glass access?Advanced
Prefer dynamic secrets so rotation is automatic. For static secrets use scheduled rotation and KV versions. Break-glass is a tightly audited, high-privilege path with alerting — used only in emergencies and rotated after.
Break-glass shouldn't be the same as day-two admin SSO. Require dual control where possible, page on use, and time-box tokens. Root token should be sealed away after init — generate temporarily for break-glass, then revoke. Database root rotation and AWS root IAM rotation keep engine credentials off human laptops.
vault write -force database/rotate-root/my-db
Interviewer often follows with: Should the root token live in the team password manager?
How do you mount and tune a secrets engine?Beginner
vault secrets enable -path=... mounts an engine; tune sets default and max lease TTLs. Paths become the policy surface — plan mount paths before apps hardcode them.
vault secrets enable -path=secret kv-v2 vault secrets tune -default-lease-ttl=1h -max-lease-ttl=24h database/
Token hierarchies and orphan tokens — why do they matter?Expert
Child tokens are revoked when their parent is; orphan tokens survive parent revocation. Use orphans carefully for long-running automation; revoke-by-prefix during incidents.
Default token treeing lets you revoke a batch of job tokens by revoking the parent. Orphans — and some auth methods — break that chain: useful for independence, dangerous if forgotten. Prefer short TTLs and renewable leases over immortal orphans. Periodic tokens renew forever until explicitly revoked — treat them like standing credentials with monitoring.
vault token revoke -mode=path auth/kubernetes/login # or revoke accessor / prefix during incident response
Interviewer often follows with: When is a periodic token justified?
How do you restrict which namespaces or paths a CI role can read?Intermediate
I'd create a dedicated policy with exact path prefixes, attach it only to that auth role, use short TTLs, and deny list where enumeration is a risk. Separate CI roles per environment.
path "secret/data/ci/prod/*" {
capabilities = ["read"]
}
path "auth/token/revoke-self" {
capabilities = ["update"]
}Vault is sealed after a node restart in production. What is your runbook?Expert
Confirm it's sealed vs down, unseal with the quorum or rely on auto-unseal KMS, verify raft peers and the active listener, then check apps retrying auth. If unseal keys aren't available, escalate to DR restore — don't try to "fix" storage blindly.
Distinguish: process crash — auto-unseal should recover; KMS outage — Vault stays sealed until KMS returns; lost Shamir quorum — organizational emergency. Health endpoint and vault status guide automation. Apps should treat Vault blips as retryable; critical control planes need cached credentials with short remaining TTL only. After recovery, review audit for what failed and whether any break-glass was used.
vault status curl -s https://vault:8200/v1/sys/health vault operator members # raft
Interviewer often follows with: How do you test auto-unseal failure without taking prod down?
What should never go into Vault as operational practice?Beginner
Don't store unique irreplaceable data only in Vault without backups of recovery material; don't share root tokens; don't disable audit "temporarily"; don't grant sudo on * to app roles.
vault token lookup vault policy read app # expect narrow paths vault audit list # at least one device
How do you renew a lease before it expires?Beginner
Clients call renew with the lease ID — or use Vault Agent to renew automatically. If max_ttl is hit, they have to check out a new secret instead of renewing forever.
vault lease renew -increment=30m <lease_id> # Agent/CSI typically renew for the app
Namespaces (Enterprise) vs path prefixes for multi-tenant Vault — how do you isolate teams?Expert
Path prefixes with tight policies work on OSS; Enterprise namespaces give stronger admin isolation per tenant. Either way, separate auth roles, no shared highly privileged tokens, and audit per tenant.
OSS multi-tenancy is mostly policy discipline on shared mounts — a Vault admin still sees everything. Namespaces delegate admin within a boundary and reduce blast radius of policy mistakes. For SaaS platforms, combine namespaces or separate clusters with per-tenant mounts and KPIs on leaked cross-tenant reads. Never give tenants root or unrestricted policy write.
path "secret/data/team-a/*" { capabilities = ["read"] }
path "secret/data/team-b/*" { capabilities = ["deny"] }Interviewer often follows with: When is a dedicated Vault cluster per tenant justified?
response_wrapping — what problem does it solve?Advanced
It wraps a secret in a single-use, short-TTL wrapping token so you can hand a courier credential that unwraps once — limiting exposure if the wrapping token leaks after use or expiry.
Common for delivering AppRole secret-ids or one-time bootstrap material. The wrapping token is useless after unwrap or TTL. Combine with tight creation ACL and audit on unwrap. Don't log wrapping tokens. Cubbyhole under the hood — unwrap needs connectivity to Vault.
vault write -f -wrap-ttl=5m \ auth/approle/role/ci/secret-id vault unwrap <wrapping_token>
Interviewer often follows with: What happens if two systems try to unwrap the same wrapping token?
How do you enable and configure Kubernetes auth at a high level?Intermediate
Enable the auth method, give Vault a way to call TokenReview — reviewer JWT or long-lived SA — then create roles that bind SA name and namespace to policies and TTLs.
vault auth enable kubernetes vault write auth/kubernetes/config \ kubernetes_host=https://kubernetes.default.svc vault write auth/kubernetes/role/app \ bound_service_account_names=app \ bound_service_account_namespaces=prod \ policies=app ttl=1h
What is a Vault lease?Beginner
A lease is Vault's time-bound handle for a dynamic secret or token. It has a TTL, can often be renewed until max_ttl, and when it expires or is revoked the credentials are invalidated.
vault lease lookup <lease_id> vault lease renew <lease_id> vault lease revoke <lease_id>
Pods authenticate to Vault fine in one namespace but get 403 permission denied in another with the same role name. How do you debug?Advanced
I'd check bound_service_account_namespaces and names on the Vault role, the SA token audience and issuer, and whether policies differ — identical role names across mounts still bind explicitly to namespace.
Kubernetes auth roles aren't "same name means access." bound_service_account_namespaces must include the pod namespace; aliases and different auth mounts confuse operators. Also verify TokenReview works for that cluster, projected token expiration, and that the policy path matches secret/data/<ns>/.... Compare vault read auth/kubernetes/role/<name> across envs. Audience mismatches after Kubernetes API changes bite too. Systematic Vault role vs K8s SA binding checks.
vault read auth/kubernetes/role/app # bound_service_account_names # bound_service_account_namespaces kubectl -n other get sa app -o yaml
Interviewer often follows with: How do you safely allow the same app SA name in many namespaces?
Audit device is full and Vault starts failing requests. What is your incident response?Expert
I'd restore audit write capacity or add a second audit device immediately, avoid disabling audit as the first move, and size and monitor audit storage so this can't silently recur.
Vault's safety stance: if configured audit devices can't write, requests fail closed depending on version and config. Response: free disk, fix sink permissions, enable a secondary audit device — file plus socket or syslog — then investigate volume. Don't temporarily disable audit without change control — that's a compliance incident. Long-term: ship audit to a remote sink with buffering, alert on audit failures and disk, and load-test audit throughput. Availability vs integrity tradeoffs with eyes open.
vault audit list vault status df -h /var/log/vault # add standby audit device before removing a broken one
Interviewer often follows with: When, if ever, is disabling audit acceptable during an incident?
You must rotate the root of trust for Vault’s auto-unseal KMS key. How do you do it without sealing the fleet?Expert
I'd follow the seal-wrap/rewrap procedure for the provider: add the new key, rewrap, then retire the old key — never delete the KMS key Vault still needs for existing sealed material.
Auto-unseal ties master key ciphertext to a cloud KMS key. Rotation is a rewrap operation — features vary by seal type — not a casual key delete. High level: create new KMS key version, configure Vault to rewrap, confirm status, update runbooks, then schedule old key disable after verification. Test in non-prod. Losing KMS access means a sealed cluster. Seal migration is a practiced DR procedure with provider-specific docs.
vault status # Seal Type: awskms / azurekeyvault / gcpckms # follow provider rewrap/migrate seal docs before disabling old key
Interviewer often follows with: What's your recovery path if the KMS key is scheduled for deletion by mistake?
A former admin’s personal token still has sudo and was never revoked. How do you hunt and eradicate standing privilege?Expert
I'd list and revoke orphan and long-lived tokens, rotate auth methods the admin controlled, review audit for their entity, and replace human root-like access with SSO groups, short TTLs, and a break-glass procedure.
Standing privilege is an org failure mode. Hunt: carefully list token accessors, identity entity aliases, and git history of policy changes. Revoke accessors, disable orphaned AppRoles, rotate secret-ids, and invalidate OIDC group mappings they influenced. Preventive: no long-lived human tokens, SSO OIDC with group policies, periodic accessor scans, and dual control for policy write. Combine audit forensics with identity hygiene — not only revoke root.
vault list auth/token/accessors vault token revoke -accessor <accessor> vault read identity/entity/id/<id>
Interviewer often follows with: How do you detect newly created orphan tokens in near-real time?
Vault rebooted after a host patch and is sealed; apps are failing secret reads. What is your unseal path and how do you prevent a solo hero unseal?Advanced
I'd confirm seal status, unseal with the Shamir quorum or verify auto-unseal/KMS health, bring standbys in sync, then fix why auto-unseal or runbooks failed so one person isn't the bottleneck next time.
Sealed Vault refuses secrets operations until unseal. Shamir: collect threshold shares via break-glass procedure, never chat apps. Auto-unseal: check KMS IAM, network, and key deletion schedule. Verify vault status on each node, Raft peers, and that audit devices still write. Post-incident: rehearse unseal, monitor seal status, and document who holds shares. Sealed isn't data loss if storage and unseal material are intact — but availability is zero until unsealed.
vault status # Sealed: true → Shamir: vault operator unseal (threshold times) # Auto-unseal: fix KMS; vault operator unseal not used
Interviewer often follows with: What do you do if one Shamir share holder is unreachable during an incident?
A dynamic DB credential lease was orphaned when the app crashed; the DB user still works. How do you find and revoke it?Advanced
I'd list leases for the database mount, revoke by lease_id or prefix, confirm the DB user is dropped, and fix the client to renew/revoke cleanly or use shorter TTLs plus periodic orphan sweeps.
Leases can outlive processes. List under sys/leases/lookup/database/... and vault lease revoke. Also check the DB for users matching Vault's naming pattern. Root cause is usually no shutdown hook, max_ttl too long, or a lost lease_id. Prefer short TTLs, renewable leases with heartbeats, and an agent or sidecar that manages lifecycle. Orphan leases are standing privilege — treat revocation as incident hygiene.
vault list sys/leases/lookup/database/creds/app/ vault lease revoke database/creds/app/<lease_id> vault lease revoke -prefix database/creds/app/
Interviewer often follows with: How do you revoke all leases for one role without touching other roles?
Kubernetes auth suddenly fails cluster-wide with TokenReview errors; pods cannot login to Vault. How do you debug?Expert
I'd verify Vault can reach the API server, that the reviewer JWT or SA still works, and that issuer/audience on projected tokens match the auth config — then renew the reviewer credential if it expired.
Vault kubernetes auth calls TokenReview. Failures: expired reviewer token, RBAC removed from reviewer SA, wrong kubernetes_host or CA, API server network policy, or bound_audiences mismatch after a K8s version change. Compare vault read auth/kubernetes/config with the current SA token iss/aud. Prefer short-lived projected tokens and automated rotation of the reviewer JWT. Separate "Vault down" from "auth method misconfigured."
vault read auth/kubernetes/config kubectl auth can-i create tokenreviews --as=system:serviceaccount:vault:reviewer # fix jwt / host / ca; retry: vault write auth/kubernetes/login ...
Interviewer often follows with: Why might only one namespace's pods fail while others succeed?
An AppRole secret_id was pasted into a Slack channel. What do you do immediately?Advanced
I'd destroy that secret_id — and rotate the role's secret_id if needed — revoke tokens minted from it via audit, rotate any secrets those tokens could read, and switch delivery to response-wrapping with tight TTLs.
secret_id is a credential. Destroy it, tidy if needed, then hunt accessors in audit for the role. Disable unused AppRoles; prefer pull identity — K8s or OIDC — over long-lived secret_ids. Wrap secret_id with wrap_ttl and single unwrap. Treat a chat paste as public disclosure until proven otherwise.
vault write auth/approle/role/ci/secret-id/destroy \ secret_id=<leaked> vault list auth/approle/role/ci/secret-id # audit: find token accessors → vault token revoke -accessor
Interviewer often follows with: When is rotating role_id also required after a secret_id leak?
You must rotate a Transit encryption key while ciphertext from the old version must still decrypt. How do you do it safely?Expert
I'd create a new key version for encrypt, keep older versions available for decrypt, rewrap ciphertext in a controlled job, then schedule min_decryption_version only after rewrap completes.
Transit rotation is versioned: encrypt uses latest; decrypt uses the version in the ciphertext blob. Rotate with vault write -f transit/keys/foo/rotate, rewrap with /rewrap, track progress, then raise min_decryption_version. Never delete key material prematurely. Apps shouldn't assume key version 1 forever. Rewrap is a data-plane migration with progress metrics, not a one-liner.
vault write -f transit/keys/payments/rotate vault write transit/rewrap/payments ciphertext=$CT vault read transit/keys/payments # check latest_version
Interviewer often follows with: What breaks if you set min_decryption_version before rewrap finishes?
Audit device disk is 100% full. Are secrets unavailable, and is Vault “sealed”? How do you respond?Expert
Vault usually isn't sealed — but if audit can't write, requests may fail closed. I'd free or replace the audit sink immediately, add a second audit device, and never just disable audit without treating it as a compliance event.
Seal state is about the master key in memory; audit-full is a different failure mode that can look like an outage. Check vault audit list, vault status — sealed false — and df -h. Remediation: free disk, ship to remote syslog or socket, dual audit devices, alert on audit write failures and disk. Disabling audit removes the accountability Vault was bought for. Distinguish sealed vs audit-block vs storage-full in the first five minutes.
vault status # Sealed? vault audit list df -h /var/log/vault # enable secondary audit before removing broken file device
Interviewer often follows with: Why is enabling a second audit device safer than disabling the full one first?
Dynamic DB creds exhausted the database max_connections; apps flap. How do you stabilize and redesign?Advanced
I'd revoke excess leases, lower TTLs and max leases for the role, fix clients that create a user per request, and move to pooled connections with fewer long-lived Vault users.
Each creds/ read can create a DB user or session. Runaways: hot loops, missing lease reuse, or too many replicas times short TTL churn. Mitigate: revoke -prefix, raise DB limits temporarily, set max_ttl and role max leases, use connection pooling, and cache credentials in the app or agent until renewal. Monitor lease count vs DB connections. Dynamic secrets need capacity planning like any pool.
vault lease revoke -prefix database/creds/app/ vault read database/roles/app # lower default_ttl/max_ttl; fix app to reuse until renew
Interviewer often follows with: How do you spot a single noisy client causing lease storms?
External Secrets Operator cannot sync; pods see stale Secrets. Vault policies look fine. Where do you look next?Advanced
I'd check the SecretStore auth — SA / K8s auth role binding — ClusterSecretStore namespace restrictions, ESO controller logs, and whether the ExternalSecret refresh interval or error backoff is hiding Vault 403s.
ESO path: controller authenticates → reads Vault path → writes K8s Secret. Failures often sit in the SecretStore referenced SA, wrong mount path for KV v2 data/, or truncated policies. Align with the kubernetes auth role bindings. Confirm the synced Secret's annotations and timestamps. Stale Secrets are worse than missing ones if apps keep running on old passwords after rotation.
kubectl describe secretstore vault-backend kubectl describe externalsecret app-db kubectl logs -n eso deploy/external-secrets
Interviewer often follows with: How should password rotation be sequenced so pods pick up new values without downtime?
A break-glass root token was used during an outage and never revoked. How do you clean up weeks later?Expert
I'd revoke the root token accessor from audit, rotate any secrets it touched, regenerate root only into sealed envelopes if still required, and replace break-glass with SSO plus short-lived elevated policies.
Root isn't a normal admin path. Hunt audit for token creation and paths written. Revoke the token, rekey if root generation was informal, and invalidate orphans. Preventive: deny root in day-to-day, MFA SSO, approval for policy write, and break-glass runbooks with dual control. Late cleanup still matters — standing root is a dormant breach.
# from audit: accessor → vault token revoke -accessor vault token lookup -accessor <accessor> # schedule generate-root only with formal ceremony
Interviewer often follows with: What's the difference between revoking a root token and rekeying the master key?
Batch tokens were issued for CI and have no accessor listing you can easily scan. One job was compromised. How do you contain it?Expert
I'd revoke by accessor if known, otherwise revoke the parent or orphan tree and rotate the AppRole or OIDC role the job used, shorten TTLs, and prefer identities with explicit accessors for CI.
Batch tokens are encrypted blobs without server-side lease tracking like service tokens — revocation and inventory differ. Design CI with short TTLs, wrapped secret_ids, and OIDC where possible. If compromised: rotate role credentials, invalidate downstream cloud keys the job obtained, and review audit for the token's operations while it was valid. Know batch vs service token tradeoffs before choosing them for CI.
# prefer OIDC/AppRole with 15m TTL # vault token revoke -self from job cleanup # rotate approle secret_id; revoke orphan accessors found in audit
Interviewer often follows with: When would you choose a service token over a batch token for automation?
KV v2 delete “removed” a secret but an old version was undeleted by a confused operator and apps got the previous password. How do you prevent that class of mistake?Advanced
I'd use destroy for compromised versions, restrict undelete and update in policies, alert on undelete in audit, and treat rotation as write-new-version plus app rollout — not casual undelete.
KV v2 soft delete is reversible; destroy is permanent for a version. Policies should separate delete vs destroy vs undelete. Compromised values need destroy plus rotate downstream. Operationally: document rotation runbooks and block broad update on prod paths. Versioning is a feature and a footgun without policy and audit alerts.
vault kv destroy -versions=3 secret/app vault kv metadata get secret/app # policy: deny undelete on secret/data/prod/*
Interviewer often follows with: Who should be allowed to undelete in production, if anyone?