The secrets threat model & secret-zero

Sprawl, standing privilege, blast radius, and the bootstrap-trust problem.

Your production database password probably lives in more places than one right now. There is the row in Vault, sure. There is also a git commit from last spring, a continuous integration (CI) log from a build that failed at 2am, and a copy somebody pasted into a Helm values file so the deploy would go through. An attacker does not have to beat the strongest copy. They pick the weakest one. So before you choose Vault or a cloud secrets manager, write a threat model: a short, honest list of who could steal what, and from where. Three questions get you there. What counts as a secret? Every place that secret rests or travels through? Who is the adversary at each of those places? Almost no breach starts with broken cryptography. It starts with a password nobody was watching, read by an account that could reach far too much.

Every secret lives in four states, and all four leak

Water is easy to guard in a sealed bottle and hard to guard once it is in a glass, a pipe, or somebody's mouth. Secrets behave the same way. A password is exposed differently depending on which of four states it is in, and a design that defends one state is not a design. At rest, it sits in storage: etcd (the key-value database behind a Kubernetes cluster), a Terraform state file, last night's backup, a KMS (key management service, the cloud box that holds encryption keys). In transit, it crosses the network, hopefully inside TLS (Transport Layer Security, the encryption behind HTTPS), though it often crosses a CI log or an error message too. In use, it is decrypted in the memory of a running process, readable by anything that can look at that process. In the audit trail, it leaves a record of who read it and when, which is sensitive all on its own. Defend only the "at rest" state, the part every encryption product advertises, and you have left the other three wide open.

Walking each state out loud is what makes the model worth writing. A database password at rest inside Vault is fine. The same password sitting in a crash dump on a shared node is not. A token in transit over TLS is fine. The same token echoed into a build log because somebody left set -x on (the shell option that prints every command as it runs) is not. Teams that map storage and stop there miss the surfaces attackers actually reach for.

Audit logs need the same care. Vault fingerprints sensitive values in its audit log using HMAC (hash-based message authentication code, a one-way hash keyed with a secret), which beats writing them in cleartext. The log still says who read which path, and when. That is correlation data, and an insider or a stolen SIEM (security information and event management, the platform where all your logs get searched) account can mine it happily. Ship audit events to storage nobody can rewrite, and treat the log itself as sensitive material.

Sprawl, standing privilege, and blast radius

Three things decide how bad a leak turns out to be. Sprawl is the count of independent copies. Every copy is another door, and another thing you have to find on rotation day. Standing privilege is how long a credential stays valid and how much it can do while nobody is using it, the way a spare key under the doormat works at 3am whether or not you are home. A database password that is valid for a year and grants full access carries enormous standing privilege. Blast radius is what one stolen secret opens. A read-only credential scoped to a single database has a small radius. A shared "app" password reused across six services has a huge one.

Inventory is how you measure sprawl. If you cannot answer "how many places hold this credential?" then you cannot rotate it safely, and you certainly cannot revoke it in a hurry. Standing privilege is the number dynamic secrets push down, because the normal state of every credential becomes "about to expire." Blast radius is the number policy and scoping push down. Different problems, different levers, and you need both.

Book a quarterly copy hunt for your ten most valuable credentials. Search every repository with grep. Read the Helm values files. List the CI variables. Dump every Kubernetes Secret and see which ones hold the same string. The result is usually embarrassing, and the embarrassment is the useful part: it turns a vague worry into a list with names and line numbers on it.

Secret-zero: the credential you need before you have any credentials

Most offices keep spare keys in a cupboard, and that cupboard has a lock of its own. So where does the cupboard key go? Every secrets system hits the same question at the very start. A workload has to authenticate to the secrets manager before it can fetch anything, and authenticating needs a credential, which is itself a secret. That first credential is secret-zero. Ship it as a static token in an environment variable or a Kubernetes Secret and you have solved nothing. You have renamed the problem. That token now carries the standing privilege to read everything, and it lives in exactly the leaky spots the diagram above lists: environment variables, mounted files, and etcd.

Match the control to the adversary, not to the tool

Different attackers arrive at different doors. Someone who has taken over a CI runner reads environment variables and whatever the build cache kept. An insider with kubectl exec opens a shell inside your running pod and reads the mounted secret file. A supply-chain attacker who forked your repository goes mining through git history for the password you deleted in 2023 and never rotated. Your threat model should list each of those people, the surfaces each one can touch, and the single control that shrinks exposure there: dynamic credentials at runtime, pre-commit scanning for git, platform authentication for secret-zero, audit for insiders.

Skip that mapping and you end up encrypting etcd with great ceremony while the same password sits in twelve Helm values files. The control has to answer the adversary and the surface. Not the tool your company bought last quarter.

cat /proc/$(pgrep -f myapp)/environ | tr "\0" "\n" | grep -i token

APP_DB_PASSWORD=s3cr3t-shared-prod
 
# readable by any user on the host with permission to read /proc
# follows the process into crash dumps, child processes, and
# any log line that dumps the environment on error

kubectl exec -n prod payments-0 -- printenv | grep -i VAULT
kubectl get secret vault-token -n prod -o jsonpath="{.data.token}" | base64 -d | head -c 20

VAULT_TOKEN=hvs.CAES...
 
hvs.CAESIJ...   # durable token in etcd + visible inside the pod

What the model tells you to build first

The model picks your first project for you. If standing privilege is your worst number, dynamic secrets and short TTLs (time to live, how long a credential stays valid) go first. If sprawl is the pain, fifty repositories each holding a copy, centralize in one manager and deliver at runtime instead. If blast radius is the thing that keeps you awake, scope a policy per workload and stop reusing credentials across services. If secret-zero is broken, fix authentication before you switch on another engine. Bolting engines onto a broken front door is decoration.

Write it down. Review it every time you add an integration. And treat "we use Vault" as a detail about one control, never as a stand-in for the model itself. A secrets program that cannot name its adversaries and its surfaces is a shopping list wearing a design's clothes.

Attack your own diagram on paper

Before you buy anything, spend twenty minutes playing the attacker against your own diagram. Pick the production database password. Where could you steal it without ever touching Vault? A CI log. A backup nobody encrypted. A developer laptop. A shared Kubernetes Secret that three teams mount. For every hit, name the control you will actually operate, not "we will be careful." If your list of controls comes back empty, what you have is static secrets with extra steps.

Then run the same drill on secret-zero. An attacker lands inside a pod. What can they authenticate as? If the answer is "anything with this namespace's shared service account," your Kubernetes auth binding is too loose and you have found this week's work. The model is a living document. Update it when you add a cluster, when you add a CI system, and when somebody signs up for a SaaS (software as a service) tool that stores API keys on your behalf.

vault read -field=token auth/token/lookup-self 2>/dev/null | head -c 16; echo
vault token lookup -format=json 2>/dev/null | jq ".data.ttl,.data.policies"

hvs.CAESIJ...
 
1200
["payments-read"]
# short TTL + narrow policy = small blast radius if stolen

Numbers worth putting on a dashboard

Measure the things you intend to move. Count the static secrets you own and watch that number fall quarter by quarter. Track the median credential lifetime and watch it fall. Time yourself revoking a credential during a drill, and watch that fall too. Count the auth methods still bootstrapped with a token somebody shipped by hand, and drive it to zero. A dashboard showing Vault uptime tells you the server is alive. It tells you nothing about whether the program is shrinking risk.

Put a name next to every number. Each static secret in the inventory gets an owning team and a date it will be rotated. Secrets with no owner do not get rotated. They get forgotten, and then a scanner finds them in git for you, usually on a Friday.

gitleaks detect --source ./deploy --redact --exit-code 1 2>&1 | tail -5

Finding: AWS_ACCESS_KEY in deploy/staging.env
WRN leaks found: 1
# threat model says: treat as burned, rotate, then clean history

Write the threat model on one page before you open a pull request full of Vault policies. Name the credential. List every copy you know about. Pick the adversary most likely to find it first: a contractor with kubectl and time on their hands, a build job that prints its whole environment when it fails, a laptop left open at a conference. Then assign exactly one control per surface. Rotate and delete the copy in git. Move the app to dynamic credentials. Bind Kubernetes auth to one service account instead of the namespace default. That page becomes the checklist for next quarter's review, and it outlives the kickoff deck by years.

When someone senior asks why you are not moving everything into the cloud provider's secrets manager and calling it finished, answer with the model. If the leak surface is process memory and crash dumps, a second encrypted store changes nothing at all. If the surface is a shared password that never expires, dynamic secrets and short lifetimes change a great deal. Tool choice follows the adversary and the surface. It never runs the other way.

Try this

Run these against a non-production Vault and a throwaway namespace. You are doing two things at once: hunting for copies you did not know existed, and measuring how much standing privilege your own token is carrying. Same walk the paper exercise asked for, now with real output to argue about.

vault token lookup -format=json | jq "{ttl:.data.ttl, orphan:.data.orphan, policies:.data.policies}"
kubectl get secrets -A -o json | jq -r '.items[] | select(.type=="Opaque") | "\(.metadata.namespace)/\(.metadata.name)"' | head -20
gitleaks detect --source . --redact --no-git 2>&1 | tail -8

ttl: 2760
orphan: false
policies: ["payments-read"]
 
payments/db-creds
payments/vault-token
ci/aws-keys
...
 
Finding: generic-api-key
File: deploy/staging.env:12
# threat model action: treat as burned, rotate, scrub history

Takeaway

Threat modeling for secrets is inventory, adversaries, and surfaces. No product name substitutes for any of the three. Shrink sprawl, standing privilege, and blast radius on purpose, and never answer secret-zero by shipping one more long-lived token.

Your next move: take one production credential, walk it through all four states, and put a date on the control that closes the weakest one. The architecture lessons ahead (Vault's storage model, dynamic engines, workload identity) only make sense once that walk exists in writing.

Related