Detecting & responding to leaked secrets

Org-wide scanning, honeytokens, and a revoke-first runbook.

A contractor pastes an Amazon Web Services access key into a support ticket so an engineer can reproduce a bug. Two days later the same key turns up in a public gist (a small code snippet anyone on the internet can read and search). Nobody meant any harm. The key leaked anyway. Prevention never reaches 100 percent, so a grown-up program plans for leaks and measures time-to-revoke: the gap between a secret leaving your control and that secret being worthless. Two things keep that gap short. Secret scanning, which is automated pattern-matching for credential shapes across your repositories and logs. And honeytokens, fake credentials that page you the second anyone touches them.

Three places to scan: the laptop, the pipeline, the whole org

Scan in three places, because each one catches a different kind of mistake. A pre-commit hook (a small script git runs before it records your change) using gitleaks or trufflehog stops most secrets before they ever become a commit. That is the cheapest catch you will ever get. A CI job (continuous integration, the server that builds and tests every push) scans each push and, importantly, the entire history of any new branch, then fails the build on a hit. On top of both, a scheduled sweep runs across every repository in the organization, because the leak that hurts most is usually the one committed years before you had any controls at all.

Each layer covers the previous one's gap. The hook catches the person in a hurry. CI catches the person who skipped the hook. The org sweep catches the past. Turn on --redact in CI so a failed scan does not print the secret a second time into build logs that half the company can read. Leaking the same key twice, once by accident and once through your own scanner, is a bad day made worse.

gitleaks detect --source . --redact --exit-code 1 --report-path gl.json
cat gl.json | jq ".[].Description"
# CI fails the pipeline on non-zero exit

Finding: AWS_ACCESS_KEY_ID in deploy/staging.env (redacted)
"AWS Access Key in environment file"
# build blocked — developer must rotate and purge before merge

trufflehog git file://. --since-commit HEAD~50 --only-verified
# org-wide scheduled job:
trufflehog github --org=acme --only-verified --json > findings.json

{"DetectorName":"AWS","Verified":true,"Redacted":"AKIA...REDACTED"}
# verified = tested against provider API, not just regex
 
Wrote 3 verified findings to findings.json

Honeytokens: an alarm that never cries wolf

A honeytoken is a decoy credential that no real system ever uses. A fake AWS key sitting in an S3 bucket (Simple Storage Service, Amazon's file store), a decoy database user, a canary token dropped into a config file. Wire an alert to fire the instant anything touches one. Since no legitimate workload has any reason to use it, every single use means somebody is inside who should not be. That is about as close to a zero false positive signal as security ever gets.

They cost almost nothing to plant and they are miserable for an attacker. Someone holding a pile of stolen credentials cannot tell the decoy from the live one, so the moment they try the wrong key they announce themselves. Sprinkle them through repositories, buckets and internal wiki pages. Rotate a honeytoken once it starts showing up in your own scan results, otherwise your team learns to ignore the very alert you planted.

aws iam create-access-key --user-name honeytoken-canary
# plant AKIA...CANARY in decoy config; alarm on any API call:
aws cloudwatch put-metric-alarm --alarm-name honeytoken-used \
  --metric-name UserAuthentication --namespace AWS/IAM ...

{
  "AccessKey": {
    "AccessKeyId": "AKIA...CANARY",
    "Status": "Active",
    "UserName": "honeytoken-canary"
  }
}
Alarm honeytoken-used created
# any use of this key = pager — zero legitimate traffic

Three jobs: stop it, spot it, kill it

Revoke first, ask questions later

When a real secret leaks, the order is fixed. Revoke, then investigate. Kill the credential first: revoke the Vault lease or token, which cascades to everything that token ever minted, flip the cloud access key to inactive, rotate the static password. That closes the attacker's window right now. Tracing blast radius (everything the credential could reach) and root cause happens afterwards, with the pressure off.

Doing it the other way round, wanting to understand the incident before touching anything, is how a contained leak becomes a breach. Short-lived dynamic credentials help here, though not in the way people assume. They do not remove the need to trust the systems handing them out. What they change is how much an attacker gets from one stolen copy, and how fast you can take it back. One command against a lease tree kills hundreds of derived credentials in seconds, instead of an afternoon of coordinated password changes across six teams.

Your audit logs are a detector too

Git scanning only finds secrets that ended up in a file. Plenty never do. Vault's audit log is a second sensor: a sudden burst of decrypt calls, an identity you do not recognize reading production paths, tokens created from an address range (a block of network addresses, written as a CIDR) where none of your servers live. On the cloud side, AWS CloudTrail (a record of every API call made in the account) can alert on CreateAccessKey and on Secrets Manager GetSecretValue coming from principals (the users or roles making the call) that have no business asking. That catches the credential pulled straight from a running process, which no repository scanner would ever see.

Send the high severity alerts to the same on-call rotation that gets paged for a production outage. A honeytoken waking someone at 3 a.m. is the system working exactly as designed. A finding that lands in a JSON report nobody opens is theater with a budget line.

vault token revoke -self
# or cascade: vault lease revoke -prefix database/creds/payments-ro/
aws iam update-access-key --access-key-id AKIA...LEAKED --status Inactive
aws iam get-access-key-last-used --access-key-id AKIA...LEAKED

Success! Revoked token (or entire lease subtree)
# attacker window closed
 
Status: Inactive
LastUsedDate: 2026-07-24T08:14:00Z  # note time for blast-radius window

Secrets hide in image layers

A container image is built in layers, like sediment. Every build step leaves a permanent stripe. If someone passes a password through ENV or ARG at build time, that value is fossilized in the layer history even when the finished image shows an empty environment. Scan images in CI with trufflehog's filesystem mode or a dedicated image scanner, and block promotion into your production registry on a verified finding, the same way you block a merge.

Better still, keep the secret out of the build entirely. Have the pipeline fetch a short-lived credential from Vault or your cloud secrets manager using OIDC federation (OpenID Connect, where the pipeline proves who it is and receives a temporary token in return) rather than reading a CI variable named DB_PASSWORD. When a build genuinely needs a secret, assume it is baked into a layer and rotate it the moment the build finishes.

trufflehog docker --image registry.acme.internal/payments:2026-07-24 --only-verified
docker history registry.acme.internal/payments:2026-07-24 --no-trunc | grep -i password || echo "no obvious password in history"

{"DetectorName":"GitHub","Verified":true,"SourceMetadata":{"Docker":{"layer":"sha256:abc..."}}}
no obvious password in history
# image scan catches layers git scan misses

Measure time-to-revoke in a drill, not on a slide. Run a tabletop exercise where the planted leak is a genuine canary key, and put a stopwatch on it: alert fires, key goes inactive or the lease dies, stop the clock. Leadership remembers one number, minutes to containment. Nobody has ever been impressed by a count of scanner integrations.

After an incident, lay three timelines side by side: what the scanner found, what the Vault audit log says, and what CloudTrail says. If the key sat in git for nine days but was first used an hour before you noticed, then your real detection gap is that one hour, and another scanner would not have changed a thing. Close that gap before buying more coverage. Breadth without automatic revocation only makes more noise.

Once a quarter, have someone plant a canary key inside a CI build artifact without warning the on-call engineer, then measure pager to revoke. Now the number is repeatable and comparable, which makes it a dashboard line for leadership instead of a story the security team tells at retros.

Teach support and sales staff never to paste a customer's credential into a ticket. A helpdesk becomes a secret store nobody planned for: looser access control than git, no scanning at all, and a retention policy measured in years.

Put the scan result inside the review itself. A checkbox on a merge request template is weak, because people tick boxes without reading them. A required link to the scan job's artifact forces the author to look at the result before a human reviewer spends a single minute on the change.

Block force-push to your default branch unless the scan has cleared. Every developer having gitleaks on their laptop counts for nothing if a protected branch can be overwritten by a push that went around every local hook.

Publish the mean time-to-revoke after each drill. A line trending down is evidence the program works. A flat line means you have been buying products and leaving the runbook untouched.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue --start-time 2026-07-24T00:00:00Z | jq ".Events[].Username"
vault audit list

[
  "payments-worker",
  "unknown-principal"
]
# investigate unknown-principal — may be leaked key
 
File + Syslog audit devices enabled

Detection starts from the assumption that prevention already failed. So look in the places people forget: git history, container layers, CI logs, ticket attachments. Hooks catch today's mistake. Historical scanners catch the password committed three years ago by someone who has since left the company, which still works fine. Treat every hit as burned until it is rotated. "It was only staging" is the sentence people say right before staging credentials turn out to reach production.

Go easy with allowlists. Every path you tell the scanner to ignore is a blind spot you created deliberately, and those outlive whoever added them. Fix the secret and take it out of the history instead of muting the path forever.

Scanner failures should block merges. Production log scanners should page. A finding sitting on your main branch with no rotation ticket attached is an open incident wearing a friendlier label.

Try this

Point a scanner at a sample repository and check that the exit code would stop a merge. Then plant a known test secret and prove the tools actually find it.

gitleaks detect --source . --redact --exit-code 1
trufflehog git file://. --only-verified=false 2>&1 | tail -15
git log -p --all -S 'AKIA' -- '*.env' | head -20

Finding: aws-access-token
File: charts/payments/values.yaml:42
Secret: AKIA...REDACTED
Commit: 9f3c2aa
 
Found verified candidate...
# exit code 1 -> CI fails the merge
# history search shows when AKIA first entered the tree

Takeaway

Work from the assumption that a copy of every secret already exists somewhere you do not control. The job is finding those copies quickly, killing them faster, and shutting the door they came through. A scanner with no rotation playbook behind it produces dashboards, never containment.

Next: switch on a gitleaks check in the pre-commit hook or CI job of your busiest repository, then open a rotation ticket for every finding older than a week. Not a triage ticket. A rotation ticket.

Related