When a secret leaks

Revoke first, then clean history and notify.

In short. The hour a secret leaks, speed beats elegance. A key sitting in a public GitHub commit is being scanned by bots within minutes.

The hour a secret leaks, speed beats elegance. A key sitting in a public GitHub commit is being scanned by bots within minutes. Your job is contain → assess → eradicate → recover → learn — not to perfect a history rewrite while the key still works.

This runbook is intentionally short. Print it. Practice it. The secrets-found course ends here because foundations without response still fail closed.

Contain first

Revoke or rotate the credential in the system that issued it (cloud IAM, IdP, database, SaaS admin). Disable CI variables. Invalidate sessions if applicable. Only then start git filter-repo, ticket updates, and blame analysis. If you cannot revoke immediately, throttle or disable the dependent integration.

# example: GitHub PAT detected
# 1) revoke at github.com/settings/tokens (or org API)
# 2) grep cloud trails / SaaS audit for use of the token
# 3) then history purge if the repo retained it
echo 'revoke → assess → purge → notify'

revoke → assess → purge → notify

Assess and clean

Pull audit logs for the credential's lifetime window. List resources touched. Preserve evidence before wiping. Purge git history only after rotation, and force-coordinate with everyone who cloned. Rebuild images that baked the secret. Invalidate chat pastes by rotating — you cannot un-index Slack.

Going deeper

Legal and customer notification may apply depending on what the secret protected. Involve your IR lead early; do not wait until history rewrite debates finish. Preserve timelines: when created, when leaked, when revoked, where used.

Honeytokens (canary keys) help detect scanning of private repos and CI logs. If a canary fires, assume broader exposure and accelerate rotation of nearby real secrets.

After action: add a scanner rule, remove the root cause (for example, forbid AKIA in CI), and train the team with the sanitized story. Incidents that only rotate without fixing process will recur.

Coordinate with cloud provider abuse teams if keys were used for mining or spam. Their timelines and blocklists help containment. Keep a contact sheet next to the revoke steps so on-call does not invent process at 3 a.m.

Close the loop with a tracked action item: scanner rule, CI condition, or owner change. An incident ticket that only says "rotated" without a preventive control is incomplete.

Try this

Tabletop: pick a fake leaked AWS key scenario and write the first five actions with owners and time targets (<15 minutes to deactivate).

printf '%s\n' '1. Deactivate access key' '2. CloudTrail lookup' '3. Rotate dependent secrets' '4. Invalidate CI vars' '5. Incident channel + timeline'

1. Deactivate access key
2. CloudTrail lookup
3. Rotate dependent secrets
4. Invalidate CI vars
5. Incident channel + timeline

Takeaway

Revoke first, then clean and learn. Foundations — scanning, managers, rotation — exist to make this hour rare and short.

When you outgrow these foundations, move to Vault production and advanced secrets courses for dynamic credentials and platform-scale delivery.

Related