You attach --rotation-period=2592000s and --next-rotation-time to a secret holding a database password. Thirty days pass and the schedule fires. What actually happens at that moment?

A message lands on your Pub/Sub topic, and nothing else changes until your own handler adds a new version. The schedule is a notification and nothing more. Your subscriber mints the credential, updates the database, and calls versions add.

You can grant roles/secretmanager.secretAccessor across a whole project or on a single secret. Why does this lesson push you toward one secret per identity?

A project-wide grant lets one stolen service account read every secret in the project, while a per-secret grant means a stolen identity opens only the one box it was handed. Scoping the grant to a single secret caps the blast radius, so one leaked identity cannot drain the whole vault.

Your rotation handler mints a new database password, updates the database, calls versions add to create version 2, then in the same script runs versions disable on version 1. Minutes later some running services start throwing authentication errors. What went wrong?

Version 1 was disabled before every consumer had re-read version 2, which broke the ones still holding cached connections or a value read once at startup. You needed an overlap window. Real systems cache the old value, so both versions stay enabled until access logs show reads on v1 have hit zero.

Courses GCP securitySecret Manager & rotation

Secret Manager & rotation

Per-secret IAM, runtime fetch, overlap rotation.

Advanced30 min · lesson 9 of 15

Bake a database password into a container image and you have published it. Anyone who can pull that image runs strings over the layers and reads the value in plain text. It then rides along in every backup and every registry mirror. It never expires on its own. Changing it means rebuilding and redeploying everything that carries it. The first fix most people reach for, an environment variable, leaks in its own way: it turns up in crash dumps and in the boot logs of any service that helpfully prints its own config. Secret Manager ends that pattern. It works like a bank vault full of safe-deposit boxes. The values stay inside the vault, your app walks up to the counter and proves who it is, and the vault hands over the one box that app is cleared to open. Nothing sensitive travels in the image or the deployment file.

One box per identity

Every secret is its own resource with its own guest list. That is the whole design, and it is where teams usually go wrong. The lazy grant hands an app read access over every secret in the project, so one leaked token opens every box in the vault. Bind one identity to one box instead. Your app runs as a service account, a robot identity, a login meant for software rather than for a person. You give that service account the role roles/secretmanager.secretAccessor on that single secret and nothing else, which is IAM (identity and access management, the system that decides who may call which Google Cloud API) saying yes to exactly one box. The point is to cap the damage. If one identity is stolen, it opens one box. Across dozens of services that adds up to a lot of small bindings, and yes, it is tedious. That tedium is what contains a breach: no service account can read anything it was not handed on purpose. The app never carries a key file. It asks for the value at runtime using the identity already attached to it, and every read writes an audit log line naming who fetched what and when. That log line is the thread the next lesson picks up, when we make audit trails you cannot rewrite.

create the box, then drop in the first value

gcloud secrets create prod-db-password \
  --replication-policy=user-managed \
  --locations=europe-west1,europe-west4 \
  --labels=env=prod,team=payments
 
printf '%s' "$INITIAL_DB_PASSWORD" \
  | gcloud secrets versions add prod-db-password --data-file=-
 
# output:
Created secret [prod-db-password].
Created version [1] of the secret [prod-db-password].

one identity, one secret, fetched at runtime

gcloud secrets add-iam-policy-binding prod-db-password \
  --member="serviceAccount:[email protected]" \
  --role="roles/secretmanager.secretAccessor"
 
# output:
Updated IAM policy for secret [prod-db-password].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/secretmanager.secretAccessor
etag: BwYX2n8s3kA=
version: 1
 
gcloud secrets versions access latest --secret=prod-db-password
 
# output (the raw payload, nothing else):
s3cr3t-P@ssw0rd-v1

Before you rotate anything, learn how versions behave. They are append-only, like pages in a ledger. You never overwrite version 1. You add version 2, and latest is an alias pointing at the newest enabled version. That immutability is what makes rotation safe, because the old value is still sitting there when the new one misbehaves. Retiring a version gives you two verbs, and the difference is expensive. disable hides the value and you can switch it back on. destroy shreds the material for good: not recoverable from a backup, not recoverable from a support ticket. Use disable during rotation. Save destroy for the day you are certain. The --locations flag above pins which regions hold copies, which is how you keep data inside a country's borders when the law says you must. You can also wrap a secret with your own key from Cloud KMS (key management service, where you hold the encryption keys yourself) through the replication policy, so the key custody from the KMS lesson carries straight over. One more shape worth knowing: the secret can live in a central project while workloads in other projects reach across to it through IAM, so a break-in in one app project does not hand over the whole vault.

Rotation is an alarm clock, not a locksmith

Here is the part people get wrong, and getting it wrong in production is expensive, so learn it once. Setting a rotation period on a secret does not change the secret. Secret Manager has never met your database. It cannot invent a valid new password, and it would not write one into the stored value even if it could. All the schedule does is ring an alarm clock. On the interval you set, it publishes a short message to a Pub/Sub topic (publish and subscribe, Google's message queue: a mailbox that pushes notes to whatever has signed up to listen). Your own code has to be the thing listening. If nothing is subscribed, the reminder fires into an empty room and the password quietly ages past its expiry. That listener, a small program you run as a Cloud Run service or a Cloud Function, is the locksmith. It mints a new credential, changes it in the real database, then stores the new value as a fresh secret version. The clock reminds you. You still change the lock and tell the bank.

wire the alarm clock: topic, publisher rights, schedule

gcloud pubsub topics create secret-rotation
 
PROJECT_NUMBER=$(gcloud projects describe payments-prod --format='value(projectNumber)')
gcloud pubsub topics add-iam-policy-binding secret-rotation \
  --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-secretmanager.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"
 
gcloud secrets update prod-db-password \
  --add-topics="projects/payments-prod/topics/secret-rotation" \
  --rotation-period="2592000s" \
  --next-rotation-time="2026-08-15T02:00:00Z"
 
# output:
Created topic [projects/payments-prod/topics/secret-rotation].
Updated IAM policy for topic [secret-rotation].
Updated secret [prod-db-password].

What makes rotation safe is overlap. When your handler adds version 2, do not disable version 1 in the same breath. Real systems cache. A connection pool can hold a database session opened with the old password for minutes. An app that reads latest once at startup can hold it for hours. Disable the old version before every consumer has re-read the new one and reconnected, and you caused the outage yourself. Never disable it, and rotation was theatre. So you run a window. Both versions stay valid, you watch the access logs until reads on the old version drop to zero, then you retire it. The audit log names the exact version each caller read, so you filter for reads of version 1 and wait for that count to reach zero. Size the window to the longest cache or connection lifetime any consumer holds, then add margin.

rotate with overlap, retire the old version last

printf '%s' "$NEW_DB_PASSWORD" \
  | gcloud secrets versions add prod-db-password --data-file=-
 
# output:
Created version [2] of the secret [prod-db-password].
 
# ...hours later, after access logs show zero reads on version 1:
gcloud secrets versions disable 1 --secret=prod-db-password
 
# output:
Disabled version [1] of the secret [prod-db-password].

The rotation cycle

1Schedule fires

Secret Manager publishes a SECRET_ROTATE message to Pub/Sub

2Handler wakes

your Cloud Run subscriber picks up the message

3Rotate the source first

set a new password on the actual database

4Add a new version

versions add, so 'latest' now points at v2

5Overlap window

v1 and v2 both valid while consumers refresh

6Disable v1

retire the old version only after reads on it hit zero

The schedule only sends the reminder. Every step after it is code you own.

The service agent has to be allowed to publish

Secret Manager sends rotation messages as itself, through a Google-managed account called a service agent, named service-PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com. If that account does not hold roles/pubsub.publisher on your topic, attaching the rotation config fails on the spot. The nastier case comes later: strip the binding after the fact and the alarm clock goes silent, with no error showing on the secret itself. Grant the publisher role on the topic before you attach it. And if you also protect the secret with a Cloud KMS key, that same service agent needs cryptoKeyEncrypterDecrypter on the key, or reads and new versions start failing with a permission error.

You get the payoff from Secret Manager only when the value never travels in the image, in the deployment YAML (the text file that describes what to run), or in the CI (continuous integration, your build pipeline) variable store. The app fetches at runtime with its own service account, ideally through a client library that caches briefly and refreshes when a new version appears. Pinning to latest is convenient. Pinning to a version number is safer when you want a rotated password promoted deliberately rather than the second it lands.

Three knobs tighten the same model: the replication policy, CMEK on secrets (customer-managed encryption keys, your own KMS key wrapping the stored value), and IAM conditions, which attach a rule to a grant so it only applies when some attribute of the caller matches. Keep the boundary this course keeps drawing clear in your head while you use them. IAM decides who may call the Secret Manager API. VPC Service Controls decides where that call is allowed to originate from, and it is the perimeter, not the role, that stops a valid identity reading your secret from a laptop in a coffee shop. Disable old versions after cutover instead of deleting them mid-incident. And do not grant secretAccessor to a human group in production. People reach production secrets through a ticketed break-glass path or through impersonation, and both leave a log line behind.

Create the secret, its single binding and its rotation topic in the same reviewed change that creates the service account, and let a pipeline apply it. A secret wired up by hand in the console has no owner and no reviewer, and the binding someone widened at 2am during an incident stays widened. Write three things down beside that module: which project holds the secret, which identity is allowed to add versions to it, and the Cloud Logging query that shows the last successful rotation. A new teammate should be able to rotate prod-db-password from that page alone, in the middle of the night, without hunting you down.

Prove both halves rather than assuming them. The access half: as an identity that should not be able to read the secret, run the access command and keep the permission-denied output next to the change ticket. The rotation half: in a lab project, set a next rotation time a few minutes out, then confirm the message actually reaches your subscriber and that your handler creates the new version. A schedule nobody has watched fire is a guess, and a handler that has never been triggered is a guess with a timer attached. Re-run both checks every quarter, because a per-secret binding that quietly widened back to project scope is the failure you will not notice any other way.

Try this

Do this in a lab project. Create a secret, grant accessor to exactly one runtime service account, then check whether your own user identity can read the payload without impersonating that account.

terminal

echo -n 's3cr3t-db-pass' | gcloud secrets create db-password \
  --data-file=- --replication-policy=automatic --project=payments-prod
gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:[email protected]" \
  --role=roles/secretmanager.secretAccessor --project=payments-prod
gcloud secrets versions access latest --secret=db-password --project=payments-prod
# expect success only if YOUR account also has accessor; revoke user grants in real prod

output

Created version [1] of the secret [db-password].
 
Updated IAM policy for secret [db-password].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/secretmanager.secretAccessor
 
s3cr3t-db-pass

Takeaway

One secret resource per value. Accessor granted only to the runtime identity that needs it. And rotation that finishes with your app reading a new version, not a calendar reminder that changes nothing.

Next you will make every read of that secret and every admin change land in logs nobody can quietly edit. A vault with no camera on it is half a control.

HSM, EKM & key custodyAudit logs & immutable trails

IAM mental modelAWS security for DevOps engineers Multi-account & org identity architectureAdvanced cloud security The IAM evaluation modelAWS security engineering Entra ID, Conditional Access & PIMAzure security Python for DevSecOps cheat sheetCheat sheet DevSecOps & supply-chain security interview questionsInterview prep

Secret Manager & rotation

One box per identity

Rotation is an alarm clock, not a locksmith

Try this

Takeaway

Related