Your team already encrypts a credentials file with a KMS key and commits it to the repo. What does moving that credential into a managed secrets service actually add?

Per-principal access, versioning, scheduled rotation, and an audit event on every read. The manager wraps the value in a lifecycle the static file cannot provide.

In one Azure Key Vault you have a cryptographic key db-cmek with a rotation policy and a secret prod-api-db holding a database password. Ninety days later, which has changed on its own?

Only the key, because Key Vault rotates key material but has no generator for secrets. Keys rotate their own material under a policy, while a secret only gets an expiry and a near-expiry event you must handle.

You rotate a database secret on AWS. The new version is AWSCURRENT and the backend accepts it. Within a minute one long-running service starts failing authentication while freshly started pods work fine. Most likely cause and fix?

The long-running service cached the old credential and never re-read; keep an overlap window and make consumers refresh before retiring the old version. Rotation without re-read breaks whatever still holds the old value, and the dual-secret overlap plus a refresh prevents it.

Courses Advanced cloud securitySecrets managers & rotation across clouds

Secrets managers & rotation across clouds

Managed secrets, rotation, and a multi-cloud strategy.

Advanced30 min · lesson 9 of 15

A house key is cut once and works forever. Whoever copies it, a curious locksmith, an ex-tenant, a thief who pressed it into wax, keeps that access until you physically change the lock. A hotel key card is a different animal. The front desk issues it under your name, programs it for one room, sets it to stop working at checkout, records every door it opens, and can deactivate it from the lobby the moment you report it lost. A production secret, a database password, an API token, a signing key, should behave like the key card and never like the house key.

A configuration value is data your program reads. A secret is a value with a lifecycle wrapped around it: issued to an identity, scoped to one resource, rotated on a schedule, revoked on demand, and logged on every read. A KMS-encrypted file (KMS is a Key Management Service, the cloud component that holds encryption keys) committed to a repository gives you the encryption and none of that lifecycle. A managed secrets service, AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, or HashiCorp Vault, adds four pieces of machinery on top. Fine-grained IAM (Identity and Access Management, the rules that say which principal may do what) binds each secret to named principals, so a workload can read prod/api/db and nothing else. Versioning lets a rotation roll forward, then roll back without a redeploy if the new credential turns out bad. A schedule replaces the human who used to change passwords once a quarter and forgot. And an audit event fires on every retrieval, which is the record that later tells you which workload read which secret, and when.

Reading a secret without shipping one

Here is the chicken-and-egg problem. To read a secret from the manager, your app must prove who it is, and if it proves that with another secret, you have only moved the problem: now there is a first credential to leak. Workload identity closes the loop. The platform vouches for the running workload, the way a building's badge reader recognizes an employee by their badge instead of asking for a password at every door. On AWS that badge is an instance profile (an IAM role bound to the virtual machine) or IRSA (IAM Roles for Service Accounts, which maps a Kubernetes service account to an IAM role). On Google Cloud it is GKE Workload Identity (GKE is Google Kubernetes Engine), tying a Kubernetes service account to a Google one. On Azure it is a managed identity, an identity the platform attaches to the resource and whose credentials it rotates for you. Across providers, workload identity federation extends the same idea over OIDC (OpenID Connect, a token-based way to prove identity), so a workload in one cloud can read a secret in another without a stored key. The read then travels over an authenticated call, and nothing sensitive ships in the image, the task definition, or Git.

terminal

# The app authenticates with its instance/pod identity, not a stored key.
# Every one of these reads is written to the audit log.
aws secretsmanager get-secret-value \
  --secret-id prod/api/db \
  --query SecretString --output text

output

{"username":"api","password":"9f3c8e21b7a4","host":"db.prod.internal","port":5432}

terminal

# GKE Workload Identity signs the call; there is no key file on disk.
gcloud secrets versions access latest --secret=prod-api-db

output

{"username":"api","password":"9f3c8e21b7a4","host":"10.24.0.5","port":5432}

terminal

# A managed identity on the pod or VM authenticates; --query pulls the value.
az keyvault secret show \
  --vault-name acme-prod-kv --name prod-api-db \
  --query value -o tsv

output

{"username":"api","password":"9f3c8e21b7a4","host":"10.30.0.7","port":5432}

Read those three again and notice what they share: no password on the command line, no key file, no environment variable carrying a credential into the process. The secret arrives over a call the platform already authenticated, and the manager writes a line in its audit trail for every access. That line, principal, secret, version, timestamp, is the signal the next lesson on immutable audit trails builds on. If an attacker ever does read prod/api/db, this is where you see it, so long as you are actually collecting and watching those events rather than letting them pile up unread.

Rotation means three different things

Rotation is where multi-cloud teams get burned, because the same word names three different behaviors. Reprogramming a hotel key card is one clean action at the front desk. Rotating a secret is nowhere near that uniform across clouds, and treating the three as equivalent is how a credential you believe changes monthly quietly stays static for a year.

AWS Secrets Manager owns the entire sequence. You point a secret at a rotation function (a Lambda, AWS's run-on-demand serverless function) and a cadence, and from then on the service creates a brand-new credential on the backend, tests it, promotes it, and retires the old one with no human present.

terminal

aws secretsmanager rotate-secret \
  --secret-id prod/api/db \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:222222222222:function:rotate-db \
  --rotation-rules 'AutomaticallyAfterDays=30'

output

{
    "ARN": "arn:aws:secretsmanager:eu-west-1:222222222222:secret:prod/api/db-a1B2c3",
    "Name": "prod/api/db",
    "VersionId": "b8f2e1a0-3c4d-4e5f-9a0b-1c2d3e4f5a6b"
}

Under the hood the swap is coordinated with staging labels, which are movable stickers on versions. The live version wears AWSCURRENT. When rotation starts, the function creates the new credential and labels it AWSPENDING, tests it against the real database, and only then moves AWSCURRENT onto the new version while the old one drops to AWSPREVIOUS. That final move is atomic (all-or-nothing), so a consumer asking for the current value never catches a half-rotated state. AutomaticallyAfterDays accepts 1 to 1000 days; swap it for a cron() ScheduleExpression (a fixed timetable, say 3am on the first Sunday of the month) when rotations must land inside a maintenance window instead of at a random hour. You verify that a rotation really happened by reading the version stages.

terminal

aws secretsmanager list-secret-version-ids --secret-id prod/api/db

output

{
    "Versions": [
        {
            "VersionId": "b8f2e1a0-3c4d-4e5f-9a0b-1c2d3e4f5a6b",
            "VersionStages": ["AWSCURRENT"],
            "CreatedDate": "2026-07-22T14:03:11.482000+01:00"
        },
        {
            "VersionId": "5d7c9e02-1a2b-4c3d-8e9f-0a1b2c3d4e5f",
            "VersionStages": ["AWSPREVIOUS"],
            "CreatedDate": "2026-06-22T14:03:09.210000+01:00"
        }
    ],
    "ARN": "arn:aws:secretsmanager:eu-west-1:222222222222:secret:prod/api/db-a1B2c3",
    "Name": "prod/api/db"
}

Two versions, one month apart, one current and one previous: that is proof the flip happened. Google Cloud Secret Manager does far less under the same word. A rotation period and a next-rotation time only make it publish a message to a Pub/Sub topic (Pub/Sub is Google's publish-and-subscribe message bus) when the clock strikes. It never generates new material. You subscribe to that topic and run the rotation yourself. One wiring step has to come first, though: the Secret Manager service agent (the Google-managed account it acts as) can only publish once you grant it the right role on the topic.

terminal

# Grant the Secret Manager service agent Pub/Sub Publisher on the topic FIRST.
# Skip this and Secret Manager cannot publish, so no rotation ping ever fires.
gcloud pubsub topics add-iam-policy-binding secret-rotation \
  --member="serviceAccount:service-482913005417@gcp-sa-secretmanager.iam.gserviceaccount.com" \
  --role=roles/pubsub.publisher

output

Updated IAM policy for topic [secret-rotation].
bindings:
- members:
  - serviceAccount:service-482913005417@gcp-sa-secretmanager.iam.gserviceaccount.com
  role: roles/pubsub.publisher
etag: BwYX7pM3nQ0=
version: 1

terminal

gcloud secrets create prod-api-db \
  --replication-policy=automatic \
  --rotation-period=2592000s \
  --next-rotation-time=2026-08-13T00:00:00Z \
  --topics=projects/acme-prod/topics/secret-rotation

output

Created secret [prod-api-db].

Two details bite here. The rotation period is written in seconds with an s suffix (2592000s is 30 days) and cannot drop below 3600s, one hour. And the ping only leaves the building because you gave that service agent (service-PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com) the Pub/Sub Publisher role a moment ago; without that grant, Secret Manager cannot publish and the rotation clock ticks in silence. The ping is the alarm. The code that mints new material and adds a version is yours to write, and yours to test.

Azure Key Vault splits the job by object type, and this trips people. Cryptographic keys rotate their own material under a rotation policy you attach. Secrets, database passwords and the like, have no built-in generator at all. For those you wire a near-expiry event through Event Grid (Azure's event-routing service) to a function that mints and writes the next version. So a key and a password living in the same vault follow completely different rotation stories.

rotation-policy.json

{
  "lifetimeActions": [
    {
      "trigger": { "timeAfterCreate": "P30D" },
      "action":  { "type": "Rotate" }
    }
  ],
  "attributes": { "expiryTime": "P90D" }
}

terminal

# KEYS rotate themselves under a policy (P30D = rotate 30 days after creation).
az keyvault key rotation-policy update \
  --vault-name acme-prod-kv --name db-cmek \
  --value @rotation-policy.json

output

{
  "attributes": {
    "created": "2026-07-22T13:05:44+00:00",
    "expiryTime": "P90D",
    "updated": "2026-07-22T13:05:44+00:00"
  },
  "id": "https://acme-prod-kv.vault.azure.net/keys/db-cmek/rotationpolicy",
  "lifetimeActions": [
    {
      "action": { "type": "Rotate" },
      "trigger": { "timeAfterCreate": "P30D" }
    }
  ]
}

terminal

# SECRETS only get an expiry stamp; near-expiry fires an Event Grid event
# that a Function handles to write the new version.
az keyvault secret set \
  --vault-name acme-prod-kv --name prod-api-db \
  --value "$NEW_DB_PASSWORD" \
  --expires 2026-08-13T00:00:00Z \
  --query id -o tsv

output

https://acme-prod-kv.vault.azure.net/secrets/prod-api-db/7d1e9c0b4a2f4c8e

That db-cmek key is not incidental. Every one of these managers encrypts your secrets at rest with a KMS key, and you can bring your own instead of the provider default: --kms-key-id on an AWS secret, --kms-key-name on a GCP secret with user-managed replication, a key like db-cmek behind an Azure vault. That customer-managed key (CMEK, customer-managed encryption key; its stricter cousin BYOK, bring your own key, goes one step further and imports key material you generated yourself) runs on its own rotation clock, which is exactly the Azure policy above. Rotating a secret and rotating the key that protects it are two separate schedules, and an auditor will ask about both.

'Enabled' is not 'rotating', and 'rotated' is not 'refreshed'

Two silent failures stack here. First, switching on rotation for a Google Cloud secret or an Azure secret only schedules a notification. If you never attach the Pub/Sub subscriber or the Event Grid function that mints new material, the value never changes while the console cheerfully reports rotation 'enabled'. Second, even a genuine rotation is an outage if consumers do not re-read: the instant the old credential is retired, anything still holding it in memory breaks. Keep old and new valid together for longer than your longest cache or lease, confirm every consumer has picked up the new version inside that overlap, and only then retire the old one.

When a credential leaks

Scheduled rotation is hygiene. A leak is an emergency, and the move that saves you is the same on every cloud: cut the grant, do not only change the value. If you rotate the password but leave the leaked IAM binding standing, the attacker can read the new one too. So you revoke access and rotate together.

terminal

# AWS: force an off-cycle rotation now, so the leaked version stops being current.
aws secretsmanager rotate-secret --secret-id prod/api/db --rotate-immediately

output

{
    "ARN": "arn:aws:secretsmanager:eu-west-1:222222222222:secret:prod/api/db-a1B2c3",
    "Name": "prod/api/db",
    "VersionId": "c9d0e1f2-3a4b-5c6d-7e8f-9a0b1c2d3e4f"
}

terminal

# GCP: destroy the exposed version, then pull the IAM grant that could read it.
gcloud secrets versions destroy 4 --secret=prod-api-db --quiet
gcloud secrets remove-iam-policy-binding prod-api-db \
  --member=serviceAccount:[email protected] \
  --role=roles/secretmanager.secretAccessor

output

Destroyed version [4] of the secret [prod-api-db].
Updated IAM policy for secret [prod-api-db].
etag: BwYX4n1a9Qk=
version: 1

terminal

# Azure: disable the exposed version, then rotate the identity behind it.
az keyvault secret set-attributes \
  --vault-name acme-prod-kv --name prod-api-db --enabled false \
  --query "attributes.enabled" -o tsv

output

false

Dynamic secrets (credentials the system mints on the spot for one short lease, never stored) shrink that whole procedure to a single act. If the credential was minted for one lease and lives for an hour, you revoke the lease and the backend account dies in seconds, with nothing scattered to hunt down. That is the strongest argument for issuing short-lived credentials instead of storing long-lived ones, and it leads straight to the last decision.

One control plane, or one per cloud

You can run the native manager in each cloud, or one platform across all of them. Native managers give the deepest IAM integration and the least to operate, and in exchange you keep three rotation stories, three audit formats, and three access models. That is real toil once you hold hundreds of secrets. A central platform like HashiCorp Vault gives you one control plane and, more usefully, dynamic secrets: instead of storing a standing database password, Vault mints a fresh short-lived one per request and revokes it when the lease ends, so there is often nothing to rotate at all.

terminal

# Vault issues a brand-new DB credential on demand, valid 1h, then revoked.
# Nothing long-lived is stored, so there is nothing to "rotate".
vault read database/creds/api-ro

output

Key                Value
---                -----
lease_id           database/creds/api-ro/2c9f4b1a-7d3e-4a6b-9c8d-1e0f2a3b4c5d
lease_duration     1h
lease_renewable    true
password           A1a-9f3c8e21b7a4d5
username           v-approle-api-ro-x7Qm9a2b

The price is that Vault becomes tier-0 infrastructure (the most critical tier, the thing every other system leans on) with its own headaches: high availability, disaster recovery, and unsealing (Vault boots locked and needs a quorum of key shares before it will hand out anything). When Vault is down, nothing gets a credential. The topology that scales keeps central issuance but hands distribution to the cloud: Vault or your pipeline stays the source of truth, and the External Secrets Operator (ESO, a Kubernetes controller that copies external secrets into the cluster) syncs each value into the cluster's own store, so application code keeps reading a plain Kubernetes Secret while the truth stays central.

external-secret.yaml

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: prod-api-db
  namespace: api
spec:
  refreshInterval: 1h                 # re-reads the source every hour
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: prod-api-db                 # the Kubernetes Secret ESO writes
  data:
    - secretKey: password
      remoteRef:
        key: prod/api/db              # path in the central store
        property: password

terminal

kubectl get externalsecret prod-api-db -n api

output

NAME          STORE           REFRESH INTERVAL   STATUS         READY
prod-api-db   vault-backend   1h                 SecretSynced   True

That refreshInterval doubles as your rotation-safety valve: it sets how often the fleet re-reads, so keep it shorter than the overlap window you hold during a rotation. Cost pushes the same direction. Native managers bill per secret and per block of API calls (AWS Secrets Manager, for example, runs about $0.40 per secret each month plus $0.05 per 10,000 calls, and Google Cloud and Azure price on a comparable per-item, per-operation basis). A chatty service that fetches on every request instead of caching inside its refresh window turns a rounding error into a line item, and buries the audit log in noise.

A rotation that does not cause an outage

1Mint the new credential

create it on the backend; leave the old one working

2Test it

authenticate with the new credential before promoting it

3Overlap: both valid

old and new work together, longer than the longest cache or lease

4Flip current to new

atomic label move, so readers see one clean value

5Confirm consumers refreshed

verify every reader picked up the new version

6Retire the old

revoke the old version and its IAM grant

Hold the same invariants on every cloud, whichever managers you run: short lifetimes, one secret's scope per workload, workload identity ahead of any bootstrap key, and an audit line on every read. Then prove each rotation instead of trusting it. Read the version stages or the audit log afterward, confirm the live credential is the new one, and confirm the fleet is actually using it. A rotation you cannot show ran is one you should assume did not.

Try this

Run gcloud secrets versions access latest --secret=prod-api-db on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: 'Enabled' is not 'rotating', and 'rotated' is not 'refreshed'. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

BYOK/HYOK, HSM & key ceremoniesCentralized audit logging &…

IAM mental modelAWS security for DevOps engineers The IAM evaluation modelAWS security engineering Resource hierarchy & org policyGCP security Entra ID, Conditional Access & PIMAzure security

Secrets managers & rotation across clouds

Reading a secret without shipping one

Rotation means three different things

When a credential leaks

One control plane, or one per cloud

Try this

Takeaway

Related