Cloud secret managers & envelope encryption
AWS/GCP/Azure managers, rotation lambdas, and CMK/BYOK.
Someone in finance asks why you are paying for Vault when AWS already ships Secrets Manager. It is a fair question. If every workload runs in one cloud, and IAM (identity and access management, the cloud's own system for deciding who may do what) already proves which task is asking, the cloud's own store may be all you need. The wrong answer is copying the same production password into three managers and calling it redundancy. AWS Secrets Manager, GCP Secret Manager and Azure Key Vault are the sensible default on a single cloud. They look different in the console. Underneath, all three do the same thing: your secret is locked with a data key, and that data key is locked with a master key held in the cloud's key service. Learn that one pattern and you know both how to use these products well and exactly what you are trusting the provider with.
The shared trick underneath: envelope encryption
Here is the mechanism. The manager generates a fresh random key just for your secret, encrypts the secret with it, then hands that key to KMS (key management service, the cloud's key vault) to be wrapped by a master key that never leaves the service's tamper-resistant hardware. Your secret ends up inside two wrappers, like a letter sealed in an envelope and then locked in a safe you cannot open on your own. That is envelope encryption: bulk data encrypted by a local data key, the data key protected by a central master key. Reading it back needs two separate permissions. You need read access on the secret, and you need kms:Decrypt on the key that wrapped it. The key policy is a second gate, with its own door and its own lock.
The default key the cloud hands you for free is the hotel's own master key. A customer-managed key is one you create, name and control, and that control is the entire point. You set the rotation schedule. You decide whether another account may use it. You can cut off every secret it ever wrapped by disabling one key. Amazon still runs the HSM (hardware security module, the sealed box that performs the crypto and never lets the key out). You hold the IAM policy and the key policy. That is the gap between a convenient product and one you can answer for in an audit.
AWS Secrets Manager: the key, the rotation, and who can read
Secrets Manager keeps every version of a secret and will rotate it for you by running a Lambda function (a small piece of code AWS runs on demand, with no server for you to patch). A read needs both secretsmanager:GetSecretValue and kms:Decrypt on the customer-managed key. The secret's resource policy, a rule attached to the secret itself rather than to the caller, can insist on TLS (transport layer security, the encryption that makes a connection private) and refuse anything that did not arrive through your VPC endpoint, a private door into the AWS API that never touches the public internet. Two labels carry the rotation story: AWSCURRENT and AWSPREVIOUS. They give you the two-password handover for free.
Tag secrets by application and environment so your IAM conditions can stay narrow. A payments role should read prod/payments/* and nothing else. Handing it secretsmanager:GetSecretValue on * is the same as giving the night porter every room key because cutting one was slower. Sharing into another account takes two yeses: the secret's resource policy has to name the foreign role, and so does the KMS key policy. People forget the key policy constantly. It is the single most common reason a cross-account read fails.
aws secretsmanager create-secret --name prod/db \--kms-key-id alias/prod-secrets \--secret-string "$(openssl rand -base64 24)"aws secretsmanager describe-secret --secret-id prod/db
{"ARN": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/db-Ab12Cd","Name": "prod/db","KmsKeyId": "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"}"RotationEnabled": false,"VersionIdsToStages": { "v1": ["AWSCURRENT"] }
aws secretsmanager rotate-secret --secret-id prod/db \--rotation-lambda-arn arn:aws:lambda:us-east-1:111122223333:function:rotate-db \--rotation-rules AutomaticallyAfterDays=30aws secretsmanager describe-secret --secret-id prod/db --query RotationEnabled
{"ARN": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/db-Ab12Cd","RotationEnabled": true}true
Parameter Store, or the real thing?
Not every value needs the expensive box. AWS Parameter Store with the SecureString type is cheaper and perfectly good for configuration that happens to contain a couple of sensitive strings. It is part of Systems Manager, which is why the commands start with aws ssm, and it has no built-in rotation. Secrets Manager adds rotation, copies to other regions, and resource policies, which is what real credentials need. GCP Secret Manager and Azure Key Vault split along the same lines: versions, an access gate (IAM on Google, RBAC or role-based access control on Azure), and the option of a key you own backed by hardware.
The rule that keeps this easy: anything that has to rotate, or that would hurt badly if it leaked, belongs in Secrets Manager or in Vault. Slow-moving configuration can live in Parameter Store. Do not move a database password to Parameter Store because it costs less per read. You are not paying for storage, you are paying for rotation and version history, and those are exactly the parts you would be giving up.
aws ssm put-parameter --name /prod/app/feature-flag --value "enabled" --type SecureString --key-id alias/prod-configaws ssm get-parameter --name /prod/app/feature-flag --with-decryption
{"Version": 1,"Tier": "Standard"}{"Parameter": { "Name": "/prod/app/feature-flag", "Value": "enabled", "Type": "SecureString" }}
Rotation, and the window where both passwords work
Managed rotation creates a new credential, tests it, then moves the AWSCURRENT label onto it while AWSPREVIOUS keeps pointing at the old one. Two passwords work at once, which is the handover you would otherwise have to build by hand. The sharp edge is how long that overlap lasts. Applications cache secrets in memory. Kill the old version before the last cached copy expires and you have taken an outage on purpose. Size the overlap to your longest cache TTL (time to live, how long a copy stays valid before the app fetches a fresh one) plus the time a full deploy takes.
The rotation Lambda has one step it must never skip. It has to change the password in the real backing store (RDS, Amazon's managed relational database service, or Redshift, or whatever sits behind the secret) and then open a connection with the new password before it promotes AWSCURRENT. A rotation that updates Secrets Manager and never touches the database leaves you with two versions of the truth: the password AWS hands out, and the password the database actually accepts. Everything that refreshes after that moment breaks.
Narrow permissions, and looking after the key
Defaults are not the secure setting anywhere in this stack. Scope each role to its own secret prefixes rather than secretsmanager:GetSecretValue on *. Require aws:SecureTransport in the resource policy so a plaintext call is refused rather than merely logged. KMS has no deletion-protection toggle the way RDS does, so your protection is procedural: keep a long waiting period on scheduled key deletion, and let very few people call ScheduleKeyDeletion or DisableKey. Alarm on key policy edits and on DisableKey events. Turning off one key silently bricks every secret it ever wrapped, and nothing in the secrets console will explain why the reads stopped.
Put CMK key policies on the same quarterly review calendar as your Vault policies. Two gates guard every secret: the IAM policy on the role, and the policy on the key. Tighten one and leave the other sitting at *, and you have spent the effort without moving the actual risk.
aws secretsmanager get-secret-value --secret-id prod/db --version-stage AWSCURRENTaws secretsmanager get-secret-value --secret-id prod/db --version-stage AWSPREVIOUS
{"SecretString":"{\"username\":\"app\",\"password\":\"new-random-value\"}"}{"SecretString":"{\"username\":\"app\",\"password\":\"previous-still-valid\"}"}# both stages valid during overlap — consumers pick up new on refresh
Reading across accounts, copying across regions
A secret shared into another AWS account needs both policies to name that foreign principal: the one on the secret and the one on the KMS key. Miss the key policy and GetSecretValue returns AccessDeniedException while the secret policy looks completely correct in the console. Teams lose days to that symptom. Write the two-policy rule into your platform runbook so the next person recognises it in a minute.
Cross-region replication copies the ciphertext, not the key. The KMS key in the replica region has to be able to decrypt what arrives, or you replicate using the same key ARN (Amazon resource name, the long unique identifier for an AWS object) where the service allows it. Add a read of AWSCURRENT from the replica region to your failover drill, the same way you rehearse promoting a Vault DR (disaster recovery) cluster. A replica nobody has ever read from is a guess, not a backup.
aws secretsmanager put-resource-policy --secret-id prod/db --resource-policy file://secret-policy.jsonaws sts get-caller-identity
{"ARN": "arn:aws:secretsmanager:us-east-1:111122223333:secret:prod/db-Ab12Cd"}{"Account": "111122223333","Arn": "arn:aws:sts::111122223333:assumed-role/platform-admin/session"}# verify foreign roles match resource policy principals
The rotation Lambda runs under its own IAM role, and that role needs Secrets Manager permissions, KMS permissions, and whatever the backing service requires to change a password. That combination is an administrator in disguise. Scope it to one secret ARN and one database instance. A rotation function that can rewrite every credential in the account is a back door that sails through code review because it is labelled maintenance.
GCP Secret Manager and Azure Key Vault have different API shapes and an identical trust model: versioned secrets, an access gate, an optional key you own (CMEK, customer-managed encryption keys, on Google; HSM-backed keys on Azure), and audit logs flowing to Cloud Logging or Azure Monitor. Platforms that span clouds usually hide all three behind External Secrets Operator or the Secrets Store CSI driver (container storage interface, the standard way Kubernetes mounts outside storage into a pod). Make sure your abstraction still tells you which cloud gate said no when a read fails.
Cost and security meet at the API call. An app that fetches its secret on every single request pays for every fetch, and hands a stolen role a much wider harvest. Cache at the workload with an explicit TTL that lines up with your rotation overlap. One setting lowers the bill and shrinks what a compromised role can quietly collect.
Turn on CloudTrail data events for Secrets Manager in production accounts. Management events record that a secret was created or rotated. Only data events record who read it and when, and that is the log you will want when someone asks what the attacker actually walked away with.
Agree a naming prefix per team and enforce it with IAM conditions. Secrets named however each person felt that day cannot be scoped by any pattern, and it always ends the same way: somebody grants GetSecretValue on * at six in the evening to unblock a deploy, and it stays there for years.
Treat the default AWS-managed KMS key as a blocker before a team goes live, not as an audit finding afterwards. Platform teams should have customer-managed keys provisioned before application teams start putting production credentials into Secrets Manager. Re-encrypting later is a migration nobody ever finds time to schedule.
Re-read the secret's resource policy whenever the key policy changes, and the other way round. The two drift apart on their own, and the break stays invisible until a deploy in the other account fails.
aws secretsmanager list-secrets --filters Key=name,Values=prod/ --query "SecretList[].Name"aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::111122223333:role/payments --action-names secretsmanager:GetSecretValue kms:Decrypt
["prod/db","prod/api-key"]EvalDecision: allowedEvalDecision: allowed# simulate before shipping new IAM — catches missing kms:Decrypt early
Cloud secrets managers sit beside Vault rather than against it. AWS Secrets Manager, Parameter Store, GCP Secret Manager and Azure Key Vault each pull hardest inside their own cloud. Reach for the cloud manager when the workload is born there and IAM already answers the secret-zero question, meaning the app can prove who it is without holding a credential first. Reach for Vault when you want one policy language across clouds, dynamic credentials for more than one provider, or Transit and PKI (public key infrastructure, the machinery that issues and trusts certificates) everywhere.
Never copy one production password into three managers for redundancy. That is sprawl with a vendor logo on it, and three places to forget the next time you rotate. Pick a system of record per class of secret and integrate the rest: Vault minting AWS STS (security token service, the AWS service that issues temporary credentials) sessions, or External Secrets Operator reading from Secrets Manager. Not both holding the same static string.
Try this
Do a normal IAM-authenticated read against a cloud secret, then mint a credential from the Vault AWS secrets engine, and watch which of the two leaves a durable key behind. The short lease does not remove the trust you place in whoever issues credentials. It shrinks the window in which a stolen one is worth anything.
aws secretsmanager get-secret-value --secret-id prod/payments/db --query CreatedDateaws sts get-caller-identityvault read aws/creds/payments-readonlyaws sts get-caller-identity # with Vault-minted keys in env
2024-11-02T14:22:01ZAccount: 123456789012Arn: arn:aws:sts::123456789012:assumed-role/payments-task/...Key Value--- -----access_key AKIA...secret_key ...security_token <none> # or session_token for STS-stylelease_duration 15mArn: arn:aws:iam::123456789012:user/vault-token-payments-...# lease expires -> key stops working without manual delete in many setups
Takeaway
Pick the cloud manager when IAM identity is already your bootstrap, and Vault when you want one control plane across clouds and secret types. Whichever way you go, a credential with a fifteen-minute lease beats a static access key someone pasted into a ticket, because the lease expires whether or not anybody remembers to clean up.
Next: find one static IAM access key an application still uses, delete it, and put an instance role or a Vault-minted credential on a fifteen-minute lease in its place.