>_SecOpsLog_
CoursesAdvanced secrets managementKMS, HSM & key hierarchies

KMS, HSM & key hierarchies

Envelope encryption, seal-wrap, PKCS#11, FIPS, and key ceremonies.

Expert35 min · lesson 12 of 15

An auditor asks one question that ends most secrets conversations: where is the master key, and who can use it? Every secrets manager, cloud or self-hosted, sits on top of a KMS (Key Management Service, a service whose only job is to hold keys and run encrypt and decrypt on your behalf). At this level you have to reason about that layer yourself. What does the KMS actually protect? Why does an HSM (Hardware Security Module, a tamper-resistant box that never lets a key leave the hardware it was born in) belong at the top? And how does a key hierarchy shrink the damage when something leaks? This is the layer regulators care about. It is the gap between saying "we encrypt secrets" and being able to prove it.

In plain terms
A key hierarchy works like a hotel. The manager's master key does not open guest rooms. It opens a cabinet of floor keys, and each floor key opens the rooms on one floor. Steal a room key and you get one room. Re-key a floor and the master never comes out of the safe.

Why nobody points the top key at real data

Systems that survive audits never encrypt data with the top key. They build a chain. A root key lives in an HSM and wraps key-encryption keys, or KEKs. Each KEK wraps a set of data-encryption keys, or DEKs. Only the DEKs ever touch your actual bytes. Every link in the chain gives you a smaller thing to rotate. Rotate a DEK and you re-protect one dataset. Rotate a KEK and every DEK beneath it gets re-wrapped, with no data re-encrypted. The root stays where it is, offline and effectively immortal.

That is the envelope pattern the cloud secrets managers use, drawn out one level further. The higher a key sits, the fewer operations it performs and the harder you guard it. Vault's barrier key, an AWS customer managed key (a KMS key you create and control, called a customer master key or CMK in docs written before 2021), and a GCP Cloud KMS key each sit on a different rung, depending on what they wrap.

A key hierarchy
root (HSM)
never exported
signs/wraps only
rarely used
ceremony to rotate
KEKs
wrap DEKs
per app / tenant
rotate to re-wrap
no data re-encrypt
DEKs
encrypt the data
per object/dataset
cheap to rotate
narrow blast radius
Lose a DEK, lose one dataset. The root does almost nothing, which is why it is almost never exposed.

Hardware that refuses to hand the key back

A Hardware Security Module is a sealed box that makes keys, uses them, and will not give them back. You send it data and ask it to sign or unwrap. It returns the answer, and the private key stays inside. Software talks to these boxes through PKCS#11 (Public-Key Cryptography Standard number 11, the common programming interface for hardware crypto devices), or through a cloud HSM or KMS API. In Vault, seal wrap uses that hardware to put one more layer of encryption around the stored values you care about most. Both seal wrap and the PKCS#11 seal are Vault Enterprise features, so Community Edition will refuse the configuration below.

An HSM turns the sentence "an admin could copy the key" into "the key physically cannot leave." Regulated environments want that second sentence in writing. Cloud KMS is not one boundary, though. Every AWS KMS key is generated and used inside a FIPS-validated HSM already, so a CloudHSM cluster buys you single tenancy and hardware you can point at in an audit, not protection you were missing. GCP Cloud KMS and Azure Key Vault are the services that make you pick a software or an HSM protection level per key. On-prem modules validated to FIPS 140-2 or 140-3 (a US government standard for cryptographic modules) satisfy auditors who want hardware attestation rather than an API promise.

vault.hcl — PKCS#11 (HSM) seal
seal "pkcs11" {
  lib            = "/usr/lib/softhsm/libsofthsm2.so"
  slot           = "0"
  pin            = "env://HSM_PIN"
  key_label      = "vault-hsm-key"
  hmac_key_label = "vault-hsm-hmac"
}
# master key wrapped by a key that never leaves the HSM.
# FIPS 140-2/3 validated HSMs satisfy "keys in hardware" requirements.
# Enterprise only: Community Edition exits with
# "Seal type pkcs11 requires Vault Enterprise".

Bringing your own key, and the ceremony around it

BYOK (Bring Your Own Key) means you generate the key material yourself and import it into the cloud KMS or HSM. You can then say exactly how it was made, and under some models you keep the ability to withdraw it later, which is a control many regulated customers insist on. It costs you automatic rotation: AWS will not rotate a key on a schedule if you imported the material, or if the key lives in a CloudHSM custom key store. Rotating those means creating a new key, moving the alias to it and re-wrapping what the old key held, on a date somebody has to put in a calendar. A key ceremony is the recorded, multi-person process for creating and backing up a root key: split custody, named witnesses, a script read out loud, tamper-evident bags.

At the top of the hierarchy, the process around the key carries as much weight as the key. You are protecting the thing that protects everything else. BYOK with no ceremony record fails an audit the same way an auto-generated key with sloppy access control fails a breach.

terminal
aws kms create-key --description "prod-secrets-root" --key-spec SYMMETRIC_DEFAULT \
  --tags TagKey=Purpose,TagValue=SecretsManager
aws kms create-alias --alias-name alias/prod-secrets-root --target-key-id abcd-1234-5678
# rotation, policy and grant calls take a key ID or ARN, never an alias
aws kms enable-key-rotation --key-id abcd-1234-5678
aws kms get-key-rotation-status --key-id abcd-1234-5678
output
{
  "KeyMetadata": {
    "KeyId": "abcd-1234-5678",
    "KeyState": "Enabled"
  }
}
 
# create-alias and enable-key-rotation both answer with an empty body.
# rotation is a separate call, not a KeyMetadata field:
aws kms get-key-rotation-status --key-id abcd-1234-5678
# { "KeyRotationEnabled": true }

The two things that actually break: deletion and IAM

AES-256 will not be the thing that fails. The key's access policy and its lifecycle will. KMS has no DeletionProtection flag to tick, so your protection is a long pending-deletion window on schedule-key-deletion, plus treating DisableKey and ScheduleKeyDeletion as high-severity events the moment either shows up. Delete a KMS key and every secret it wrapped becomes unrecoverable, quietly, with no error until something tries to read. Scope kms:Decrypt to named roles. Alarm on key-policy changes and on scheduled deletions.

Treat emergency key disable as a runbook you have actually run, not a paragraph somebody wrote in a wiki. And never park backups of wrapped data next to the keys that unwrap them. The scenario that ruins a quarter is a snapshot tarball sitting in the same S3 bucket as the KMS key policy JSON.

Emergency key revocation
1compromise detected
key policy or usage anomaly
2disable the key
kms:DisableKey, effective at once
3secrets unreadable
all wrapped data keys fail decrypt
4restore from backup
after new key + re-wrap plan
Disabling the key is a nuclear option. Rehearse the re-wrap path before you need it live.

Which key wraps what

Vault auto-unseal, Secrets Manager keys, Kubernetes etcd encryption and SPIRE signing keys can all sit on KMS or an HSM. They should not all sit on the same one. Give each trust domain its own KMS key, so that a compromise or a revocation in one layer does not brick three unrelated systems. Write down which key wraps what, and who holds kms:Decrypt on each.

An auditor will ask for the key inventory: key ID, purpose, rotation schedule, custodians, date of the last ceremony. If you cannot produce that table in an afternoon, you do not have a key hierarchy. You have a pile of keys.

terminal
aws kms list-aliases --query "Aliases[?contains(AliasName, 'prod')].{Alias:AliasName,KeyId:TargetKeyId}"
aws kms describe-key --key-id alias/prod-secrets-root --query "KeyMetadata.KeyState"
aws kms get-key-rotation-status --key-id abcd-1234
output
[
  { "Alias": "alias/prod-secrets-root", "KeyId": "abcd-1234" },
  { "Alias": "alias/prod-vault-unseal", "KeyId": "efgh-5678" }
]
 
"Enabled"
 
{
    "KeyRotationEnabled": true
}
# describe-key resolves an alias for you. get-key-rotation-status will not,
# which is why the third call reuses the key ID from the first.
# State and rotation come from two different calls, and neither one
# returns a field named Rotation.
terminal
vault status | grep -E "Sealed|Seal Type"
# verify seal type matches your KMS/HSM design
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DisableKey --max-results 5
output
Seal Type       awskms
Sealed          false
 
# no recent DisableKey events. Alarm if this list is non-empty in prod.

FIPS labels, and what they do not cover

Regulated workloads often demand modules validated to FIPS 140-2 or 140-3: AWS CloudHSM, GCP HSM-backed keys, Azure Managed HSM, or on-prem boxes from Luna or Thales. Read the label carefully. It covers the hardware boundary, not your application code. You still have to scope IAM, rotate keys and run the ceremony properly, or the FIPS certificate is wallpaper.

Vault seal-wrap backed by cloud KMS answers most compliance questionnaires about master key protection. A PKCS#11 seal answers the stricter readings, the ones that want proof the key cannot be exported from hardware at all. Pick the tier that matches your regulator's definition of a customer-managed cryptographic key, then write down why you picked it.

terminal
aws kms create-custom-key-store --custom-key-store-name prod-hsm-store \
  --cloud-hsm-cluster-id cluster-abc \
  --trust-anchor-certificate file://customerCA.crt \
  --key-store-password "$KMSUSER_PASSWORD"
aws kms connect-custom-key-store --custom-key-store-id cks-1234
aws kms describe-custom-key-stores --custom-key-store-id cks-1234 \
  --query "CustomKeyStores[0].ConnectionState"
aws kms create-key --origin AWS_CLOUDHSM --custom-key-store-id cks-1234 \
  --description "FIPS-backed prod root"
output
{
    "CustomKeyStoreId": "cks-1234"
}
# that ID is the whole response. A new store is DISCONNECTED, and
# connect-custom-key-store answers with an empty body, so poll
# describe-custom-key-stores until it stops reading CONNECTING:
 
"CONNECTED"
 
{
  "KeyMetadata": {
    "KeyId": "fips-7890",
    "CustomKeyStoreId": "cks-1234",
    "Origin": "AWS_CLOUDHSM"
  }
}

Split key administrators from secret administrators. In AWS, kms:CreateKey does not grant secretsmanager:PutSecretValue, yet the same three people usually hold both anyway. Splitting the roles is what stops one insider from reading a secret and then re-keying the KMS key to bury the evidence.

Ceremony documentation should cover the bad days too: the HSM is unreachable, a quorum holder is on a plane, the cloud KMS region is down. The ceremony describes how keys are born, and also how you get moving again when the hardware or the API says no in the middle of an incident.

Auditors now ask for evidence that software cannot export the root key material. A screenshot of HSM key attributes showing non-exportable, or a KMS key showing Origin=AWS_CLOUDHSM, carries more weight than any architecture slide. File those with the ceremony record.

Rehearse a key disable in a sandbox account once a quarter. A team that has never watched every decrypt in the cluster fail at the same instant will keep treating key policy edits like ordinary config changes.

terminal
aws kms list-grants --key-id abcd-1234
aws kms describe-key --key-id alias/prod-secrets-root --query "KeyMetadata.{State:KeyState,DeletionDate:DeletionDate}"
output
{
  "Grants": []
}
 
{
  "State": "Enabled",
  "DeletionDate": null
}
# no stray grants, key Enabled and not pending deletion: baseline KMS hygiene

KMS and HSM backings decide your answer when a regulator asks where keys live and who administers them. Cloud KMS buys you managed durability and IAM policy. An HSM buys you stronger physical and logical guarantees at a higher operational cost. Vault's seal key and its Transit keys can sit on either, and the design question is identical in both cases: who can call decrypt on the seal key, and who can schedule it for deletion?

Keep the duties apart. The team that runs Vault nodes should not be able to delete the seal key on its own, and the team that owns KMS should not hold Vault root. Dual control on destructive KMS actions is boring, and it is the control that saves you.

Label every key by purpose: seal, transit, pki intermediate. Then alarm on any use that does not match the label. A seal key decrypt at 3 a.m. from a role nobody recognises is an incident until someone proves otherwise.

Try this

In your lab, read the seal type, then read the key policy sitting behind it. You are hunting for one fact: which IAM principal is allowed to decrypt the seal wrapper.

terminal
vault status | egrep 'Seal|Recovery'
# get-key-rotation-status and get-key-policy reject an alias, so resolve it first
SEAL_KEY=$(aws kms describe-key --key-id alias/vault-seal --query KeyMetadata.KeyId --output text)
aws kms get-key-rotation-status --key-id "$SEAL_KEY"
aws kms get-key-policy --key-id "$SEAL_KEY" --policy-name default --query Policy --output text | head -40
output
Seal Type                awskms
Recovery Seal Type       shamir
Sealed                   false
 
{
    "KeyRotationEnabled": true
}
 
{
  "Sid": "Allow Vault role to decrypt seal",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::123456789012:role/vault-server"},
  "Action": ["kms:Decrypt","kms:DescribeKey"],
  "Resource": "*"
}
# no human admin roles in the decrypt statement, which is what you want

Takeaway

KMS and HSM choices move your trust out of the algorithm and into key policy and hardware. Guard the seal key and the Transit keys with named principals, turn on rotation wherever the service supports it, and alarm on both decrypt and delete.

Next: open your seal key policy and look for a human role or a wildcard in the decrypt statement, switch rotation on if it is off, and add a CloudTrail alert for ScheduleKeyDeletion on every key in your inventory.

The algorithm is fine. Deletion and access are what bite
The waiting period is the only undo you get. schedule-key-deletion takes a pending window of 7 to 30 days and defaults to 30, so refuse the ticket that asks you to shorten it for a cleanup sprint. Then pair that window with dual control: nobody who can call kms:Decrypt on a key should also be able to schedule that same key for deletion on their own.
Quick check
01You want to show an auditor that alias/prod-secrets-root rotates. You run aws kms describe-key --key-id alias/prod-secrets-root --query "KeyMetadata.KeyState" and it prints "Enabled". What have you actually shown?
Incorrect — Disabled means somebody turned the key off for use. Rotation has nothing to do with that state.
Correct — State and rotation come back from two different APIs, and that second call is the one that prints KeyRotationEnabled.
Incorrect — There is no Rotation field under KeyMetadata, so that query comes back empty and you learn nothing.
Incorrect — list-aliases maps alias names to key IDs. It reports neither key state nor rotation.
02A regulator wants written proof that no software path can export your master key material. vault status reports Seal Type awskms, and the seal key policy already names only the vault-server role. Which change actually answers that requirement?
Incorrect — Rotation says the key changes on a schedule. It says nothing about where the key material can travel.
Incorrect — Seal wrap adds a layer on top of whatever seal you already run. Backed by awskms there is still no hardware boundary to point at.
Incorrect — That proves who may call decrypt, which is a different control. The question here is where the key lives, not who uses it.
Correct — Both put the key somewhere it cannot leave, and the non-exportable attribute or Origin=AWS_CLOUDHSM is the evidence you file with the ceremony record.
03Your key inventory shows one KMS key wrapping the Vault seal, the Secrets Manager secrets and the etcd encryption key. Someone proposes revoking that key to contain a suspected compromise in one of those layers. What do you raise?
Correct — Shared key means shared fate. One KMS key per trust domain is what stops a revocation in one layer bricking two unrelated systems.
Incorrect — Rotating a wrapping key re-wraps the keys beneath it. None of the data underneath gets re-encrypted.
Incorrect — AES-256 is not the part that fails. Access policy and key lifecycle are what bite you.
Incorrect — Named principals limit who can read. They do nothing once the key itself is disabled or scheduled for deletion.
Cloud secret managers & envelope…Zero-downtime rotation & the…

Related

What counts as a secretSecrets management foundationsWhy production Vault existsVault from dev to production