CMK, Managed HSM & at-rest
Own the key, the rotation, and the kill switch.
Every hotel room has a safe. You pick the combination, your passport goes in, and you sleep fine. Then you learn the front desk keeps a master override for guests who forget their code. Nothing about the safe is weak. The question is who else can open it.
Azure encrypts every byte at rest by default, at no cost, using platform-managed keys (keys Microsoft creates, stores, and rotates on your behalf). The cryptography is fine. Microsoft holds the override. A customer-managed key, or CMK, replaces that override with a key you create, rotate on your own schedule, and can pull out of the lock entirely. What follows is when that ownership is worth its operational weight, and how to wire it without locking yourself out of your own data.
Envelope Encryption, Or Why Rotation Is Free
A courier depot does not lock ten thousand parcels with one key. Each parcel gets its own cheap padlock, and every padlock key goes into a single strongbox by the door. Lose the strongbox key and nothing opens. Change the strongbox key and not a single parcel is touched.
Azure works the same way, and the pattern has a name: envelope encryption. A data encryption key (DEK) does the byte-level work on your blobs, disks, and database pages. It is a symmetric AES-256 key (AES, the Advanced Encryption Standard, is the fast bulk cipher that locks and unlocks with the same secret), quick enough to run at line rate. That DEK is then encrypted, or wrapped, by a key encryption key (KEK): an RSA key you own in Key Vault or Managed HSM. RSA is a public-key cipher where the key that wraps is not the key that unwraps, so the vault can expose the wrapping half and still guard the half that opens it. HSM there stands for hardware security module, a tamper-resistant appliance built to hold keys and never surrender them in the clear. Only the wrapped DEK sits next to the ciphertext. The KEK never leaves the vault's cryptographic boundary. When a service needs the DEK, it ships the wrapped bytes to the vault and gets the plaintext DEK back.
That single layer of indirection explains everything else in this lesson. Rotating the KEK re-wraps the DEK. One RSA operation on a few hundred bytes, milliseconds, so a petabyte-scale account rotates without rewriting anything. Disabling the KEK means the wrapped DEK can no longer be opened, so every dependent service starts failing data operations while the ciphertext sits there byte-for-byte intact. And because the KEK is the only piece you hold, your entire control surface (custody, rotation cadence, revocation) collapses down to governing one key. CMK does not buy you stronger cryptography. It buys you governance.
Three Rungs Of Key Custody
Platform-managed keys are the floor: invisible, untouchable, no rotation work, no revoke switch, zero cost. For most workloads that is the right answer, and choosing CMK because it sounds safer is how teams acquire an outage they never needed.
CMK in Key Vault is the middle rung. You own the KEK, so you set the rotation schedule, you can put the key admin identity and the data admin identity into different hands, and you gain a disable switch for incident containment. The Standard tier holds key material in software, validated to FIPS 140-2 Level 1 (FIPS is the US government's yardstick for cryptographic modules; Level 1 is the entry tier). The Premium tier backs keys with hardware security modules in a multi-tenant pool validated to FIPS 140-2 Level 3, where the hardware is shared across Azure customers but key material stays isolated per tenant.
Managed HSM is the top rung: a single-tenant pool of dedicated HSM partitions on FIPS 140-3 Level 3 validated hardware, reserved to you alone. You generate and hold the security domain, the encrypted blob that is the only route to recovering or migrating the pool, which is why Microsoft operators cannot extract your key material. It is the right call when a regulator wants attestable hardware custody, and the wrong call for everything else, because it is always-on infrastructure that bills whether you use it or not.
Prepare The Vault Before You Prepare The Key
Two settings matter before any key exists. RBAC authorization (role-based access control, the Azure-wide model where you grant named roles instead of hand-editing a per-vault list) replaces the older access-policy approach and is what Microsoft now recommends. The second setting is purge protection. Deleting a key does not wipe it on the spot: it drops into a recycle bin, the soft-delete window, where it can still be recovered. Purge protection is a lock on that bin, so nobody empties it early and destroys a key other resources still depend on. Azure Storage refuses a CMK from a vault without purge protection, so treat it as a hard prerequisite rather than a preference.
# Premium tier so key material can live in an HSM; purge protection is mandatory for storage CMK.az keyvault create -g rg-sec -n kv-acme-cmk -l westeurope \--sku premium \--enable-rbac-authorization true \--enable-purge-protection true \--retention-days 90 \--query "{name:name, sku:properties.sku.name, rbac:properties.enableRbacAuthorization, purge:properties.enablePurgeProtection, softDeleteDays:properties.softDeleteRetentionInDays}" -o jsonc# The KEK itself, allowed to perform exactly two operations and nothing else.az keyvault key create --vault-name kv-acme-cmk --name storage-cmk \--kty RSA-HSM --size 3072 \--ops wrapKey unwrapKey \--query "{kid:key.kid, kty:key.kty, ops:key.keyOps}" -o jsonc
{"name": "kv-acme-cmk","purge": true,"rbac": true,"sku": "premium","softDeleteDays": 90}{"kid": "https://kv-acme-cmk.vault.azure.net/keys/storage-cmk/3a7d1e9c04b64f8ab2c5d10e7f39a682","kty": "RSA-HSM","ops": ["wrapKey","unwrapKey"]}
The key type RSA-HSM requires the Premium tier; ask for it in a Standard vault and the create call fails outright. Restricting the operations to wrapKey and unwrapKey is least privilege applied at the key itself. Even an identity holding full crypto rights cannot use this key to sign a token or decrypt an arbitrary payload, because the key refuses those operations regardless of who is asking. Storage accepts RSA keys of 2048, 3072, or 4096 bits.
Wiring A Storage Account To Your Key
The storage account needs a badge that opens exactly one door. Use a user-assigned managed identity, which is a Microsoft Entra ID identity you create as its own standalone resource. Entra ID is the current name for what used to be called Azure Active Directory, or Azure AD; the CLI and the docs both moved, the underlying directory did not. Two reasons to prefer user-assigned over the account's built-in system-assigned identity. It outlives the storage account, so deleting and recreating the account never orphans your role assignment. And it exists before the account does, which is what lets you create an account with CMK switched on from its very first second instead of leaving a window where data lands under a platform key.
The role is Key Vault Crypto Service Encryption User. It grants three things: read the key's public metadata, wrap, unwrap. It cannot decrypt arbitrary ciphertext, cannot sign, cannot export key material, cannot read a single secret. If an attacker steals that identity, they can wrap and unwrap DEKs against that one key and go no further.
MI_ID=$(az identity create -g rg-data -n mi-acmedata-cmk --query id -o tsv)MI_PRINCIPAL=$(az identity show -g rg-data -n mi-acmedata-cmk --query principalId -o tsv)KV_ID=$(az keyvault show -n kv-acme-cmk --query id -o tsv)# Assign by object id with an explicit principal type: avoids a Graph lookup that# fails on identities Entra ID has not finished replicating yet.az role assignment create \--assignee-object-id "$MI_PRINCIPAL" \--assignee-principal-type ServicePrincipal \--role "Key Vault Crypto Service Encryption User" \--scope "$KV_ID"
{"condition": null,"conditionVersion": null,"createdOn": "2026-07-22T09:20:41.882914+00:00","description": null,"id": "/subscriptions/8c1e4f70-2b93-4a11-9d6e-71c0a5f3bd42/resourceGroups/rg-sec/providers/Microsoft.KeyVault/vaults/kv-acme-cmk/providers/Microsoft.Authorization/roleAssignments/6b1f4a2d-8e30-4c77-b0a9-2d5417ce90fb","name": "6b1f4a2d-8e30-4c77-b0a9-2d5417ce90fb","principalId": "8f1c2d3e-6b45-4d0a-92f7-c3e81a75b604","principalType": "ServicePrincipal","roleDefinitionId": "/subscriptions/8c1e4f70-2b93-4a11-9d6e-71c0a5f3bd42/providers/Microsoft.Authorization/roleDefinitions/e147488a-f6f5-4113-8e2d-b22465e65bf6","scope": "/subscriptions/8c1e4f70-2b93-4a11-9d6e-71c0a5f3bd42/resourceGroups/rg-sec/providers/Microsoft.KeyVault/vaults/kv-acme-cmk","type": "Microsoft.Authorization/roleAssignments"}
Without those two assignee flags you get "Cannot find user or service principal in graph database" on roughly one run in five against a brand-new identity, and it reads like a permission problem when it is a timing problem. The role definition GUID e147488a-f6f5-4113-8e2d-b22465e65bf6 is worth recognising in an audit log, because it is the only role that should ever appear on a CMK vault for a service principal.
# Attach the identity AND point the account at the key in one call.# Empty version = follow whatever the newest key version is.az storage account update -g rg-data -n stacmedata01 \--identity-type UserAssigned \--user-identity-id "$MI_ID" \--encryption-key-source Microsoft.Keyvault \--encryption-key-vault https://kv-acme-cmk.vault.azure.net/ \--encryption-key-name storage-cmk \--encryption-key-version "" \--key-vault-user-identity-id "$MI_ID" \-o none# VERIFY: did Storage actually reach the vault and resolve a concrete version?az storage account show -g rg-data -n stacmedata01 --query encryption -o jsonc
{"identity": {"federatedIdentityClientId": null,"userAssignedIdentity": "/subscriptions/8c1e4f70-2b93-4a11-9d6e-71c0a5f3bd42/resourcegroups/rg-data/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mi-acmedata-cmk"},"keySource": "Microsoft.Keyvault","keyVaultProperties": {"currentVersionedKeyExpirationTimestamp": null,"currentVersionedKeyIdentifier": "https://kv-acme-cmk.vault.azure.net/keys/storage-cmk/3a7d1e9c04b64f8ab2c5d10e7f39a682","keyName": "storage-cmk","keyVaultUri": "https://kv-acme-cmk.vault.azure.net/","keyVersion": "","lastKeyRotationTimestamp": "2026-07-22T09:24:06+00:00"},"requireInfrastructureEncryption": false,"services": {"blob": {"enabled": true,"keyType": "Account","lastEnabledTime": "2026-03-02T14:41:55.905314+00:00"},"file": {"enabled": true,"keyType": "Account","lastEnabledTime": "2026-03-02T14:41:55.905314+00:00"}}}
The verification step is the whole point. The field currentVersionedKeyIdentifier is Storage telling you it reached the vault, found the key, and locked onto a specific version. If it comes back null, the wiring failed and the account is already refusing data operations while your deployment pipeline reports success. The empty keyVersion is deliberate, and it is what makes rotation automatic. Pin an explicit version instead and every future rotation becomes a manual chore somebody will forget on a Friday.
One neighbouring control worth knowing: az storage account create takes --require-infrastructure-encryption, which adds a second, platform-managed AES layer underneath your CMK layer so the data is encrypted twice with independent keys. It can only be set at creation. There is no way to retrofit it onto an existing account.
Rotate Without Rewriting A Byte
Rotation belongs in the vault, on a schedule, so nobody ever has to touch the storage account again.
{"lifetimeActions": [{"trigger": { "timeAfterCreate": "P90D" },"action": { "type": "Rotate" }},{"trigger": { "timeBeforeExpiry": "P30D" },"action": { "type": "Notify" }}],"attributes": { "expiryTime": "P120D" }}
# Needs Key Vault Crypto Officer on the key. The encryption-user role cannot do this,# which is the separation you wanted: the service can use the key, only humans rotate it.az keyvault key rotation-policy update --vault-name kv-acme-cmk --name storage-cmk \--value ./rotation-policy.json -o jsonc# Force one rotation now and prove the whole path end to end.az keyvault key rotate --vault-name kv-acme-cmk --name storage-cmk --query key.kid -o tsv
{"attributes": {"created": "2026-07-22T09:31:07+00:00","expiryTime": "P120D","updated": "2026-07-22T09:31:07+00:00"},"id": "https://kv-acme-cmk.vault.azure.net/keys/storage-cmk/rotationpolicy","lifetimeActions": [{"action": { "type": "Rotate" },"trigger": { "timeAfterCreate": "P90D", "timeBeforeExpiry": null }},{"action": { "type": "Notify" },"trigger": { "timeAfterCreate": null, "timeBeforeExpiry": "P30D" }}]}https://kv-acme-cmk.vault.azure.net/keys/storage-cmk/1b70d9c4f0a24e7bb2a1c8e35d64f097
The expiryTime field is mandatory in a rotation policy and has a floor of 28 days. Rotation has to land comfortably ahead of expiry, so 90 days of rotation against 120 days of expiry leaves 30 days of slack: if one rotation fails, you get a month of alarms before the key version actually dies under you. The Notify action raises an Event Grid event 30 days out so a human hears about it.
Storage checks the vault for a new key version about once a day. After a forced rotation, currentVersionedKeyIdentifier advances within roughly 24 hours, not within a second, and that lag is normal rather than a fault. Rehearse it once in a non-production subscription: rotate, watch the identifier move, and only then call the path proven. Never disable an old key version until every consumer bound to it has moved across, including the ones you forgot about, like a disk encryption set or a SQL server pointed at the same key.
Managed HSM, And The Blueprints You Hold
A Premium Key Vault is a shared vault room inside a bank. Numbered boxes, good hardware, other customers standing in the same room. Managed HSM is your own room in that building, and the only set of blueprints for rebuilding it lives with you.
Those blueprints are the security domain. At activation you hand the HSM between three and ten RSA public keys plus a quorum number. The HSM generates its own internal keys, wraps them into a security domain file encrypted to your public keys, and hands the file back. Recovering or migrating that HSM later requires a quorum of the matching private keys, and Microsoft cannot produce them. That is the actual mechanism behind the claim that Microsoft operators cannot read your keys. It is also a loaded gun: lose the security domain file or the private keys and the pool becomes unrecoverable.
# 1. Activate the HSM: three officer certificates, any two of which can recover it.az keyvault security-domain download --hsm-name mhsm-acme \--sd-wrapping-keys ./sd-officer1.cer ./sd-officer2.cer ./sd-officer3.cer \--sd-quorum 2 \--security-domain-file ./mhsm-acme-SD.json# 2. Generate the KEK inside the HSM. RSA-HSM private material has no export API.az keyvault key create --hsm-name mhsm-acme --name tde-cmk \--kty RSA-HSM --size 3072 \--ops wrapKey unwrapKey \--query "{kid:key.kid, kty:key.kty, ops:key.keyOps}" -o jsonc# 3. Grant wrap/unwrap on that ONE key, using Managed HSM's own local RBAC.az keyvault role assignment create --hsm-name mhsm-acme \--role "Managed HSM Crypto Service Encryption User" \--assignee-object-id "$SQL_MI_PRINCIPAL" \--scope /keys/tde-cmk \--query "{scope:scope, principalId:principalId}" -o jsonc
# security-domain download writes ./mhsm-acme-SD.json and prints nothing on success{"kid": "https://mhsm-acme.managedhsm.azure.net/keys/tde-cmk/8c4f2b6d1e074a93b5f0c2a7d38e6104","kty": "RSA-HSM","ops": ["wrapKey","unwrapKey"]}{"principalId": "b7d4e21f-3c88-4a6e-9f10-52c7ab904d33","scope": "/keys/tde-cmk"}
The Managed HSM data plane runs its own local RBAC, entirely separate from Azure RBAC. A subscription Owner has zero key access unless someone assigns them a Managed HSM role, and those assignments come from the administrators set at provisioning, backed by the security domain quorum. That separation is the reason to be here: key control sits outside ordinary subscription privilege, so compromising a cloud admin account does not hand over the keys. Scoping to /keys/tde-cmk rather than / narrows the grant to a single key instead of the whole pool.
RSA-HSM private material is non-exportable by construction; no API returns it. The one exception is secure key release, where a key is created with --exportable and an attestation-backed release policy so an attested confidential computing enclave can receive it. That is opt-in at creation time. A key created the way shown above can never be exported by anyone, including you.
Managed HSM has no free tier. It provisions multiple HSM partitions for high availability and bills by the hour from the moment it exists, roughly three dollars an hour for a standard pool, on the order of a couple of thousand dollars a month whether you perform one unwrap or a billion. Climb this rung when a control requirement genuinely forces it, and keep ordinary CMK in a Premium vault for everything else.
The Kill Switch
Disabling the KEK is like pulling the barrel out of a lock. The door is still hanging there, the ciphertext is still sitting on disk, and nobody gets in.
# CONTAIN: disable the current key version.az keyvault key set-attributes --vault-name kv-acme-cmk --name storage-cmk --enabled false \--query "{kid:key.kid, enabled:attributes.enabled, updated:attributes.updated}" -o jsonc# VERIFY the blast radius with a real data-plane read.az storage blob download --account-name stacmedata01 -c logs -n app.log \-f /dev/null --auth-mode login# RESTORE once the incident is contained.az keyvault key set-attributes --vault-name kv-acme-cmk --name storage-cmk --enabled true \--query "attributes.enabled" -o tsv
{"enabled": false,"kid": "https://kv-acme-cmk.vault.azure.net/keys/storage-cmk/1b70d9c4f0a24e7bb2a1c8e35d64f097","updated": "2026-07-22T11:04:52+00:00"}The key vault key is not found to unwrap the encryption key.RequestId:9f3b1a27-701e-0044-3d0c-4a7e6b000000Time:2026-07-22T11:07:19.4471033ZErrorCode:KeyVaultEncryptionKeyNotFoundtrue
Read that error carefully, because it is not an authorization failure. KeyVaultEncryptionKeyNotFound comes back as HTTP 409 Conflict and says the key needed to unwrap the encryption key cannot be found. Knowing the difference saves twenty minutes of chasing RBAC when the real cause is a key that is disabled, expired, or sitting behind a vault firewall somebody tightened this morning.
Two honest caveats. Revocation propagates quickly but is not instantaneous, because services cache the unwrapped DEK for a period, so plan for minutes rather than milliseconds and do not assume an in-flight download dies mid-stream. And the blast radius is every resource bound to that key, not the one you had in mind. If a single shared KEK protects four storage accounts and a SQL server, disabling it takes all five down together. During a live compromise that may be exactly right. On a normal Tuesday it is the most expensive keystroke of your week, which is why the revoke-and-restore path gets rehearsed in a scratch subscription long before you need it under pressure.
Alert On The Key, Not On The Data
A smoke alarm earns its keep by going off before you smell smoke, not after the kitchen is alight. Alerting on failed blob reads is smelling smoke: by the time customers report errors, the key has been unusable for a while. Watch the key itself instead. And flag any new account that ships without a customer-managed key, so a missing key turns up on a compliance report rather than in an outage.
# Send the vault's audit trail to a Log Analytics workspace ($LAW_ID) you can query.az monitor diagnostic-settings create -n kv-audit \--resource "$KV_ID" \--workspace "$LAW_ID" \--logs '[{"category":"AuditEvent","enabled":true}]' -o none# Look the built-in policy up by display name instead of memorising a GUID.POLICY_ID=$(az policy definition list \--query "[?displayName=='Storage accounts should use customer-managed key for encryption'].id | [0]" -o tsv)az policy assignment create --name require-cmk-storage \--scope "/subscriptions/$SUB_ID/resourceGroups/rg-data" \--policy "$POLICY_ID" \--query "{name:name, enforcement:enforcementMode, policy:policyDefinitionId}" -o jsonc
{"enforcement": "Default","name": "require-cmk-storage","policy": "/providers/Microsoft.Authorization/policyDefinitions/6fac406b-40ca-413b-bf8e-0bf964659c25"}
One caveat on that policy. The built-in definition audits, it does not block. A storage account created without a customer-managed key still gets created, and Azure Policy marks it non-compliant a short while later, which is usually soon enough to catch the mistake before real data lands. If you want a hard stop at creation time, you author a custom policy with a Deny effect. For most teams, audit plus a real alert is enough.
// There is no "KeyDisable" operation. Disabling a key is an update to its attributes,// so the audit trail records KeyUpdate. Alert on that or you alert on nothing.AzureDiagnostics| where ResourceProvider == "MICROSOFT.KEYVAULT"| where OperationName in ("KeyUpdate", "KeyDelete", "KeyPurge", "KeyRecover")| project TimeGenerated, OperationName, id_s, identity_claim_oid_g,CallerIPAddress, ResultSignature| order by TimeGenerated desc
That naming detail catches people out. A KQL rule (KQL is the Kusto Query Language that Log Analytics runs) written against an operation called KeyDisable matches nothing, forever, while showing a healthy green tick on the dashboard. Pair the log alert with Event Grid on the vault, which emits Microsoft.KeyVault.KeyNearExpiry, Microsoft.KeyVault.KeyExpired, and Microsoft.KeyVault.KeyNewVersionCreated. The first two warn you that a rotation did not happen. The third is how you confirm one did, without polling the storage account.
Before a CMK goes in front of anything that matters, run the drill once end to end in a scratch subscription: wire the identity, confirm currentVersionedKeyIdentifier resolves, force a rotation, watch the identifier advance a day later, disable the key, read the 409 with your own eyes, re-enable, confirm reads come back. That hour buys you a calm thirty minutes during a real incident. Next, Secrets, references & rotation moves from encryption keys to the connection strings, tokens, and certificates your applications carry, which have the same rotation and revocation burden with no envelope trick to make it painless.
Try this
Run az storage account show -g rg-data -n stacmedata01 --query encryption -o jsonc on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: the managed identity is load-bearing, not an access grant. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.