BYOK/HYOK, HSM & key ceremonies

External key stores, FIPS HSMs, and provable custody.

A bank's safe-deposit box is handy right up until you read the fine print. The bank keeps a guard key, and with a court order it can drill your lock open. Buy your own safe, bolt it to your own floor, and the bank can still warehouse the sealed box for you, but it has no way inside. Cloud key management is that same ladder, one rung at a time. A provider-default key is the bank's box. A customer-managed key hands you a second key plus a logbook of every time the box was opened. BYOK (bring your own key) lets you cut the key yourself and mail it in sealed, so you can prove exactly where it came from. HYOK (hold your own key) keeps the key inside your building, so the provider has to phone you for every single open, and the day you stop answering, the box is welded shut. Every rung up buys provable control and charges for it in availability risk and operational ritual. The real skill is naming the rung a requirement actually needs, not grabbing the highest one you can reach.

The Custody Ladder in Plain Words

Start with the words, because the marketing smears them together. A provider-default key (an AWS-managed key, a Google-managed key, a Microsoft-managed key) is generated, stored, rotated, and used entirely by the provider. You cannot read its policy and you cannot revoke it. A customer-managed key (an AWS KMS key, once called a CMK or customer master key; a GCP CMEK or customer-managed encryption key; an Azure Key Vault key) still lives inside the provider's KMS (key management service, the managed service that stores your keys and runs the actual encrypt and decrypt calls). The difference is that now you own the access policy, the rotation schedule, and a disable switch. Flip that switch and the key management service refuses every new decrypt call, and the data behind the key goes dark. That switch is the most useful single control on the whole ladder, and most teams never once test that it works.

BYOK narrows the trust by one notch. You generate the raw key bytes in your own HSM (hardware security module, a tamper-resistant box that makes and holds keys and refuses to hand them out in the clear), wrap them so only the provider can open them, and import the sealed blob. Now you can prove the key was born in your custody and shred your local copy. HYOK goes all the way. Sold as an external key store on AWS and an external key manager on GCP, it keeps the key bytes outside the provider's KMS entirely, and every encrypt or decrypt is forwarded out to a store you run. Each rung answers a sharper question about who, in the worst case, could read your data.

BYOK: Mailing in a Sealed Key

You would never hand a spare house key to a courier bare, trusting the whole chain not to copy it. BYOK works like a tamper-proof envelope instead. The provider mints a one-time envelope for you, an RSA (Rivest-Shamir-Adleman, a public-key crypto system) public key whose matching private half lives locked inside the provider's own HSM. You seal your key bytes inside that envelope using RSA-OAEP (optimal asymmetric encryption padding, the modern padding that makes the wrapping safe), and you mail the sealed envelope back with a one-time import token that binds it to this one key slot and a short time window. Only the provider's HSM can open the envelope, and only inside the short window before that wrapping key expires. Here is the whole flow on AWS.

# 1. make a key slot that AWS puts NO material into
KID=$(aws kms create-key --origin EXTERNAL \
        --description byok-app-data \
        --query KeyMetadata.KeyId --output text)
 
# 2. ask AWS for a one-time RSA wrapping key + import token
aws kms get-parameters-for-import --key-id "$KID" \
  --wrapping-algorithm RSAES_OAEP_SHA_256 \
  --wrapping-key-spec RSA_2048

{
    "KeyId": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "ImportToken": "AQECAHh8m4...WY0Zpc9k=",
    "PublicKey": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A...IDAQAB",
    "ParametersValidTo": "2026-07-23T10:15:41.502000+00:00"
}

# wrap_pub.der and token.bin were base64-decoded out of the response above.
# wrap your own 32-byte AES-256 (advanced encryption standard) key to AWS's
# public key, RSA-OAEP / SHA-256. Set the MGF1 hash to SHA-256 too, or AWS
# rejects the blob. The wrapping key expires (ParametersValidTo).
openssl pkeyutl -encrypt -pubin -keyform DER -inkey wrap_pub.der \
  -in material.bin -out wrapped.bin \
  -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 \
  -pkeyopt rsa_mgf1_md:sha256
 
aws kms import-key-material --key-id "$KID" \
  --encrypted-key-material fileb://wrapped.bin \
  --import-token fileb://token.bin \
  --expiration-model KEY_MATERIAL_DOES_NOT_EXPIRE
 
shred -u material.bin   # destroy the last local plaintext copy
 
# prove it took
aws kms describe-key --key-id "$KID" \
  --query 'KeyMetadata.{State:KeyState,Origin:Origin,Exp:ExpirationModel}'

{
    "State": "Enabled",
    "Origin": "EXTERNAL",
    "Exp": "KEY_MATERIAL_DOES_NOT_EXPIRE"
}

Google packages the same wrapping key into a reusable import job, then imports key versions against it. The import job's protection level decides where the wrapping key lives, so an hsm job is what lets your bytes land in Google's hardware.

# a key slot that will only ever hold imported material (no generated v1)
gcloud kms keys create app-data \
  --location us-central1 --keyring prod \
  --purpose encryption --protection-level hsm \
  --skip-initial-version-creation
 
# an import job holds the RSA wrapping key inside Google's HSM
gcloud kms import-jobs create prod-import \
  --location us-central1 --keyring prod \
  --import-method rsa-oaep-3072-sha256-aes-256 \
  --protection-level hsm

Created key [app-data].
Created import job [prod-import].

# gcloud fetches the job's public key, wraps material.bin locally, uploads it
gcloud kms keys versions import \
  --import-job prod-import --location us-central1 --keyring prod \
  --key app-data --algorithm google-symmetric-encryption \
  --target-key-file material.bin
 
# prove the live version is HSM-backed and came from your import job
gcloud kms keys versions describe 1 --key app-data \
  --location us-central1 --keyring prod \
  --format='value(state, protectionLevel, importJob.basename())'

ENABLED  HSM  prod-import

Azure expects the wrapping to happen offline first, so you hand it a finished .byok blob (your key already wrapped to the vault's key exchange key, or KEK). The imported result is non-exportable and shows up as an HSM-backed symmetric key.

# app-data.byok was produced offline: your key wrapped to the HSM's
# KEK (key exchange key). Azure never sees the plaintext.
az keyvault key import --hsm-name contoso-mhsm --name app-data \
  --byok-file app-data.byok

{
  "attributes": {
    "enabled": true,
    "exportable": false
  },
  "key": {
    "kid": "https://contoso-mhsm.managedhsm.azure.net/keys/app-data/8f3c1e...",
    "kty": "oct-HSM",
    "keyOps": ["encrypt", "decrypt", "wrapKey", "unwrapKey"]
  }
}

Two honest limits sit under all of that. First, import proves provenance, not exclusivity. It proves the key was born in your custody. It does not prove the provider cannot use it, because the moment your bytes are unwrapped inside the provider's KMS, the provider runs the cryptography on your behalf and could, under the right legal or insider pressure, run it without you. Second, the provider will not auto-rotate material it did not generate. On AWS you rotate imported bytes by importing fresh bytes yourself; on GCP you import a new version; on Azure you import a new key version. That standing re-import chore is the cost people forget at signing time, and it is the thing that quietly rots six months later.

HSMs and the Key Ceremony

An HSM is a safe that shreds its own contents if you tilt it. More precisely, it is a tamper-resistant appliance validated against FIPS 140-2 (Federal Information Processing Standard 140-2, the US government yardstick for crypto hardware; the newer edition is 140-3). It generates keys inside a sealed boundary, never emits them in plaintext, and at Level 3 it actively zeroizes (wipes every key to zero) the instant it detects someone physically prying it open. Each cloud rents you one. AWS has CloudHSM, which you can put behind KMS as a custom key store. GCP has an hsm protection level on ordinary KMS keys. Azure has Managed HSM, a single-tenant pool of Level 3 hardware. All three sit at FIPS 140-2 Level 3 today, and the newest AWS CloudHSM hardware is validated to FIPS 140-3 Level 3.

When you create a top-level key in one of these, a BYOK root or a certificate authority (CA) signing key, you do it as a key ceremony. Think launch codes. No lone officer can turn the key alone. You bring several custodians, split the secret so no one person holds all of it, run a written and witnessed script, and record the whole thing. This is not theater. Azure makes it concrete and unforgiving: a freshly provisioned Managed HSM is inert until you download its security domain, which encrypts the HSM's root of trust to three or more customer-held RSA keys under an M-of-N quorum (you decide that any M of the N holders, say any 2 of 3, are needed to recover it). Those quorum holders are the only path back. Lose enough of them and the data is gone for good. Leak them and your custody is broken. Here are hardware-backed keys on all three clouds.

# key generated INSIDE Google's FIPS 140-2 Level 3 HSM, never exportable
gcloud kms keys create payments-root \
  --location us-central1 --keyring prod \
  --purpose encryption --protection-level hsm
 
# pull the hardware attestation: proof the bytes were born in an HSM,
# signed up a chain you can verify against the HSM vendor
gcloud kms keys versions describe 1 --key payments-root \
  --location us-central1 --keyring prod \
  --format='value(protectionLevel, attestation.format)'

HSM  CAVIUM_V2_COMPRESSED

# provision a single-tenant, FIPS 140-2 Level 3 Managed HSM pool
# (--administrators takes Entra object IDs, OIDs, not names)
az keyvault create --hsm-name contoso-mhsm --resource-group prod-rg \
  --location eastus2 --administrators "$ADMIN_OID" --retention-days 28
 
# THE CEREMONY: split the HSM's root of trust across 3 custodian keys,
# any 2 of which are needed to ever recover it. This also activates the HSM.
az keyvault security-domain download --hsm-name contoso-mhsm \
  --sd-wrapping-keys cust1.cer cust2.cer cust3.cer \
  --sd-quorum 2 \
  --security-domain-file contoso-mhsm-SD.json

{
  "name": "contoso-mhsm",
  "properties": {
    "hsmUri": "https://contoso-mhsm.managedhsm.azure.net/",
    "provisioningState": "Succeeded"
  }
}
 
Security domain downloaded to 'contoso-mhsm-SD.json'. HSM 'contoso-mhsm' is now activated.

# the CloudHSM cluster must already hold >= 2 active HSMs in different AZs
aws kms create-custom-key-store \
  --custom-key-store-name prod-cloudhsm \
  --cloud-hsm-cluster-id cluster-1a2b3c4d5e6 \
  --trust-anchor-certificate file://customerCA.crt \
  --key-store-password "$KMSUSER_PW"
 
aws kms connect-custom-key-store --custom-key-store-id cks-1234567890abcdef0
# (connecting runs several minutes; state goes CONNECTING -> CONNECTED)
 
# verify the KMS-to-HSM link is live before you put keys in it
aws kms describe-custom-key-stores \
  --custom-key-store-id cks-1234567890abcdef0 \
  --query 'CustomKeyStores[0].ConnectionState' --output text

{
    "CustomKeyStoreId": "cks-1234567890abcdef0"
}
 
CONNECTED

HYOK: Cutting the Provider Out

Sometimes the requirement is blunt: the provider must be cryptographically unable to read the data, full stop. Import will not do it, because the bytes still land inside the provider's KMS. HYOK moves them out. The provider becomes a courier who carries a locked box to your door and back for every single read. AWS External Key Store (XKS) routes every KMS operation through a proxy to your HSM. GCP External Key Manager (EKM) binds a key version to an external URI that your key manager answers, over the public internet or privately over a VPC (virtual private cloud) connection. The provider now performs no cryptography of its own. It forwards the request and relays your answer. Block the proxy and even a subpoenaed provider hands back nothing but ciphertext.

# register YOUR external store; every KMS op will proxy to this endpoint
aws kms create-custom-key-store \
  --custom-key-store-name onprem-xks \
  --custom-key-store-type EXTERNAL_KEY_STORE \
  --xks-proxy-connectivity PUBLIC_ENDPOINT \
  --xks-proxy-uri-endpoint https://xks.acme.example.com \
  --xks-proxy-uri-path /kms/xks/v1 \
  --xks-proxy-authentication-credential \
      'AccessKeyId=AKIAXKSEXAMPLEACCESS,RawSecretAccessKey=REDACTED'
 
aws kms connect-custom-key-store --custom-key-store-id cks-ext0987654321fedcba
 
# a KMS key whose bytes live in your store, referenced by its external ID
aws kms create-key --origin EXTERNAL_KEY_STORE \
  --custom-key-store-id cks-ext0987654321fedcba \
  --xks-key-id external-aes-9f2e \
  --description residency-locked \
  --query 'KeyMetadata.{Origin:Origin,Xks:XksKeyConfiguration.Id}'

{
    "CustomKeyStoreId": "cks-ext0987654321fedcba"
}
 
{
    "Origin": "EXTERNAL_KEY_STORE",
    "Xks": "external-aes-9f2e"
}

# an EKM key: material stays at an external URI your key manager serves
gcloud kms keys create ext-data \
  --location us-central1 --keyring prod \
  --purpose encryption --protection-level external \
  --skip-initial-version-creation
 
gcloud kms keys versions create \
  --key ext-data --location us-central1 --keyring prod \
  --external-key-uri https://ekm.acme.example.com/v0/keys/9f2e
 
# verify the version is bound to YOUR endpoint, not Google hardware
gcloud kms keys versions describe 1 --key ext-data \
  --location us-central1 --keyring prod \
  --format='value(state, externalProtectionLevelOptions.externalKeyUri)'

Created key [ext-data].
Created version [1] of key [ext-data].
ENABLED  https://ekm.acme.example.com/v0/keys/9f2e

Azure has no customer-hosted external store for at-rest encryption of its services. Its ceiling is Managed HSM BYOK, single-tenant hardware that you still do not physically hold. That is a real gap to know before you promise an auditor a provider-exclusion control on Azure. And the HYOK bill is steep everywhere it exists. Every encrypt and every decrypt now crosses the network to your endpoint, so your store's latency becomes your data plane's latency and your store's uptime becomes your data's uptime. An external key store outage is a data outage, not a slowdown. There is no cached fallback by design, because a cache would defeat the entire point of keeping the key out.

Pick the Rung, Then Price It

Climb only as high as the requirement genuinely demands, and put a number on the rung before you commit. A CloudHSM custom key store needs a cluster of at least two active HSMs in separate availability zones (AZs, the isolated data-center groups inside a cloud region), billed per HSM-hour whether or not you use them. A Managed HSM reserves a hardware pool with a standing monthly floor. An external store adds a self-run, highly available proxy fleet plus per-operation network latency on every read. Against all that, a plain HSM-backed customer-managed key already gives most regulated workloads their disable switch, their FIPS attestation, and their audit trail at a fraction of the weight. The grown-up pattern is a mixed portfolio: HSM-backed keys for the bulk of at-rest data, BYOK where an auditor demands provable origin, and HYOK held back for the narrow slice, a specific data-residency or provider-exclusion rule, that truly cannot tolerate the provider ever touching the key.

Whichever rung you land on, verify it instead of trusting the console. Read the key back. A KeyState of Enabled with Origin EXTERNAL means your import took. A protectionLevel of HSM with a CAVIUM attestation format means the bytes really were born in hardware. A ConnectionState of CONNECTED, or an externalKeyUri on the live version, means your external store is actually wired in. Then test the two things that bite in production. Flip the disable switch in a staging account and watch a read fail, so you know your kill switch is real before you ever need it. And measure your external key store against a hard uptime budget, because from the moment you choose HYOK, your key manager's worst five minutes are your application's worst five minutes. The key is only half the system anyway. The credentials and tokens it protects still have to be issued, scoped, and rotated at runtime, which is where managed secrets stores come in next.

Try this

Run shred -u material.bin # destroy the last local plaintext copy on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: bYOK proves origin, not exclusion. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related