HSM, EKM & key custody

FIPS hardware and keys Google cannot read.

A customer-managed encryption key (CMEK, meaning the key is yours rather than one Google quietly generated for you) proves ownership. It does not stop Google from using it. The controls that come with it are real: you rotate the key, you disable it, and every service that touched it shows up in the audit log. But when a bucket or a disk actually scrambles bytes, Google's software does the math, with your key sitting in Google's memory. For most workloads that is fine. For regulated data, or a contract that says the provider must never be able to read our data, owning a key and holding a key turn out to be two different things.

The custody ladder

A safe deposit box at a bank comes in four arrangements, ordered here from least control to most. With a Google-managed key, the bank owns both your box and the key, and trust is all you have. With software CMEK, the key is yours, but it hangs on a hook in the bank's back office where any teller can lift it and open your box. With Cloud HSM (hardware security module), your key sits inside a sealed strongbox that nobody can crack open or photocopy, though bank staff still fetch it and turn it on your behalf. With Cloud EKM (External Key Manager), the key never enters the bank at all. It stays in a vault inside your own building, and every single time the bank wants your box opened, it has to phone you, and you turn the key at your end.

Every rung up the ladder buys more control and costs more operational weight. Climb only as high as the requirement actually demands. Each level adds availability risk and a ritual somebody has to keep up forever.

Cloud HSM: keys sealed in hardware

An HSM is tamper-resistant hardware built to hold keys and never let them out. Cloud HSM gives you keys that live in FIPS 140-2 Level 3 hardware. FIPS 140-2 (Federal Information Processing Standard 140-2) is the US government standard for crypto hardware, and Level 3 means the device fights back physically. Drill into it or pop the lid, and it wipes the key rather than hand it over. The key is generated inside the module and never leaves in plaintext, not even to Google. Getting there is one flag.

$ gcloud kms keys create signing --location=europe-west1 --keyring=app \
    --purpose=asymmetric-signing --protection-level=hsm \
    --default-algorithm=ec-sign-p256-sha256
 
$ gcloud kms keys versions describe 1 --key=signing --keyring=app \
    --location=europe-west1 \
    --format='value(state,protectionLevel,attestation.format)'
ENABLED	HSM	CAVIUM_V2_COMPRESSED

That describe call hands back an attestation, a signed statement from the hardware vendor's chip proving this key really was generated inside a genuine FIPS Level 3 module and cannot be exported. That is the artifact an auditor wants to see. For a true root of trust, say the signing key of a certificate authority (the service that issues the certificates your systems trust), you generate it in the HSM as a key ceremony: several named custodians, dual control so nobody acts alone, split knowledge so no single person knows the whole secret, all of it witnessed and written down. Assured Workloads can then pin the whole project to one region, to vetted support staff, and to hardware-key requirements for sovereignty regimes.

Cloud EKM: the key never leaves your building

Cloud HSM stops Google exporting your key, but Google's servers still run the encryption. If your threat model says the provider must not be able to decrypt at all, even when handed a subpoena (a court order compelling a company to produce data), the key material itself has to live outside Google. That is Cloud EKM. You run a key service of your own (Thales, Fortanix and Futurex all sell one), and Google keeps only a pointer to it. Every encrypt and every decrypt turns into a call out to your service, which answers or refuses.

There are two ways to wire it up. One goes over the public internet: Google reaches your key at a web address, an HTTPS URI (uniform resource identifier, the https:// address of your key service). The other, external-VPC, reaches your key service over a private network path inside your own VPC (virtual private cloud, your isolated network inside Google) using Service Directory, and that is what most production setups use. Register the connection first, then create a key that points at it.

$ gcloud kms ekm-connections create ext --location=europe-west1 \
    --service-directory-service="projects/keys-prod/locations/europe-west1/namespaces/ekm/services/vault" \
    --hostname=ekm.corp.example.com \
    --server-certificates-files=ekm-ca.pem \
    --key-management-mode=cloud-kms \
    --crypto-space-path="v0/2f9c-cryptospace"
Created ekmConnection [projects/keys-prod/locations/europe-west1/ekmConnections/ext].
 
$ gcloud kms keys create ekm-data --location=europe-west1 --keyring=app \
    --purpose=encryption --protection-level=external-vpc \
    --crypto-key-backend="projects/keys-prod/locations/europe-west1/ekmConnections/ext"
 
$ gcloud kms keys describe ekm-data --location=europe-west1 --keyring=app \
    --format='value(primary.state,versionTemplate.protectionLevel)'
ENABLED	EXTERNAL_VPC

Now bolt that key onto real data and see what the custody actually buys you. Create a bucket that encrypts every object with the EKM key. Then, days later, disable that key inside your own external manager and try to read something back.

$ gcloud storage buckets create gs://prod-regulated-eu --location=europe-west1 \
    --default-encryption-key="projects/keys-prod/locations/europe-west1/keyRings/app/cryptoKeys/ekm-data"
Creating gs://prod-regulated-eu/...
 
$ gcloud storage cp gs://prod-regulated-eu/q3-report.csv .
ERROR: (gcloud.storage.cp) HTTPError 400: FAILED_PRECONDITION: the external
key projects/keys-prod/.../cryptoKeys/ekm-data is disabled in the external
key management system and cannot be used to decrypt the object.

That FAILED_PRECONDITION is the whole point. You flipped a switch in your own key manager, and an object sitting on Google's disks went dark, to you and to Google alike. No court order served on Google reverses it, because Google never held the material in the first place.

When the phone line goes down

So monitor it the way you monitor production. Turn on Cloud KMS (Key Management Service) data-access logging, count the failed crypto operations, and page the on-call engineer before anyone gets around to filing a ticket.

$ gcloud logging metrics create ekm-op-failures \
    --description="Cloud KMS crypto ops returning an error (HSM/EKM backends)" \
    --log-filter='resource.type="cloudkms_cryptokey" protoPayload.status.code!=0'
Created [ekm-op-failures].

displayName: "KMS crypto operation failures (HSM/EKM)"
combiner: OR
conditions:
- displayName: "Decrypt/Encrypt errors in 5 min"
  conditionThreshold:
    filter: 'metric.type="logging.googleapis.com/user/ekm-op-failures" AND resource.type="cloudkms_cryptokey"'
    comparison: COMPARISON_GT
    thresholdValue: 0
    duration: 300s
    aggregations:
    - alignmentPeriod: 60s
      perSeriesAligner: ALIGN_SUM
notificationChannels:
- projects/keys-prod/notificationChannels/9137654201234567890
 
$ gcloud monitoring policies create --policy-from-file=ekm-alert.yaml
Created alert policy [projects/keys-prod/alertPolicies/7420193856120004417].

Notice that none of your application code changes as you climb. You call the same Cloud KMS APIs whether the key is software, HSM or EKM. What moves is the protectionLevel field and the failure mode hiding behind it. With HSM, the material is locked inside FIPS-validated hardware that Google still operates, which closes off a whole class of worry about software-resident keys without adding a third-party network hop. With EKM, the decrypt operation leaves Google entirely and waits on your own hardware or a partner's. Google never sees the plaintext key material. Auditors love that sentence. Operators have to love the other half of it equally: if your EKM endpoint is unreachable, every read of that data fails until the path comes back.

Pick a rung and write down why. Google-managed keys for low-sensitivity data. Software CMEK when you want a kill switch and IAM (Identity and Access Management, Google's permission system) applied to the key itself. HSM when hardware custody is a contractual checkbox. EKM when the contract says the provider must not be able to decrypt. Do not move your busiest analytics lake onto EKM because it sounds impressive. The added latency and the availability coupling are real, and the people running queries will find them before you do.

Rotation and disabling act on key versions, not on the key name, so an old version can still be unwrapping old data long after you thought you had moved on. Inventory the protectionLevel field across every project too. One SOFTWARE key quietly sitting under a dataset the policy said must be HSM is the kind of finding that turns up in an audit report rather than on a dashboard.

Put the key configuration where the rest of your infrastructure already lives: a reviewed change, a pipeline that applies it, and a check that fails the build when someone reverts it. A fix clicked into the console evaporates the week after the audit. Write down the project, the identity allowed to run these gcloud commands, and the log query that proves they worked, then rehearse it until a new teammate can follow the path without asking anyone in the hallway.

Try this

Compare protection levels across the keys you already have, and, in a lab project, describe an HSM-backed key. Real EKM needs a partner key service you probably do not have sitting around, so the drill here is reading protectionLevel and key state well enough to spot SOFTWARE against HSM against EXTERNAL in an inventory listing.

gcloud kms keys list --location=europe-west1 --keyring=payments \
  --format="table(name.basename(),primary.protectionLevel,primary.state,purpose)"
gcloud kms keys describe hsm-cmek \
  --location=europe-west1 --keyring=payments \
  --format="yaml(primary.protectionLevel,primary.algorithm,primary.state)"

NAME         PROTECTION_LEVEL  STATE    PURPOSE
bucket-cmek  SOFTWARE          ENABLED  ENCRYPT_DECRYPT
hsm-cmek     HSM               ENABLED  ENCRYPT_DECRYPT
 
primary:
  protectionLevel: HSM
  algorithm: GOOGLE_SYMMETRIC_ENCRYPTION
  state: ENABLED

Takeaway

CMEK proves the key is yours. HSM and EKM change where the private material physically sits, and only the EKM rung puts it outside Google's reach. Climb as far as your contracts and threat model demand, no further, and budget for the day your own key service stops answering the phone.

Next you stop baking passwords into container images at all. Secret Manager is the subject, because key custody does nothing for you while the database password is still sitting in an image layer.

Keys protect data sitting on disk. Applications also need secrets they read at runtime: database passwords, API tokens, service-account credentials. Those belong in Secret Manager, which encrypts them with the very software, HSM or EKM keys you have just built. Next you set it up and rotate those secrets without taking anything down.

Related