You are mid-incident. Attackers have a foothold in the project that owns gs://payments-data, so you disable key version 1, the version that encrypted every object already sitting in that bucket. What happens to those objects?

Every read fails until you re-enable the version; the data is intact, but sealed. Disabling the version that wrapped the data stops KMS unwrapping its DEKs, so reads come back with a 403 KMS error while the ciphertext sits untouched. Re-enable the version and access returns with nothing lost.

You created a customer-managed encryption key (CMEK) with --rotation-period=90d. Ninety days later Cloud KMS rotates it. What happens to the objects that were written under the previous key version?

They keep working. The old version stays enabled and still decrypts what it originally wrapped, while only new writes use the new version. Rotation sets a new primary for new writes and leaves old versions in place, so rotating never forces a re-encryption.

You create a KMS key, set it as a bucket's default encryption key with gcloud storage buckets update, and the command succeeds. Then every new upload to that bucket fails with a permission error naming the key. What fixes it?

Grant the Cloud Storage service agent roles/cloudkms.cryptoKeyEncrypterDecrypter on the key, because the bucket encrypts as that robot account, not as you. The bucket's writes run as the Cloud Storage service agent, so until that agent may use the key, every write is denied.

Courses GCP securityCloud KMS & CMEK

Cloud KMS & CMEK

Envelope encryption, key IAM, rotation, kill switch.

Advanced35 min · lesson 7 of 15

Every byte you store in Google Cloud is already encrypted on disk. Create a bucket, drop a file in, and Google has quietly scrambled it with a key it generated and looks after for you. The encryption box on your security review is ticked before you touch anything. The question that actually decides the outcome is a different one. Who holds the key? Out of the box, the answer is Google, and only Google. Customer-managed encryption keys (CMEK, meaning the key protecting your data is one you create, control and can switch off) hand that answer over to you.

Your flat came with the lock the building fitted, and the property manager keeps a copy. That is fine for coats and furniture. It stops being fine the day you store something you would rather the manager could not reach, so you add your own padlock and keep the only key. CMEK is that padlock. You create a key inside a Google service called Cloud KMS (Key Management Service, a hosted vault that holds keys and does the sealing and opening on request), then point a bucket, a database or a disk at it. Google still runs the encryption machinery. It cannot open your data without asking your key first, and you decide who is allowed to ask.

There is a trick underneath, and rotation and revocation only make sense once you have seen it. Encrypting a terabyte of data straight from a key that lives inside KMS would mean pushing a terabyte through KMS on every read. Nobody does that. Instead Google generates a fast local key for that one pile of data, the data encryption key (DEK, the key that scrambles the actual bytes), and runs it over the file. Then it takes your KMS key, the key encryption key (KEK, a key whose only job is to encrypt other keys), and wraps the DEK with it. The wrapped DEK is stored right next to the data. Your KEK never leaves KMS. One key sealed inside another is called envelope encryption. To read anything back, the service has to ask KMS to unwrap the DEK with your key. Cut off access to your key and every DEK stays sealed. That is what turns a key into a kill switch.

Bring your own padlock

A key does not float around loose. It lives in a key ring, a named container tied to one region, like a labelled pegboard you hang related keys on. Choose the region with care. A key in europe-west1 can only protect resources in a compatible location, and you cannot move it later. Here is a ring for payments data, and inside it a key that rotates itself every 90 days.

create the key ring + a rotating key

gcloud kms keyrings create payments --location=europe-west1
 
gcloud kms keys create bucket-cmek \
  --location=europe-west1 --keyring=payments \
  --purpose=encryption \
  --rotation-period=90d --next-rotation-time=2026-10-14T00:00:00Z
 
gcloud kms keys describe bucket-cmek \
  --location=europe-west1 --keyring=payments

gcloud kms keys describe (output)

name: projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek
purpose: ENCRYPT_DECRYPT
primary:
  name: projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek/cryptoKeyVersions/1
  state: ENABLED
  protectionLevel: SOFTWARE
  algorithm: GOOGLE_SYMMETRIC_ENCRYPTION
rotationPeriod: 7776000s
nextRotationTime: '2026-10-14T00:00:00Z'
createTime: '2026-07-16T09:12:44.021Z'
versionTemplate:
  algorithm: GOOGLE_SYMMETRIC_ENCRYPTION
  protectionLevel: SOFTWARE

Rotation means Cloud KMS mints a fresh version of the key on a schedule and makes that version the primary one for new writes. Old versions stay exactly where they are, still able to decrypt whatever they originally wrapped, so a rotation never forces you to re-encrypt anything. New objects get the new version. Old objects carry on with theirs. The next-rotation-time flag sets when the first rotation fires, and rotation-period sets the gap between every one after that.

One catch trips up nearly everyone the first time. The bucket does not encrypt as you. It encrypts as its own robot account, the Cloud Storage service agent, an identity Google creates in your project so the service can do work on your behalf. It is the building's mailroom clerk who actually turns the lock for you. Give that clerk no permission to use your padlock and every write dies with a permission error. So you grant the service agent the one KMS role that means 'you may encrypt and decrypt with this key', then attach the key to the bucket as its default.

authorize the service agent, attach the key

gcloud storage service-agent --project=payments-prod
 
gcloud kms keys add-iam-policy-binding bucket-cmek \
  --location=europe-west1 --keyring=payments \
  --member="serviceAccount:[email protected]" \
  --role=roles/cloudkms.cryptoKeyEncrypterDecrypter
 
gcloud storage buckets update gs://payments-data \
  --default-encryption-key=projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek

console (output)

[email protected]
 
Updated IAM policy for key [bucket-cmek].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYh3n2K9pE=
version: 1
 
Updating gs://payments-data/...
  Completed 1/1

Prove the padlock is on

Never trust that a control landed because the command exited cleanly. Ask the bucket which key it is using, then check a real object that is already sitting in there.

confirm the bucket + object use the key

gsutil kms encryption gs://payments-data
 
gsutil stat gs://payments-data/ledger-2026-07.parquet

gsutil (output)

Default encryption key for gs://payments-data:
projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek
 
gs://payments-data/ledger-2026-07.parquet:
    Creation time:      Thu, 16 Jul 2026 09:20:31 GMT
    Storage class:      STANDARD
    Content-Length:     5242880
    KMS key:            projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek/cryptoKeyVersions/1

Attaching a key changes the default for new objects only. Files already in the bucket keep whatever key wrote them, so retrofitting CMEK onto old data means rewriting those objects to move them onto the new key. The same key covers more than buckets. Point a BigQuery dataset at it and every table created inside inherits it as the default, as long as the dataset lives in the key's region.

same key as the default for a BigQuery dataset

bq update \
  --default_kms_key=projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek \
  payments-prod:ledger

bq (output)

Dataset 'payments-prod:ledger' successfully updated.

Who can use the key is the real fence

Here is the part people miss. The key carries its own IAM (Identity and Access Management, Google's record of who is allowed to do what to a resource). That record is a separate fence from the bucket's, and it is the one nobody thinks to read. For Cloud Storage you never hand the key to the person downloading a file. The service agent opens the padlock for them behind the scenes, so a reader only needs permission on the bucket itself. What matters is keeping the list of identities you grant encrypterDecrypter to as short as you can bear: the service agent, maybe one application account, nobody else. Everyone on that list can ask KMS to decrypt with the key, so one careless grant quietly widens who can turn your data back into plaintext, however tight the bucket looks. Split the roles as well. Whoever can rotate, disable or destroy the key (roles/cloudkms.admin) should not be the same identity that can decrypt with it (roles/cloudkms.cryptoKeyEncrypterDecrypter). Treat 'can use this key' as the sensitive permission it is, and audit who holds it the way you would audit who has a master key to the vault.

Blast radius is the reason to stop and think before reusing one key everywhere. A single key can be the default for a dozen buckets, several datasets and a fleet of disks spread across projects. Tidy, right up to the morning you disable it and all of them go dark together. Scope keys to a blast radius you can live with, usually one per workload or per sensitivity tier, rather than one key for the whole organisation. Cost is not what holds you back here. A key version runs a few cents a month plus a tiny charge per crypto operation, so having more keys is cheap.

The kill switch

Mid-incident, the fastest way to make a pile of data unreadable to an attacker who is already inside is to pull the key. You delete nothing. You disable the key version that wrapped the data, and every DEK sealed under it stops unwrapping at once. A read that would have returned plaintext a second earlier now fails at the KMS boundary.

disable a version, then try to read

gcloud kms keys versions disable 1 \
  --location=europe-west1 --keyring=payments --key=bucket-cmek
 
gsutil cp gs://payments-data/ledger-2026-07.parquet .

console (output)

name: projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek/cryptoKeyVersions/1
state: DISABLED
 
Copying gs://payments-data/ledger-2026-07.parquet...
AccessDeniedException: 403 Cloud KMS error: the key version
projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek/cryptoKeyVersions/1
used to decrypt this object is disabled.

Re-enable the version and access comes straight back with no data lost, which is exactly why disabling rather than destroying is the move you make under pressure. Software CMEK like this still has Google running the crypto with key material Google stores. When you need harder separation, up to the point where the provider cannot decrypt at all, you move to keys held in tamper-resistant hardware or kept entirely outside Google. That is where the next lesson goes.

A destroyed key takes the data with it

Destroying a key version is permanent, and it destroys access to the data along with it. Once the scheduled-destruction window passes, that version cannot be recovered, and every object, table and disk still encrypted under it becomes unreadable for good. No support ticket brings it back. Two habits keep this from ending a career. First, lean on the waiting period. A scheduled destroy does not take effect straight away, because Cloud KMS parks it for a delay first (at least 24 hours, and you can set it far longer), so a mistaken destroy can be cancelled by restoring the version while it is still pending. Second, remember the coupling runs both ways. If the KMS region holding your key has an outage, the data it protects is unreadable until that region recovers, even though the stored bytes are perfectly intact. Your data's availability now rides on your key's availability.

One CMEK key, its access, and its blast radius

Cloud KMS · europe-west1

key ring: payments

regional container, can't be moved

key: bucket-cmek

rotates every 90d, version 1 primary

disable a version

instant, reversible kill switch

who may touch the key

roles/cloudkms.admin

rotate / disable / destroy, no decrypt

cryptoKeyEncrypterDecrypter

storage service agent + app account, use only

resources encrypted by this one key

gs://payments-data

CMEK default for new objects

BigQuery dataset: ledger

new tables inherit the key

disable the key, all go dark

that is the blast radius

One key can sit behind many resources across several projects. Handy, until you disable it and every one of them goes unreadable at the same moment. Scope keys to a blast radius you can live with, and keep the admin who can destroy the key apart from the identities that only use it.

Envelope encryption is why CMEK scales. Your KEK in Cloud KMS never bulk-encrypts terabytes; it wraps short-lived DEKs that sit next to the objects. That same design is what makes disabling a version such a sharp lever, because every read path has to ask KMS to unwrap, so a disabled version fails closed across every object still sealed under it.

The operating habits matter as much as the tick box. Put keys in the region that matches the data, name rings after the workload they serve, keep the admin role away from the crypto-user role, and rehearse a disable and a re-enable in a lab long before you need it at 3 a.m. Attaching a key only changes the default for new writes, so plan the rewrite job for legacy objects rather than assuming history re-keyed itself.

Try this

If you created a CMEK earlier in the lesson, describe it again and confirm the bucket still names that key. If not, create a ring and a key in a lab project and attach it to a throwaway bucket.

terminal

gcloud kms keys describe bucket-cmek \
  --location=europe-west1 --keyring=payments \
  --format="yaml(name,primary.state,rotationPeriod)"
gsutil kms encryption gs://payments-data
gcloud kms keys get-iam-policy bucket-cmek \
  --location=europe-west1 --keyring=payments

output

name: projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek
primary:
  state: ENABLED
rotationPeriod: 7776000s
 
Default encryption key for gs://payments-data:
projects/payments-prod/locations/europe-west1/keyRings/payments/cryptoKeys/bucket-cmek
 
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter

Takeaway

CMEK settles who holds the kill switch. It does not invent the encryption, because Google already did that part. Scope each key to a blast radius you can survive, hand encrypterDecrypter to the service agents that genuinely need it and to nobody else, and reach for disable rather than destroy when an incident means sealing the room right now.

Next you climb the custody ladder to HSM (hardware security module, tamper-resistant hardware that holds key material and never lets it out) and external keys, for the cases where Google holding even the wrapped key material is still too much trust.

VPC Service ControlsHSM, EKM & key custody

IAM mental modelAWS security for DevOps engineers Multi-account & org identity architectureAdvanced cloud security The IAM evaluation modelAWS security engineering Entra ID, Conditional Access & PIMAzure security Python for DevSecOps cheat sheetCheat sheet DevSecOps & supply-chain security interview questionsInterview prep