>_SecOpsLog_
CoursesKubernetes attack & defenseControl plane & etcd

Control plane & etcd

Encrypt secrets at rest, lock the API server, guard the CA.

Expert35 min · lesson 10 of 15

Every request to the API server, the cluster's single front door, gets frisked. Who are you (that's authentication), and are you allowed to do this (that's authorization, which Kubernetes runs through Role-Based Access Control, or RBAC). The database sitting behind that front door frisks no one. That database is etcd, the cluster's key-value store, and it will hand its entire contents to anyone who can open a connection to it. Think of etcd as the building's master key cabinet. The front desk checks badges and logs every visitor who walks in. The cabinet in the back just holds every key to every room, and if someone leaves it unlocked, all the rules at the front desk quietly stop meaning anything.

Everything Kubernetes knows is a row in etcd: pods, config maps, the RBAC roles themselves, and Secrets. A Secret is just the object you use to hand a password or a token to a workload. Look at one through the API and the values come back scrambled, wrapped in base64, which is exactly why people assume they're safe. They aren't. Base64 isn't encryption. It's a reversible encoding, about as secret as writing a word backwards, and one short command turns it back into the original. By default the real password lands on etcd's disk in the clear. So read access to etcd, or to a single etcd backup file sitting in some storage bucket, is read access to every credential the cluster holds, all at once.

Reading a Secret straight out of etcd

To read etcd directly you first have to reach it, and in practice that means one of three things: a foothold on a control-plane node, a copy of an etcd snapshot, or an etcd port (2379) left listening without client-certificate authentication. Getting onto the node is the next lesson's problem. Say you're already there. The etcd client, etcdctl, plus the certificates that kubeadm (the standard tool that bootstraps a cluster) already dropped on the box are all you need. Before going under the hood, look at what the API server is willing to show you on its own.

create a Secret, look at it through the API
$ kubectl create secret generic db-creds \
    --from-literal=username=admin --from-literal=password='S3cr3t-P@ss'
secret/db-creds created
 
$ kubectl get secret db-creds -o jsonpath='{.data.password}'
UzNjcjN0LVBAc3M=
 
$ echo 'UzNjcjN0LVBAc3M=' | base64 -d
S3cr3t-P@ss

That base64 string reverses with a single command, so treat what the API returned as plaintext. Now drop below the API server and read the same Secret from etcd itself. Every object lives under a key that starts with /registry, so a Secret sits at /registry/secrets/<namespace>/<name>. Pipe the raw bytes through hexdump, a tool that prints a file as hex on the left and readable characters on the right, so the human-readable parts jump out.

etcdctl reads the Secret with no RBAC, no audit (BEFORE)
$ ETCDCTL_API=3 etcdctl \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/default/db-creds | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 64 62 2d 63 72 65  |s/default/db-cre|
00000020  64 73 0a 6b 38 73 00 0a  0c 0a 02 76 31 12 06 53  |ds.k8s.....v1..S|
000000d0  0a 08 70 61 73 73 77 6f  72 64 12 0b 53 33 63 72  |..password..S3cr|
000000e0  33 74 2d 50 40 73 73 1a  06 0a 04 61 64 6d 69 6e  |[email protected]|

Read the ASCII column down the right-hand side. The password is sitting right there in plaintext, and pulling it cost you no RBAC permission and left no entry in the Kubernetes audit log, because you never asked the API server for anything. That's the part that stings defenders. And in real incidents the attacker rarely needs a live control-plane node at all. etcd gets backed up on a schedule, those snapshots get copied off to object storage, and that storage is usually guarded far more loosely than the cluster it came from. One readable snapshot is the entire credential store in a single file. The cluster's own audit trail is blind to all of it, so your detection has to live somewhere the API server can't see.

Detecting the reach for etcd

You actually get two chances here. Most attackers who already hold a token won't bother with etcd at all. They'll just list Secrets through the API, and that path is audited. A single service account (SA, the identity a workload runs as) asking for every Secret in the cluster at once is a loud signal, the kind of thing you can alert on, like a build robot suddenly reading all the keys at three in the morning. For the raw etcd path, where the audit log can't help you, you instrument the host instead. Falco, a runtime sensor that watches Linux syscalls (the low-level requests a program makes to the operating system), fires the moment a process that has no business touching etcd's data directory or its private keys does exactly that.

detect the API precursor: a bulk list of Secrets
$ jq -c 'select(.verb=="list" and .objectRef.resource=="secrets" and .objectRef.namespace==null)' \
    /var/log/kubernetes/audit.log
{
  "verb": "list",
  "user": { "username": "system:serviceaccount:dev:ci-runner" },
  "objectRef": { "resource": "secrets", "apiVersion": "v1" },
  "requestURI": "/api/v1/secrets?limit=500",
  "responseStatus": { "code": 200 },
  "stageTimestamp": "2026-07-16T09:14:22Z"
}
detect the host path: Falco catches the etcd key read
$ falco -r /etc/falco/rules.d/etcd.yaml
09:15:02.114218 Warning Sensitive etcd file read by unexpected process
  (file=/etc/kubernetes/pki/etcd/server.key proc=etcdctl user=root
   parent=bash container=host command=etcdctl get /registry/secrets/)

Locking the cabinet: encryption at rest

So lock the cabinet. Kubernetes can encrypt Secrets before they ever reach etcd, so what lands on disk is ciphertext instead of your password. You do it by handing the API server an EncryptionConfiguration, a short file that lists encryption providers in order, and the order is the thing people get wrong. The first provider in the list encrypts new writes. On reads, every provider is tried from the top down until one works. Put identity, the 'store it as-is' no-op, first, and you've encrypted precisely nothing. Put a real provider first and identity last, and old plaintext rows still read back fine while every new write goes encrypted. For production you back the key with a Key Management Service (KMS, an external key vault) through the KMS v2 provider. That uses envelope encryption: each Secret is locked with a local data key, and that data key is itself locked by a master key that never leaves the vault. Steal the disk and all you've got is a locked box next to a locked key. The example below uses aescbc, which needs no external service and is plenty for watching the bytes on disk actually change.

/etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: dGhpcnR5LXR3by1ieXRlLWFlcy1jYmMta2V5LTIwMjY=
      - identity: {}
harden the API server: point it at the config, close anonymous auth
$ sudo grep -E 'anonymous-auth|authorization-mode|encryption-provider|etcd-(ca|cert|key)file|audit-log-path' \
    /etc/kubernetes/manifests/kube-apiserver.yaml
    - --anonymous-auth=false
    - --authorization-mode=Node,RBAC
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
    - --encryption-provider-config-automatic-reload=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --audit-log-path=/var/log/kubernetes/audit.log

The kubelet (the agent Kubernetes runs on every node) notices the changed manifest and restarts the API server pod on its own. Now the catch that bites almost everyone. Turning encryption on only affects Secrets written from this moment forward. Everything already in etcd stays exactly as plaintext as it was until you force a rewrite. So re-save every Secret to trigger a fresh encrypted write, then read the same key back out of etcd and check.

re-encrypt existing Secrets, then re-read etcd (AFTER)
$ kubectl get secrets --all-namespaces -o json | kubectl replace -f -
secret/db-creds replaced
 
$ ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/default/db-creds | hexdump -C | head -3
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000020  64 73 0a 6b 38 73 3a 65  6e 63 3a 61 65 73 63 62  |ds.k8s:enc:aescb|
00000030  63 3a 76 31 3a 6b 65 79  31 3a c2 4f 9a 1e 7b 55  |c:v1:key1:.O..{U|
Encryption at rest doesn't rewrite what's already there
Flipping on an EncryptionConfiguration only encrypts Secrets written after the API server reloads the config. Every Secret that existed before is still sitting in etcd in plaintext until you force a rewrite with kubectl get secrets -A -o json | kubectl replace -f -. And with aescbc the key lives in a file on the control-plane node, right next to etcd, so whoever can read etcd can usually read the key too. That's why production uses a KMS provider: the unlock key stays in an external vault, and a stolen etcd snapshot is just noise.
What a stolen etcd is worth depends on the provider
Attacker reads etcd or steals a backup
no RBAC check, no API audit event
no encryption (identity)
every Secret in plaintext
game over: all credentials exposed at once
aescbc / secretbox
ciphertext in etcd
but the unlock key sits on the same node
KMS v2 provider
ciphertext + external key
a stolen snapshot is useless on its own
Identity means the cabinet is unlocked. A local key means it's locked with the key taped to the door. KMS keeps the key in a different building.
Quick check
01You add an EncryptionConfiguration with aescbc, kube-apiserver restarts cleanly, and brand-new Secrets show a k8s:enc: prefix in etcd. But etcdctl get on a Secret you created last week still shows its password in plaintext. Why?
Correct — The API server encrypts on write, so pre-existing rows stay plaintext until you re-save them and trigger a re-encrypt.
Incorrect — No. No provider silently rewrites old rows. The gap is about write-time encryption, not which provider you chose.
Incorrect — If identity were first, the new Secrets wouldn't show a k8s:enc: prefix either. New writes are clearly encrypting, so ordering isn't the problem here.
Incorrect — No. etcdctl reads raw bytes straight from etcd and knows nothing about the encryption config; it can't decrypt anything.
02Reading a Secret with etcdctl on a control-plane node costs no RBAC permission and leaves nothing in the Kubernetes audit log. Why is that, and what closes the gap?
Incorrect — No. The lesson's own jq query catches a read: a bulk list of Secrets by a service account shows up with verb list and a 200. Reads through the API are audited; the etcd path isn't, for a different reason.
Incorrect — No. The cluster's audit trail is blind to a direct etcd read, which is exactly why the detection in this lesson lives on the host rather than waiting for a log from underneath the API server.
Correct — Both the badge check and the visitor log live at the front door. Go around the front door and you leave no trace there, so the detection has to sit where the API server can't see: syscalls on the node.
Incorrect — No. Audit policy governs what the API server records about requests it actually handles. No level makes a direct etcdctl read appear, because that request never arrives at the API server.
03You inherit a cluster. The grep on kube-apiserver.yaml shows --encryption-provider-config pointing at enc.yaml and the API server is running fine, but a Secret you just created reads back out of etcd through hexdump -C with its password sitting in the ASCII column and no k8s:enc: prefix anywhere. What do you do next?
Incorrect — No. hexdump only prints the bytes it is handed. When encryption is working, the same read comes back with a k8s:enc:aescbc:v1:key1: prefix followed by unreadable bytes.
Correct — The first provider in the list encrypts new writes, so identity at the top encrypts precisely nothing. Fix the order first, and only then force the rewrite.
Incorrect — No. That rewrite is the right move only once the config actually encrypts. While the no-op provider is taking writes, you would just re-save every Secret in the clear.
Incorrect — No. That flag only makes the API server reload the file when it changes. The config here has already loaded; the file itself is ordered so that nothing gets encrypted.

Try this

Work through “Locking the cabinet: encryption at rest” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: encryption at rest doesn't rewrite what's already there. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Webhook security & bypassesNode & kubelet security

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningKubernetes interview questionsInterview prep