>_SecOpsLog_
CoursesKubernetes administrationetcd — the cluster store

etcd — the cluster store

The single source of truth, and why it is guarded.

Intermediate10 min · lesson 4 of 65
In plain terms
etcd is the cluster’s notebook — it writes down everything the cluster is supposed to be. Lose the notebook with no photocopy and you’ve lost the entire plan, which is why you guard it and back it up.

Every object in your cluster lives in exactly one place, and it isn't on the worker machines that run your apps. It sits in a single database on the control plane (the cluster's management brain) called etcd (say it 'et-cee-dee'). etcd is a key-value store, which is the same shape as a giant dictionary that maps a name to a blob of data. Think of a warehouse's master inventory ledger. The shelves don't decide what's in stock. The ledger does, and staff keep restocking until the shelves match what's written down. Kubernetes runs the same loop. You write what you want into etcd, and controllers keep nudging the real cluster until it matches.

One rule shapes everything else. Only the API server (the front door that every kubectl command and every controller talks to) is allowed to read or write etcd. Your Pods (the smallest deployable unit, one or more containers that run together) never touch it. The kubelet (the agent on each node that actually starts your containers) never touches it either. They all go through the API server, which is the single writer standing in front of the store. That's on purpose. One doorway gives you a single place to check who's calling and confirm an object is valid before it's ever stored. It's also the one place to log what happened. And etcd only ever has to trust one client instead of thousands.

How a write actually lands

Say you run three copies of etcd for safety. When you create a Deployment (a request to run some copies of your app), that change can't just be scribbled into one copy and called done, because that copy could die a second later and take the only record with it. Think of a small committee that refuses to act until more than half its members have signed off. etcd works the same way, through a voting protocol called Raft. One member is elected leader, the write goes to the leader, the leader ships it to the followers, and the change counts as committed only once a majority have written it to their own disk. Majority is the word to hold onto. With three members, two is a majority, so the cluster keeps serving even if one dies.

A write's path through etcd
1kubectl applyyou declare desired state2API serverthe only client etcd trusts3etcd leaderreceives the write4followersreplicate to a majority5committedon a quorum of disks = durable
Only a majority write makes it real. With 3 members, 2 is a majority, so one node can die and the change still stands.

This is why you run an odd number of members. Three tolerates one failure. Five tolerates two. Four sounds safer than three but isn't: a majority of four is still three, so you survive only one failure, and now you have an extra machine that can break. There's a nastier case too. If a network split drops two members on each side of a four-node cluster, neither side holds a majority, and the whole store goes read-only until the split heals. So the answer is almost always three, occasionally five for bigger or more critical clusters. Going past five mostly just slows writes down, since every commit now waits on more disks.

Reading the store, and why Secrets scare people

Everything etcd holds is filed under a key that starts with /registry. A Pod lands at /registry/pods/<namespace>/<name>, a Secret at /registry/secrets/<namespace>/<name>, and the value is the object serialized into a compact binary format (protobuf). If you're on a control-plane node with the etcd certificates, you can read a key straight from the store. Here's the part that catches people out. A Secret's value sits in etcd as good as plaintext. The base64 you see in kubectl get secret -o yaml is just an encoding for moving bytes around, not protection. Anyone who can read etcd's data files, or an old snapshot, can read every password in the cluster.

terminal
sudo ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-cred \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | strings
output
/registry/secrets/default/db-cred
k8s
v1Secret
db-creddefault"*$9d1c7f0e-...z
Opaque
password
S3cr3t-P@ssw0rd

The fix is encryption at rest. You give the API server an EncryptionConfiguration that tells it to encrypt certain resource types (Secrets first) before they ever reach etcd, ideally with keys held in a KMS (Key Management Service, an external vault the API server calls so the real key never lives on the node). After that, the same etcdctl get returns ciphertext instead of your password. Lock the network path down as well: mutual TLS between the API server and etcd (both ends prove who they are with certificates), and a firewall so nothing but the API server can reach port 2379.

A snapshot is the whole cluster

Because every bit of state lives here, one etcd snapshot is a complete point-in-time backup of the control plane. Not just your Deployments. All of it, down to the last RBAC (Role-Based Access Control) rule and Secret. Before you back anything up, check the store is healthy and that you're pointed at the right endpoint.

terminal
sudo ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
output
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://127.0.0.1:2379 | 8e9e05c52164694d |  3.5.15 |   31 MB |      true |      false |         4 |     158204 |             158204 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

One leader, a healthy database size, an empty errors column. Now take the snapshot itself. The command streams the live database out to a single file you can copy anywhere.

terminal
sudo ETCDCTL_API=3 etcdctl snapshot save /var/backups/etcd-2026-07-16.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
output
{"level":"info","ts":"2026-07-16T09:12:03.441Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/backups/etcd-2026-07-16.db.part"}
{"level":"info","ts":"2026-07-16T09:12:03.658Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-07-16T09:12:03.871Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","total-bytes":32505856,"took":"0.213s"}
Snapshot saved at /var/backups/etcd-2026-07-16.db

Restoring is a disaster-recovery move, and it happens offline. You stop the API server and etcd, restore the snapshot into a fresh data directory with etcdutl snapshot restore (current etcd moved restore out of etcdctl into the etcdutl helper), point etcd's static Pod manifest at that new directory, and start everything back up. Rehearse it on a throwaway cluster before you ever need it for real, because a backup you have never restored is a wish, not a plan. And store snapshots off the cluster and encrypted, since each one carries every Secret you own.

When etcd runs out of room

etcd keeps a history. Every change bumps a global revision number, and old versions stick around, which is what makes watches and rollbacks work. That history is already being pruned for you: the API server compacts it every five minutes out of the box. What compaction never does is shrink the file. The pages it frees stay inside the database to be reused, so heavy churn (a flood of Events is the usual culprit), a few very large objects, or plain fragmentation can still walk the file up to its size limit (the default backend quota is about 2 GiB). At that point etcd raises a NOSPACE alarm and flips the whole store to read-only. Your cluster stops accepting changes, and kubectl apply comes back with 'etcdserver: mvcc: database space exceeded', even though every node still looks perfectly fine. Clearing it takes two steps. First, compact by hand so no old revision is still holding pages. Then defragment, because that is what hands the freed room back to the filesystem.

terminal
export ETCDCTL_API=3
E="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"
 
etcdctl alarm list $E
rev=$(etcdctl endpoint status --write-out=json $E | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
etcdctl compact "$rev" $E
etcdctl defrag $E
etcdctl alarm disarm $E
output
memberID:10276657743932975437 alarm:NOSPACE
compacted revision 158204
Finished defragmenting etcd member[https://127.0.0.1:2379]
A full etcd freezes the entire cluster
The trap at 3 a.m. is assuming nobody ever switched compaction on. Somebody did: kube-apiserver ships with --etcd-compaction-interval set to 5m, and etcd can prune on its own with --auto-compaction-retention. So a default cluster that fills up is telling you something else is going on, usually fragmentation, a flood of Events, or a handful of oversized objects. Watch the DB SIZE column and defrag on a schedule, one member at a time inside a maintenance window, because defrag blocks the member it is running on. And whatever you clear, the alarm stays armed until you disarm it by hand.

Quorum matters more than raw disk speed. A majority of members must agree or the cluster refuses writes. Two of three dead is a read-only disaster waiting for the next change.

Encryption at rest for Secrets is configured at the API server, but the ciphertext still lives in etcd. Snapshot files are as sensitive as the live database.

Restore is offline work: stop the static pods, restore to a fresh data dir, repoint the manifest, bring the API back. Practice it before you need it.

Try this

If you have a kubeadm control-plane box, list a few keys under /registry with etcdctl using the local certs, then ask the store how it is doing. See that Secrets and Deployments are just keys in one place. Everything below is a read, and on a box you care about it should stay that way: compaction, defrag and snapshot save are maintenance work, and defrag blocks the member while it runs, so on a single-member control plane the API server stalls until it finishes.

terminal
$ sudo ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-cred \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | strings
 
$ sudo ETCDCTL_API=3 etcdctl get /registry/deployments/ --prefix --keys-only \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
$ sudo ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
 
$ sudo ETCDCTL_API=3 etcdctl alarm list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Takeaway

etcd is the source of truth. Back it up, protect its certs, and never treat a casual etcdctl write as a substitute for the API.

Quick check
01You run a 3-member etcd cluster. During maintenance, two members go offline at the same time. What happens to writes?
Incorrect — One member out of three is not a majority, so it cannot commit anything by itself.
Correct — Two of three is the majority; with only one member up there is no quorum, so nothing commits and the control plane freezes for changes.
Incorrect — etcd members are fixed control-plane peers. Worker nodes never run etcd, so nothing can be promoted there.
Incorrect — Writes need a majority to commit; a single member out of three can never form one.
02A colleague argues Secrets are safe because kubectl get secret -o yaml shows the value as base64. Why is that wrong, and what actually protects a Secret at rest?
Incorrect — base64 is an encoding for moving bytes around, not a cipher; anyone can decode it instantly.
Incorrect — Pods never touch etcd at all; only the API server does, and that fact encrypts nothing.
Incorrect — ConfigMaps are also stored in plaintext in etcd and are no more protected than a Secret.
Correct — the lesson shows the raw password readable straight from etcd and points to encryption at rest with a KMS-held key as the real fix.
03kubectl apply fails with etcdserver: mvcc: database space exceeded and no new objects can be created, yet every node reports Ready. etcdctl alarm list returns a NOSPACE alarm. What is the correct sequence to recover?
Correct — compaction frees space inside the file, defrag shrinks the file itself, and the NOSPACE alarm stays armed until you disarm it by hand.
Incorrect — a restart does not reclaim space or clear the alarm, which must be disarmed after compact and defrag.
Incorrect — the limit is etcd's on-disk backend quota, not node memory or Pods; deleting Pods does nothing for it.
Incorrect — new members just replicate the same data and don't clear a full store; you must compact and defrag the existing one.
The API serverControllers & reconciliation

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep