>_SecOpsLog_
CoursesKubernetes administrationBackup & restore

Backup & restore

etcd snapshots and an offline restore.

Advanced12 min · lesson 60 of 65
In plain terms
Backing up etcd is photocopying the cluster’s only notebook and storing the copy off-site. Restoring rebuilds everything from that copy — and you only trust a copy you’ve actually practised restoring.

Lose etcd and you lose the whole cluster. The containers already running on your machines keep humming along for a while, but every record of what is supposed to exist is gone. etcd (you say it "et-cee-dee") is the cluster's key-value database, the one place Kubernetes writes down every fact about itself: which Deployments exist, every Secret, every RBAC (Role-Based Access Control) rule, every ConfigMap. It's the cluster's only logbook. A backup is a photocopy of that logbook. A restore hands Kubernetes a fresh copy and says, this is the truth now. Nodes, control-plane binaries, certificates, you can rebuild all of those from files and images you already have. What's written inside etcd you cannot rebuild from anything except a backup.

What a snapshot actually captures

Because etcd holds every object, a single snapshot is a complete point-in-time copy of the entire cluster. One file, everything in it. There's a big exception that catches people out, though. The snapshot does not include the data inside your persistent volumes. A persistent volume, or PV, is the real disk that a database Pod writes its rows onto (a Pod is the smallest thing Kubernetes runs, one or more containers sharing a single network address). etcd only stores the PV object, the little index card that says "this volume exists and mounts here." The rows themselves live on your storage backend and need their own separate backup. Restore etcd and every object definition comes back. The bytes on the disks do not.

terminal
$ sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key
terminal
{"level":"info","ts":"2026-07-16T09:12:01Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-07-16T09:12:01Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","took":"138ms"}
Snapshot saved at /backup/etcd-2026-07-16.db

That stack of certificate flags isn't decoration. etcd only talks to clients that can prove they're allowed in, so you hand it a certificate and key it already trusts, the ones kubeadm generated for etcd itself, and 127.0.0.1:2379 is etcd's client port on the control-plane node. Two habits separate a real backup from a decorative one. Copy the .db file off the node the moment it's written, because a snapshot that dies with the control-plane node saved you nothing. And encrypt it, because that file holds every Secret in the cluster, and unless you turned on encryption at rest they sit inside it in a form anyone can decode with one command. Don't do this by hand once and call it done, either. Put the snapshot on a schedule, keep the last several, and ship each one somewhere the cluster can't drag down with it. Before you trust any snapshot, read it back and confirm it holds roughly the number of keys you expect. etcdutl does that offline, without touching the live cluster. (etcdutl is the newer offline companion to etcdctl. Older guides run etcdctl snapshot status and restore, which still work but are now deprecated in favor of etcdutl.)

terminal
$ sudo etcdutl snapshot status --write-out=table /backup/etcd-2026-07-16.db
terminal
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 8d3f21ab |   248117 |       1974 |     6.4 MB |
+----------+----------+------------+------------+

Restoring is an offline operation

You can't rebuild an engine while it's spinning, and you can't restore etcd while etcd is live. Writing a restore over a running database corrupts it, so the procedure is deliberately offline. On a kubeadm cluster the steps are always the same. etcd and the API server (the front door every kubectl command and every controller talks to) both run as static Pods, which means the kubelet (the agent running on every node) starts them straight from manifest files in /etc/kubernetes/manifests and restarts them if they crash. Move those two manifests out of that folder and the kubelet stops the Pods. etcd is now down. Restore the snapshot into a brand-new data directory, point the etcd manifest at it, and move the manifests back. The kubelet spots them, starts etcd on the restored data, brings the API server up behind it, and the cluster wakes at the snapshot's moment. On a single control-plane node, that's the whole job. Run three stacked etcd members and you restore on every node separately, and the flags are not all the same. Each node gets its own --name and its own --initial-advertise-peer-urls, matching that member's identity, while --initial-cluster (the full list of all three members) and --initial-cluster-token have to be identical everywhere, or they refuse to form a cluster.

terminal
$ sudo mv /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml /tmp/   # stop both
$ sudo etcdutl snapshot restore /backup/etcd-2026-07-16.db \
    --data-dir=/var/lib/etcd-restore
$ sudo sed -i 's#/var/lib/etcd#/var/lib/etcd-restore#g' /tmp/etcd.yaml
$ sudo mv /tmp/{etcd,kube-apiserver}.yaml /etc/kubernetes/manifests/  # restart
terminal
2026-07-16T09:20:11Z	info	snapshot/v3_snapshot.go:260	restoring snapshot	{"path": "/backup/etcd-2026-07-16.db", "data-dir": "/var/lib/etcd-restore"}
2026-07-16T09:20:11Z	info	membership/cluster.go:421	added member	{"cluster-id": "cdf818194e3a8c32", "member-id": "8e9e05c52164694d"}
2026-07-16T09:20:11Z	info	snapshot/v3_snapshot.go:287	restored snapshot	{"path": "/backup/etcd-2026-07-16.db"}

A restore is a time machine, and it moves the whole cluster, not just the part you were trying to fix. Anything created after the snapshot is gone. Anything deleted after it comes back. So check the result: the etcd Pod should be Running, the nodes Ready, your workloads sitting where they stood when the snapshot ran. For the gaps etcd can't fill, the PV data and pulling back one deleted namespace, many teams add Velero, a tool that copies Kubernetes objects (and can snapshot volumes) to object storage and restores them one namespace at a time. That last part matters more than it sounds. An etcd restore is all-or-nothing across the cluster, so if all you lost was a single namespace someone deleted by mistake, rewinding every other change since last night just to get it back is a rotten trade. Velero lets you pull only the thing you actually lost.

terminal
$ kubectl get nodes
$ kubectl get pods -n kube-system -l component=etcd
terminal
NAME     STATUS   ROLES           AGE   VERSION
cp-1     Ready    control-plane   90d   v1.31.4
node-1   Ready    <none>          90d   v1.31.4
node-2   Ready    <none>          90d   v1.31.4
 
NAME        READY   STATUS    RESTARTS   AGE
etcd-cp-1   1/1     Running   0          41s

One more thing worth knowing before you build any of this. If your cluster runs on a managed Kubernetes service (Amazon EKS, Google GKE, Azure AKS), you never touch etcd at all. The provider runs the control plane for you and won't hand you the keys, so the whole snapshot-and-restore routine simply doesn't apply. On those clusters an object backup with Velero is the main lever you've got, plus whatever volume snapshots your storage class supports. Knowing which kind of cluster you're on tells you which half of this page is even yours to run.

The step almost everyone skips is proving the restore works before they need it. A snapshot file that has never been restored is a guess, not a backup. Spin up a throwaway cluster, feed it last night's snapshot, and watch what actually comes back. The first time you run these commands should not be at 3 a.m. with production down and your pulse in your ears. Rehearse it when the stakes are zero, write down the exact order of steps, and the real event turns into a script you follow instead of a problem you solve.

Three layers of a real backup strategy
etcd snapshot: control-plane state
every object
Deployments, Secrets, RBAC, ConfigMaps
one .db file
point-in-time, complete
off-cluster + encrypted
it holds every Secret
PV backup: the actual data
disk bytes
database rows, files on volumes
Velero or storage backend
snapshots to object storage
etcd does NOT cover this
a separate job
tested restore: the part people skip
offline procedure
stop, restore, repoint, restart
rehearsed on a throwaway cluster
not during the outage
The two backups cover different things: the etcd snapshot brings back object definitions, the PV backup brings back the data those objects point at. Neither is real until you've run the restore for practice.
The etcd manifest names /var/lib/etcd in three places
When you restore into a new directory, the etcd static-Pod manifest still mentions the old path three times: the --data-dir flag, the volumeMounts path that says where the directory shows up inside the container, and the hostPath volume that says which directory on the node to mount there. Repoint --data-dir but leave the volumeMounts path alone and etcd writes to a path with nothing mounted on it, so its data lands in the container's throwaway layer and kubectl comes back showing a cluster that looks freshly installed. Leave the hostPath alone instead and etcd starts on the old, pre-restore directory. Either way your restore appears to have done nothing. Change all three (the global sed above does that), or clear /var/lib/etcd and restore straight into it.

Volume backups are separate from etcd. App data on PVCs needs its own story.

Test restore in a sibling environment. Untested backups are rumors.

Protect snapshot files like Secrets. They contain everything.

Try this

Take an etcd snapshot on a lab control plane, note the file size, and write down the restore steps without executing a destructive restore unless you are in a throwaway cluster.

terminal
$ sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key
 
$ sudo mv /etc/kubernetes/manifests/{etcd,kube-apiserver}.yaml /tmp/   # stop both
$ sudo etcdutl snapshot restore /backup/etcd-2026-07-16.db \
    --data-dir=/var/lib/etcd-restore
$ sudo sed -i 's#/var/lib/etcd#/var/lib/etcd-restore#g' /tmp/etcd.yaml
$ sudo mv /tmp/{etcd,kube-apiserver}.yaml /etc/kubernetes/manifests/  # restart
 
$ kubectl get nodes
$ kubectl get pods -n kube-system -l component=etcd

Takeaway

etcd snapshots capture cluster state. Restore is offline: new data dir, repointed static pod, API back last.

Quick check
01You restore last night's etcd snapshot. A namespace you created this morning is now gone, and a namespace you deleted at noon yesterday has reappeared. What does this tell you?
Correct — etcd holds all object state, so restoring it is a full point-in-time rollback. Everything after the snapshot is undone, in both directions.
Incorrect — Nothing is corrupt. This is exactly how a point-in-time restore behaves, and skip-hash-check would not change it.
Incorrect — A snapshot captures every object in the cluster, not just namespaces.
Incorrect — Backwards. A snapshot restores object definitions and never the data inside persistent volumes.
02An etcd snapshot is a complete point-in-time copy of the cluster's objects. What does it NOT contain?
Incorrect — RBAC rules, ConfigMaps, and Secrets are all objects stored in etcd, so they are captured.
Incorrect — those object definitions are exactly what the snapshot captures; they come straight back on restore.
Correct — the bytes on your storage backend need their own separate backup (Velero or volume snapshots); etcd keeps only the index card.
Incorrect — it captures objects but not persistent-volume contents, which is the classic gap that catches people out.
03You restore a snapshot into /var/lib/etcd-restore and edit the etcd static-Pod manifest's --data-dir to point there, then move the manifests back. kubectl comes up showing a cluster that looks freshly installed, as if the restore did nothing. What went wrong?
Correct — The volumeMounts path and the hostPath still say /var/lib/etcd, so etcd's writes land in the container's throwaway layer and it comes up empty.
Incorrect — a freshly-installed look points to the wrong data directory, not corruption, and skip-hash-check would not change it.
Incorrect — restoring over a live etcd corrupts it; the procedure is deliberately offline, and that is not the cause here.
Incorrect — the symptom is etcd serving an empty data directory, not a client-side context problem.
Upgrading a clusterA troubleshooting method

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep