StatefulSets & stable storage

Stable identity and per-pod persistent volumes.

Scale a Deployment from three pods to five and Kubernetes doesn't care which two it adds, what they're named, or where they land. That indifference is the whole point of a Deployment, and it's exactly wrong for a database. Postgres db-0 has rows on its disk that db-1 has never seen. Kill db-0, get back a fresh pod with a new name and an empty volume, and you haven't replaced a replica, you've dropped one. StatefulSets are for the workloads where pods are not interchangeable and losing the wrong one costs you data.

Think about seating. A Deployment is a food court: any open table works, and the moment you leave, the table gets wiped for the next person. A StatefulSet is assigned seating with a name card on the chair and a locker behind it that only your key opens. Step away, come back, same chair, same locker, same contents. Kubernetes gives each pod in a StatefulSet three things that stay glued to it for its whole life: a stable name, a stable network address, and its own disk.

The three things that stay stuck to a pod

First, the name. Pods in a StatefulSet are numbered from zero (db-0, db-1, db-2), and that number is the pod's identity, its ordinal. Reschedule db-1 onto a different node and it comes back as db-1, not some random suffix. Second, the network address. An ordinary Service spreads traffic across interchangeable pods, but a StatefulSet fronts itself with a headless Service (a Service with clusterIP set to None, so DNS, the Domain Name System that turns a name into an address, hands back the pods' own addresses instead of one shared virtual IP address). Treat it as a signpost, not a receptionist: it routes nothing, it just publishes one DNS name per pod, so db-0.db resolves straight to db-0. Third, the disk. A volumeClaimTemplate tells the controller to stamp out one PersistentVolumeClaim (a PVC is just a request for a chunk of disk) per pod: data-db-0, data-db-1, data-db-2. db-0 always reattaches to data-db-0. The storage follows the name.

apiVersion: v1
kind: Service
metadata:
  name: db
  labels: { app: db }
spec:
  clusterIP: None        # headless: one DNS record per pod, no load balancing
  selector: { app: db }
  ports:
  - { name: pg, port: 5432 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db        # must name the headless Service above
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
      - name: postgres
        image: postgres:16
        ports: [{ containerPort: 5432 }]
        volumeMounts:
        - { name: data, mountPath: /var/lib/postgresql/data }
  persistentVolumeClaimRetentionPolicy:   # stable since v1.32
    whenScaled: Retain
    whenDeleted: Retain
  volumeClaimTemplates:
  - metadata: { name: data }
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: standard
      resources:
        requests: { storage: 20Gi }

$ kubectl apply -f db.yaml

service/db created
statefulset.apps/db created

Here's what the controller actually does with that, and it's stricter than a Deployment. The StatefulSet controller brings pods up one at a time, lowest number first. It creates db-0, waits until db-0 is both Running and Ready (its readiness probe passing), and only then starts db-1. Scale down and it runs the same logic in reverse, highest number first, one pod at a time. That's the default OrderedReady policy, and it exists because clustered software often has to bootstrap in sequence: the first member forms the cluster, the rest join it. If your app doesn't care about order, set podManagementPolicy to Parallel and they all come up at once. Under each pod, the scheduler picks a node, and before the container can start, the pod's volume has to be attached to that node and mounted. The kubelet (the agent running on each node) is what holds the container back until that storage is in place.

$ kubectl get statefulset db
$ kubectl get pods -l app=db

NAME   READY   AGE
db     3/3     3m
 
NAME   READY   STATUS    RESTARTS   AGE
db-0   1/1     Running   0          3m
db-1   1/1     Running   0          2m31s
db-2   1/1     Running   0          2m3s

Look at the ages. db-0 is the oldest, db-2 the youngest, which is the ordered rollout showing itself in the timestamps. Each pod also has its own PVC, and they're bound to real volumes.

$ kubectl get pvc -l app=db

NAME        STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-db-0   Bound    pvc-3f9c1a2b   20Gi       RWO            standard       3m
data-db-1   Bound    pvc-9a7d4e01   20Gi       RWO            standard       2m31s
data-db-2   Bound    pvc-c1b820ff   20Gi       RWO            standard       2m3s

Two safety defaults are baked in here. Scale db back down to 1 and Kubernetes deletes db-2 and db-1 but keeps data-db-2 and data-db-1 sitting there. Delete the whole StatefulSet and it keeps every PVC too. Your data is meant to outlive the pods, so a recreated StatefulSet reattaches to the volumes that were already there. That field in the manifest, persistentVolumeClaimRetentionPolicy (stable since v1.32), is how you opt out: flip whenScaled or whenDeleted to Delete and the volumes get cleaned up automatically. Leave it on Retain and you accept that orphaned PVCs pile up and keep billing until someone deletes them by hand.

You can watch the stable network identity resolve. Spin up a throwaway pod and look db-0 up by name. This is the whole reason the headless Service exists: other apps can point at db-0.db as the primary and treat db-1.db and db-2.db as replicas, and the name keeps pointing at the same pod with the same data even after that pod moves to a new node and picks up a new IP address.

$ kubectl run dns --rm -it --restart=Never --image=busybox:1.36 -- nslookup db-0.db

Server:    10.96.0.10
Address:   10.96.0.10:53
 
Name:      db-0.db.default.svc.cluster.local
Address:   10.244.1.7
pod "dns" deleted

When the rollout won't move

The ordered rollout is a gift right up until it wedges. Because db-1 is never created until db-0 is Running and Ready, a single stuck pod-0 freezes the entire StatefulSet. The most common cause is storage: if db-0 can't get its volume, it sits in Pending and db-1 and db-2 never even get created. So when a StatefulSet is stalled at one pod, look at the PVC before you look at the pod.

$ kubectl get pods -l app=db

NAME   READY   STATUS    RESTARTS   AGE
db-0   0/1     Pending   0          90s

$ kubectl describe pod db-0 | grep -A4 Events:
$ kubectl get pvc data-db-0

Events:
  Type     Reason            Age   From                Message
  ----     ------            ----  ----                -------
  Warning  FailedScheduling  80s   default-scheduler   0/3 nodes are available: pod has unbound
                                                        immediate PersistentVolumeClaims.
 
NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-db-0   Pending                                      standard       90s

That output tells the whole story. The pod is unschedulable because its PVC is unbound, and the PVC is Pending because nothing satisfied the claim, usually a StorageClass named wrong, a StorageClass with no dynamic provisioner behind it, or a cluster with no default StorageClass at all. Fix the StorageClass, the PVC binds, db-0 goes Ready, and the controller finally moves on to db-1. The same ordered logic governs updates: change the image and the controller replaces pods from the highest number down, one at a time, waiting for each to go Ready. A bad image takes out db-2 first, which gives you a window to catch it before it ever reaches db-0.

Before you scale someone else's StatefulSet down, check what that will do to the disks. Keeping the PVCs is still the default today, not some legacy behavior that has since been reversed, and the field that governs scale-down is whenScaled, not whenDeleted. Read the one you are actually about to trigger.

$ kubectl get statefulset db -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}'

{"whenDeleted":"Retain","whenScaled":"Retain"}

Rolling updates have a brake you can pull as well. Set updateStrategy.rollingUpdate.partition to 2 and the controller only replaces pods whose ordinal is 2 or higher, so db-2 takes the new image while db-1 and db-0 stay exactly where they are. Watch db-2 for as long as you want, then lower the partition one number at a time to let the change reach the rest. Set updateStrategy.type to OnDelete and the controller updates nothing at all until you delete a pod yourself, which is what people reach for when each ordinal needs a human to run a migration first.

Try this

Deploy a simple StatefulSet with a volumeClaimTemplate, watch ordinal pods and stable names appear, then delete one pod and confirm identity returns.

$ kubectl apply -f db.yaml
 
$ kubectl get statefulset db
$ kubectl get pods -l app=db
 
$ kubectl get pvc -l app=db
 
$ kubectl run dns --rm -it --restart=Never --image=busybox:1.36 -- nslookup db-0.db
 
$ kubectl get pods -l app=db
 
$ kubectl describe pod db-0 | grep -A4 Events:
$ kubectl get pvc data-db-0

Takeaway

StatefulSets give stable network identity and per-pod PVCs. Ordinals and headless Services are part of the contract.

Related