Persistent storage

Lockers that outlive the pod.

Gym lockers run on a simple deal. You ask at the desk for a locker, you get one, and your gear stays inside after you shower, change, and go home. Kubernetes copies that deal for files. A program asks for storage, gets some, and its data stays put long after the program itself has been shut down and started fresh. That is persistent storage, and it is the piece that lets Kubernetes run real databases.

So why does this need a whole mechanism? Your app runs inside a Pod. A Pod is the smallest unit Kubernetes runs: one or more containers (running copies of your app, packaged with everything they need) that share a network address and start and stop as a group. Pods are built to be thrown away. Kubernetes deletes and recreates them constantly: when you ship a new version, when a server reboots, when it shuffles work onto a quieter machine. The whole set of machines Kubernetes manages is called the cluster, and one worker machine inside it is a node. A Pod can be moved from one node to another at any moment. Every time a Pod comes back, its container filesystem (the files and folders inside it) starts empty. For a stateless web page, one that remembers nothing between visits, that costs you nothing. For anything that has to remember, wiping the filesystem on every restart is a disaster.

You ask, the cluster provides

Kubernetes splits storage into two halves so you never have to think about a physical disk. You write a PersistentVolumeClaim (PVC for short, which is literally a claim ticket for storage). That is your request slip: I need 1 gigabyte. The cluster then finds or creates a PersistentVolume (PV, the real storage behind the ticket): a cloud disk, a network share, something with actual bytes on it. Your Pod points at the claim by name and never at a particular disk. That keeps the Pod portable, and it lets the storage exist on its own, with a life separate from any Pod. Delete the Pod, recreate it, and it reattaches to the same claim and finds its files exactly where it left them. The locker stayed locked while the person using it got swapped out. On most clusters none of this needs setup from you: create the claim, and the cluster builds a matching disk to fit.

Putting it on paper

Here is what that looks like written down. Kubernetes configuration lives in YAML, a plain-text format that stores settings as indented lines of key and value. Save the following as data.yaml. It creates two things in one go: the claim, and a small Pod that mounts it (mounting means attaching the storage so it shows up as a folder inside the app).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteOnce]        # one node mounts it read-write at a time
  resources:
    requests:
      storage: 1Gi                    # "I need 1Gi"
---
apiVersion: v1
kind: Pod
metadata:
  name: keeper
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data            # the claim appears here inside the container
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data               # the Pod names the CLAIM, not a disk

Two lines deserve a plain-English translation. accessModes: [ReadWriteOnce] means one node can mount this storage for reading and writing at a time, which is the normal choice for a single database. The --- line separates two YAML documents, so this one file defines both the claim and the Pod. Now look at the Pod again. It names the claim with claimName: data and mentions no disk anywhere. That is the whole idea in one line. The Pod asks for the thing called data and leaves the physical storage to the cluster. Inside the container, that storage shows up at the path /data.

Run it

You talk to a Kubernetes cluster through a command-line tool called kubectl. Apply the file, then check on the claim. You need a running cluster for this. A small one on your own laptop is fine, from minikube or kind (both free tools that spin up a single-machine Kubernetes cluster for practice).

$ kubectl apply -f data.yaml
persistentvolumeclaim/data created
pod/keeper created
 
$ kubectl get pvc
NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data   Bound    pvc-6f3c1e2a-9b40-4d7e-8a11-2c5f9e7d1b03   1Gi        RWO            standard       8s

Bound is the word to watch. It means your claim got matched to a real PersistentVolume and the storage is ready to use. That random-looking VOLUME name is the disk the cluster made for you, and STORAGECLASS is the recipe it followed to make it. You asked for 1Gi, you got 1Gi, and you never touched a disk. Claims do not always reach Bound, though, and the way they get stuck is worth seeing on purpose.

The claim that never binds

Sometimes a claim sits on Pending and stays there. Pending means the cluster has not found or made a disk that satisfies the request. The most common beginner cause is a StorageClass mismatch: you asked for a class the cluster does not have, maybe a value copied out of another tutorial, so there is no recipe for making the disk. Nothing fails loudly. The claim waits. When something in Kubernetes is stuck rather than broken, the move is always the same: describe it and read the Events at the bottom.

$ kubectl get pvc
NAME   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data   Pending                                      fast-ssd       30s
 
$ kubectl describe pvc data
Name:          data
Namespace:     default
StorageClass:  fast-ssd
Status:        Pending
Volume:
Used By:       keeper
Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  8s (x3 over 30s)  persistentvolume-controller  storageclass.storage.k8s.io "fast-ssd" not found
 
$ kubectl get storageclass
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  40m

The event says exactly what went wrong: the class named fast-ssd does not exist here. The last command lists the classes this cluster actually has, so the fix is to ask for one of those (standard, in this case) or to drop storageClassName altogether and let the default take over. There is a knock-on effect worth catching. While the claim is Pending, the keeper Pod cannot start either. A Pod only begins running once every volume it mounts is ready, so a stuck claim quietly holds your app hostage. Check the keeper Pod while this is going on and you will find it sitting in Pending too, waiting on the claim that cannot bind.

Test the promise

Talk is cheap, so run the experiment on a healthy Bound claim. Write a file into the mounted storage, destroy the Pod outright, recreate it from the same file, and read the file back.

$ kubectl exec keeper -- sh -c 'echo "hello from the first pod" | tee /data/note.txt'
hello from the first pod
 
$ kubectl delete pod keeper
pod "keeper" deleted
 
$ kubectl apply -f data.yaml
persistentvolumeclaim/data unchanged
pod/keeper created
 
$ kubectl exec keeper -- cat /data/note.txt
hello from the first pod

Read that last line twice. A brand-new Pod, created after the old one was deleted, opened the exact file the first Pod wrote. The claim itself was never touched (kubectl even reports persistentvolumeclaim/data unchanged); only the Pod got replaced. That is persistent storage doing its one job, and every database running on Kubernetes leans on that behavior.

What this makes possible

This is the feature that lets Kubernetes run stateful apps, the ones that have to remember things: databases, message queues, anything with saved state. A database Pod can be restarted, upgraded, or moved to a different node, and its data rides along in the PersistentVolume instead of dying with the Pod. When you need several copies of such an app, each with its own private storage, Kubernetes has a built-in tool for exactly that, called a StatefulSet, which you will meet in the administration course. It gives every Pod its own stable claim. The PVC is the foundation it stands on.

So here is the shape of it. A PersistentVolumeClaim asks the cluster for durable storage, and a provisioner (the component that creates disks on demand) builds a PersistentVolume that outlives any single Pod. Reach for a PVC whenever you would want the data back after losing a node, which covers every database. The alternative, emptyDir, is scratch space that dies with the Pod, so do not bet real data on it.

Two settings steer what you actually get. accessModes decides between RWO (ReadWriteOnce, one node mounts it for reading and writing) and RWX (ReadWriteMany, several nodes mount it at once), and the storage class decides which backend serves the disk. When a PVC sits on Pending, the cause is nearly always one of three: no provisioner running, a class name that does not exist, or not enough capacity left. Describe the PVC and read its events before you change anything else.

There is a cost side to weigh. Durable disks are billed by the gigabyte, and they tie your Pod to one zone, which makes moving workloads around harder. Plenty of apps run better as throwaway Pods that talk to a managed database somewhere else. A PVC is a decision, not a default. Ask whether this app is the one actually holding the data before you attach a disk to it.

Your turn

Run the full loop yourself on a practice cluster. Create a claim, mount it in a Pod, write a file, delete the Pod, bring up a fresh Pod on the same claim, and read the file back. Clean up at the end so the disk does not sit around costing you.

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: note-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
EOF
persistentvolumeclaim/note-pvc created
 
$ kubectl get pvc note-pvc
NAME       STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
note-pvc   Bound    pvc-…    1Gi        RWO            standard       5s
 
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: note
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: note-pvc
EOF
pod/note created
 
$ kubectl exec note -- sh -c 'echo keep-me > /data/note; cat /data/note'
keep-me
$ kubectl delete pod note
pod "note" deleted
 
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: note
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: note-pvc
EOF
$ kubectl exec note -- cat /data/note
keep-me
 
$ kubectl delete pod note; kubectl delete pvc note-pvc
pod "note" deleted
persistentvolumeclaim "note-pvc" deleted

Takeaway

Three things to carry out of this lesson. A claim is only real once it reads Bound. A Pod names the claim and never a disk, which is what keeps it portable. And the reclaim policy on the storage class decides whether deleting a claim also deletes the disk behind it, so check that policy before you type kubectl delete pvc against anything holding production data.

Related