Volumes: keeping files around

Why container storage vanishes.

A container's writable layer is thrown away when the container is replaced. Files you wrote there do not come back. A container is a running copy of the app. A Pod is the wrapper Kubernetes uses to place one or more containers on a node. This lesson is how a Pod keeps files across a container restart — and what still dies when the Pod dies.

Why a container forgets

Every container starts from an image: a read-only snapshot of the app and its files, frozen at build time. The container can read those files but cannot edit the image. On top it gets a writable layer of its own. That layer belongs to that container only. When the container crashes or you roll a new image, the kubelet (the agent on every node) deletes the old container and starts a new one from the image. The writable layer goes with it. Fine for a stateless web server. A problem for anything that must remember files.

Don't take that on faith. Watch it happen. Run one container with no volume, save a file inside it, then delete the Pod and start it again from scratch.

kubectl run scratch --image=busybox:1.36 -- sleep 3600
kubectl exec scratch -- sh -c 'echo "keep me" > /tmp/note.txt'
kubectl delete pod scratch
kubectl run scratch --image=busybox:1.36 -- sleep 3600
kubectl exec scratch -- cat /tmp/note.txt

pod/scratch created
pod "scratch" deleted
pod/scratch created
cat: can't open '/tmp/note.txt': No such file or directory
command terminated with exit code 1

The last line carries the whole lesson. cat cannot find the file, and the shell hands back exit code 1 (an exit code is the number a program leaves behind to report how it ended, where 0 means fine and anything else means trouble). Deleting the Pod threw the container away, writable layer and all, and the replacement booted clean from the image. A crash does the same thing on its own, without you lifting a finger.

kubectl run boom --image=busybox:1.36 -- sh -c 'echo starting; exit 1'
kubectl get pod boom

pod/boom created
NAME   READY   STATUS             RESTARTS      AGE
boom   0/1     CrashLoopBackOff   4 (38s ago)   2m

boom's program exits the instant it starts, so the kubelet keeps binning the dead container and booting a fresh one, waiting a little longer between each attempt. That parked state is called CrashLoopBackOff, one of the first failures you will meet in a real cluster, and the RESTARTS column counts the swaps. Every swap is what wipes the writable layer. To find out why it died, run kubectl logs boom --previous (the logs of the container that ran before the current one) or kubectl describe pod boom. A container's own disk is a dead end for anything you need to keep. The fix is to put the file somewhere outside the container.

A volume clips onto the Pod

A volume is storage attached to the Pod, not to one container. A container mounts it at a folder inside itself. Mounting means the storage shows up as an ordinary folder; the app reads and writes there. Because the volume belongs to the Pod, it survives a container restart. Two containers in the same Pod can mount the same volume and share files. The plainest kind is emptyDir: created empty when the Pod starts, gone when the Pod is gone. Good for scratch, and for passing files to a helper container in the same Pod (a sidecar). Here is a Pod that shares one emptyDir between two containers. It is YAML; copy the spacing exactly.

apiVersion: v1
kind: Pod
metadata:
  name: notes
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /data
    - name: sidecar
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /shared
  volumes:
    - name: scratch
      emptyDir: {}

Read it top to bottom. The Pod runs two containers, app and sidecar. Under each one, volumeMounts says which volume that container wants and where to attach it. Both point at the same volume, named scratch, but each mounts it at a different folder: app sees it at /data, sidecar sees it at /shared. Near the bottom, the volumes section defines scratch once as an emptyDir with empty braces. Those braces mean give me a blank scratch volume with the default settings. One volume, two windows onto the same files.

kubectl apply -f shared-scratch.yaml
kubectl exec notes -c app -- sh -c 'echo "hello from app" > /data/note.txt'
kubectl exec notes -c sidecar -- cat /shared/note.txt

pod/notes created
hello from app

The note app wrote showed up when sidecar read it. One volume, two windows, the same files. And because that volume hangs off the Pod rather than off either container, it is not sitting in the throwaway writable layer you watched disappear a minute ago. A single container in this Pod can crash and be restarted by the kubelet, and the files under /data stay right where they were. What a volume like this cannot survive is losing the Pod itself.

Where emptyDir stops

There is a hard edge here, and it catches people. An emptyDir volume lives and dies with its Pod. Delete the Pod, or let Kubernetes replace it during a routine update, and the volume goes too, along with everything inside it. Watch the note vanish.

kubectl delete pod notes
kubectl apply -f shared-scratch.yaml
kubectl exec notes -c sidecar -- cat /shared/note.txt

pod "notes" deleted
pod/notes created
cat: can't open '/shared/note.txt': No such file or directory
command terminated with exit code 1

A brand-new Pod gets a brand-new, empty volume, so the read fails. That is the ceiling on emptyDir. It is excellent for scratch space and for passing files around inside one Pod, and wrong for anything that has to outlive the Pod, like a database's files or a user's uploaded photos. Durable data (data that has to stick around) needs storage that sits outside the Pod. That is a PersistentVolumeClaim, a written request for lasting storage, and it is the whole of the next lesson. One more volume type is worth being able to recognize so you can steer clear of it as a beginner: hostPath, which mounts a folder straight from the node's own disk. It pins the Pod to one machine and opens security holes, so it is not the answer for ordinary app data.

Two details about emptyDir are easy to miss. It is carved out of the disk of the node currently running the Pod, so if that Pod is replaced and lands on a different machine, it starts empty there as well. And it is still a much better home for temporary files than the container layer, because a container restart inside a healthy Pod leaves an emptyDir untouched. Caches, unpacked archives, a file a sidecar is about to read: all fine. Anything you would be upset to lose: not fine.

Volumes are also how ConfigMaps and Secrets reach your app as files. A ConfigMap holds plain configuration, a Secret holds sensitive values, and mounting either one as a volume turns each key into a file in a folder. For a nested config file that would be awkward to cram into a single environment variable, files read far better. Two things to keep an eye on: the file permissions the mount gives you, and the lag before an updated ConfigMap or Secret shows up inside a container that is already running.

When someone reports that a file vanished after a restart, the first thing to check is whether a volume was mounted at all. A missing volumeMount is one of the most common beginner outages, and it looks exactly like a bug in the app right up until you read the Pod spec.

Try this

Do it with your own hands. Run a Pod with an emptyDir, write a file into it, delete the Pod, recreate it, and confirm the file is gone. Then compare that with writing only into the container layer, which you already saw fail even earlier, on a plain restart.

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: vol
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      emptyDir: {}
EOF
pod/vol created
 
$ kubectl exec vol -- sh -c 'echo hello > /data/note; cat /data/note'
hello
 
$ kubectl delete pod vol
pod "vol" deleted
 
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: vol
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      emptyDir: {}
EOF
$ kubectl exec vol -- cat /data/note
cat: can't open '/data/note': No such file or directory
 
$ kubectl delete pod vol
pod "vol" deleted

Takeaway

emptyDir buys you one thing: files that survive a container restart inside a Pod that is still alive. It does not survive the Pod. Keep it for scratch you can afford to lose, and send anything you would have to apologise to a user about to persistent storage instead.

Related