Volumes: keeping files around

Why container storage vanishes.

A whiteboard in a rented meeting room works like this. You cover it in notes during your hour, you pack up, and the cleaner wipes it blank for the next group. Nothing you wrote survives. A container behaves the same way. A container is a running copy of your app, packaged so it runs the same on any machine. It writes files happily while it runs, and the moment it restarts, the board is blank again. A Pod is the small wrapper Kubernetes puts around one or more containers so it can place them on a machine and run them together. This lesson is about how a Pod hangs on to files instead of losing them every time a container blinks.

Why a container forgets

Every container boots from an image. An image is a read-only snapshot of your app plus the files it ships with, frozen at the moment it was built and never changed while the container runs. Read-only means the container can look at those files but cannot edit them. On top of that frozen snapshot, each container gets a thin writable layer of its own, like a sheet of tracing paper laid over a printed page. Your app can scribble on the tracing paper all it likes. The catch is that the tracing paper belongs to that one container and to nothing else. When a container crashes, or you roll out a new version of the image, the kubelet (the Kubernetes agent running on every node, where a node is one worker machine in your cluster) throws the old container away and starts a clean one from the image. The tracing paper goes in the bin. Whatever the old container wrote goes with it. For a stateless web server, one that keeps nothing on disk between requests, that is fine. For anything that has to remember something, it is a real problem.

Don't take that on faith. Watch it happen. Run one container with no volume, save a file inside it, then delete the Pod and start it again from scratch.

kubectl run scratch --image=busybox:1.36 -- sleep 3600
kubectl exec scratch -- sh -c 'echo "keep me" > /tmp/note.txt'
kubectl delete pod scratch
kubectl run scratch --image=busybox:1.36 -- sleep 3600
kubectl exec scratch -- cat /tmp/note.txt

pod/scratch created
pod "scratch" deleted
pod/scratch created
cat: can't open '/tmp/note.txt': No such file or directory
command terminated with exit code 1

The last line carries the whole lesson. cat cannot find the file, and the shell hands back exit code 1 (an exit code is the number a program leaves behind to report how it ended, where 0 means fine and anything else means trouble). Deleting the Pod threw the container away, writable layer and all, and the replacement booted clean from the image. A crash does the same thing on its own, without you lifting a finger.

kubectl run boom --image=busybox:1.36 -- sh -c 'echo starting; exit 1'
kubectl get pod boom

pod/boom created
NAME   READY   STATUS             RESTARTS      AGE
boom   0/1     CrashLoopBackOff   4 (38s ago)   2m

boom's program exits the instant it starts, so the kubelet keeps binning the dead container and booting a fresh one, waiting a little longer between each attempt. That parked state is called CrashLoopBackOff, one of the first failures you will meet in a real cluster, and the RESTARTS column counts the swaps. Every swap is what wipes the writable layer. To find out why it died, run kubectl logs boom --previous (the logs of the container that ran before the current one) or kubectl describe pod boom. A container's own disk is a dead end for anything you need to keep. The fix is to put the file somewhere outside the container.

A volume clips onto the Pod

A volume is a bag of storage you clip onto the Pod itself rather than onto any single container. To use it, a container mounts the bag at a folder inside itself. Mounting means attaching outside storage so it shows up as an ordinary folder; the app reads and writes through that folder and never knows the files really live in the volume. Because the bag hangs off the Pod, it stays put when a container restarts. And two containers in the same Pod can mount the same bag to hand files to each other. The plainest kind of volume is called emptyDir. Kubernetes creates it empty the moment the Pod starts, and it lives for exactly as long as that Pod does. Good for scratch files, and good for letting a main app pass files to a helper container. A helper that rides along in the same Pod doing a small support job for the main app is usually called a sidecar, after the little car bolted to the side of a motorcycle. Here is a Pod that shares one emptyDir between two containers. It is written in YAML, the plain-text format Kubernetes reads its settings from, where the indentation carries meaning, so copy the spacing exactly.

apiVersion: v1
kind: Pod
metadata:
  name: notes
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /data
    - name: sidecar
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /shared
  volumes:
    - name: scratch
      emptyDir: {}

Read it top to bottom. The Pod runs two containers, app and sidecar. Under each one, volumeMounts says which volume that container wants and where to attach it. Both point at the same volume, named scratch, but each mounts it at a different folder: app sees it at /data, sidecar sees it at /shared. Near the bottom, the volumes section defines scratch once as an emptyDir with empty braces. Those braces mean give me a blank scratch volume with the default settings. One volume, two windows onto the same files.

kubectl apply -f shared-scratch.yaml
kubectl exec notes -c app -- sh -c 'echo "hello from app" > /data/note.txt'
kubectl exec notes -c sidecar -- cat /shared/note.txt

pod/notes created
hello from app

The note app wrote showed up when sidecar read it. One volume, two windows, the same files. And because that volume hangs off the Pod rather than off either container, it is not sitting in the throwaway writable layer you watched disappear a minute ago. A single container in this Pod can crash and be restarted by the kubelet, and the files under /data stay right where they were. What a volume like this cannot survive is losing the Pod itself.

Where emptyDir stops

There is a hard edge here, and it catches people. An emptyDir volume lives and dies with its Pod. Delete the Pod, or let Kubernetes replace it during a routine update, and the volume goes too, along with everything inside it. Watch the note vanish.

kubectl delete pod notes
kubectl apply -f shared-scratch.yaml
kubectl exec notes -c sidecar -- cat /shared/note.txt

pod "notes" deleted
pod/notes created
cat: can't open '/shared/note.txt': No such file or directory
command terminated with exit code 1

A brand-new Pod gets a brand-new, empty volume, so the read fails. That is the ceiling on emptyDir. It is excellent for scratch space and for passing files around inside one Pod, and wrong for anything that has to outlive the Pod, like a database's files or a user's uploaded photos. Durable data (data that has to stick around) needs storage that sits outside the Pod. That is a PersistentVolumeClaim, a written request for lasting storage, and it is the whole of the next lesson. One more volume type is worth being able to recognize so you can steer clear of it as a beginner: hostPath, which mounts a folder straight from the node's own disk. It pins the Pod to one machine and opens security holes, so it is not the answer for ordinary app data.

Two details about emptyDir are easy to miss. It is carved out of the disk of the node currently running the Pod, so if that Pod is replaced and lands on a different machine, it starts empty there as well. And it is still a much better home for temporary files than the container layer, because a container restart inside a healthy Pod leaves an emptyDir untouched. Caches, unpacked archives, a file a sidecar is about to read: all fine. Anything you would be upset to lose: not fine.

Volumes are also how ConfigMaps and Secrets reach your app as files. A ConfigMap holds plain configuration, a Secret holds sensitive values, and mounting either one as a volume turns each key into a file in a folder. For a nested config file that would be awkward to cram into a single environment variable, files read far better. Two things to keep an eye on: the file permissions the mount gives you, and the lag before an updated ConfigMap or Secret shows up inside a container that is already running.

When someone reports that a file vanished after a restart, the first thing to check is whether a volume was mounted at all. A missing volumeMount is one of the most common beginner outages, and it looks exactly like a bug in the app right up until you read the Pod spec.

Try this

Do it with your own hands. Run a Pod with an emptyDir, write a file into it, delete the Pod, recreate it, and confirm the file is gone. Then compare that with writing only into the container layer, which you already saw fail even earlier, on a plain restart.

$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: vol
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      emptyDir: {}
EOF
pod/vol created
 
$ kubectl exec vol -- sh -c 'echo hello > /data/note; cat /data/note'
hello
 
$ kubectl delete pod vol
pod "vol" deleted
 
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: vol
spec:
  containers:
    - name: c
      image: busybox:1.36
      command: ["sleep","300"]
      volumeMounts:
        - name: scratch
          mountPath: /data
  volumes:
    - name: scratch
      emptyDir: {}
EOF
$ kubectl exec vol -- cat /data/note
cat: can't open '/data/note': No such file or directory
 
$ kubectl delete pod vol
pod "vol" deleted

Takeaway

emptyDir buys you one thing: files that survive a container restart inside a Pod that is still alive. It does not survive the Pod. Keep it for scratch you can afford to lose, and send anything you would have to apologise to a user about to persistent storage instead.

Related