Access modes & reclaim policy

RWO/ROX/RWX, and what happens on delete.

Scale a database to three replicas and watch two of the Pods (the smallest thing Kubernetes runs, usually one container) stall in ContainerCreating and never reach Running. The image is fine. The nodes have room, and the scheduler placed all three without complaint. The volume just can't be in two places at once, and Kubernetes refuses to pretend otherwise. That one physical fact is what access modes are about, and it's the difference between a storage design that works and one that quietly wedges the first time a Pod moves.

Two settings on a PersistentVolume (PV), the object that stands for real durable storage, decide almost everything about how that storage behaves. Access modes control how many Pods can mount the volume, and from where. The reclaim policy controls what happens to your data when someone deletes the claim that owns it. Both ship with quiet defaults. Both tend to bite during incidents rather than demos, so setting them on purpose instead of by accident is worth the ten minutes.

Access modes: who gets to hold the pen

Think of a volume as a single paper notebook. The access mode is the rule about who's allowed to write in it. One person at a desk with the notebook and a pen is ReadWriteOnce (RWO): read and write, but from one node at a time. Hand out photocopies everyone can read and nobody can change, and that's ReadOnlyMany (ROX). Let a whole team scribble in the same notebook at once, and that's ReadWriteMany (RWX), which only works if the notebook was built for a crowd. There's a fourth, the strictest: ReadWriteOncePod (RWOP), exactly one Pod, no sharing with anyone, even a neighbor on the same machine.

Here's the catch that traps people. The access mode isn't a wish you write down. It's a property of the storage sitting underneath. Cloud block disks (Amazon's Elastic Block Store, Google Persistent Disk, Azure Disk) attach to one node at a time, so the ordinary answer for them is RWO and nothing else. A few premium disk types bend that rule (EBS Multi-Attach on io1 and io2, Azure shared disks), but only for claims that ask for volumeMode: Block, and what comes back is a raw device with no filesystem on it. The application has to do its own locking, because an ordinary filesystem like ext4 or XFS corrupts within minutes if two nodes mount it read-write at once. If you want many-writer sharing that behaves like a normal directory, RWX, you need a shared filesystem behind the claim: NFS (Network File System), CephFS, or a managed one like Amazon's Elastic File System (EFS). Ask an ordinary block class for RWX and the claim (a PersistentVolumeClaim, or PVC, a Pod's request for storage) never binds, because no backend on the cluster can honor it. It just sits Pending while you wonder what you broke.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-content
spec:
  accessModes: [ReadWriteMany]     # many Pods across nodes, all writing
  storageClassName: nfs            # a shared-filesystem class, NOT block
  resources:
    requests:
      storage: 10Gi

$ kubectl get pvc web-content

NAME          STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
web-content   Bound    pvc-a31f   10Gi       RWX            nfs            9s

When would you reach for ReadWriteOncePod? Rarely, but it matters. RWO guarantees one node, not one Pod. Two Pods scheduled to the same node can both mount an RWO disk and stamp on each other's writes, which is exactly the split-brain that corrupts a database during a messy failover. RWOP closes that gap by refusing any second Pod anywhere. It's been stable since Kubernetes v1.29 and needs a CSI driver (Container Storage Interface, the standard plugin layer between Kubernetes and your storage vendor) that supports it. Whatever you asked for, verify what you actually got: run kubectl get pv and read the ACCESS MODES column before you trust the design.

Now the failure you'll actually meet in production. You've got a Deployment on an RWO disk, and you either bump the replica count or run a routine rolling update. A new Pod gets scheduled onto a different node and tries to mount the same disk. But the disk is still attached to the old node. Think of the cluster keeping a sign-out sheet for each disk, with room for exactly one node's name. A control-plane loop called the attach/detach controller writes down which node currently holds the volume, in an object called a VolumeAttachment, and the CSI driver handles the physical attach. A block disk allows one name on that sheet at a time, so the second node's request gets rejected and the Pod hangs in ContainerCreating.

$ kubectl describe pod web-2 | grep -A4 Events

Events:
  Type     Reason              Age   From                     Message
  ----     ------              ----  ----                     -------
  Warning  FailedAttachVolume  47s   attachdetach-controller  Multi-Attach error for volume "pvc-a31f": Volume is already exclusively attached to one node and can't be attached to another

Reclaim policy: shred it or keep it

Rent a storage unit, fill it, then cancel the rental. Does the company shred everything inside the minute you hand back the key, or move your boxes to a back room in case you come back? That choice is the reclaim policy, and it's set per volume. Delete means that once the PVC is really gone, the underlying disk and every byte on it is destroyed with it. Retain means the PV sticks around, data intact, waiting for you to come get it. Same word, wildly different outcome the day someone runs the wrong command.

Dynamically provisioned volumes almost always default to Delete, because that's tidy for throwaway data. It's also how databases die. Someone runs kubectl delete pvc in the wrong namespace, or a Helm uninstall sweeps up the claim, and a control-plane loop called the PV controller notices the claim is gone, reads Delete, and tells the CSI driver to call DeleteVolume on the real cloud disk. There's no trash can and no confirmation. One thing does stand in the way, and it is worth recognising. While a Pod still has the claim mounted, a finalizer (a tag on an object that blocks its removal until some condition clears) named kubernetes.io/pvc-protection holds the PVC in Terminating instead of deleting it. That is why a claim you deleted can sit in Terminating for an hour and look wedged when it is only waiting for the last Pod to let go. It also means the destruction lands later, often during the next rollout, when nobody connects the two events. So the first thing you do with any StorageClass you plan to trust with real data is run kubectl get storageclass and check its RECLAIMPOLICY column.

$ kubectl get pv

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM               STORAGECLASS
pvc-a31f   10Gi       RWX            Retain           Released    prod/web-content    nfs
pvc-77c2   20Gi       RWO            Delete           Bound       prod/payments       gp3

Look at pvc-77c2: a payments database on RWO block storage with Delete. That's a loaded gun. You don't have to recreate anything to fix it, because reclaim policy is a property of the live PV, and you can flip it in place. Patch the existing PV to Retain and the data now survives a stray delete.

$ kubectl patch pv pvc-77c2 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

persistentvolume/pvc-77c2 patched

The other half is knowing the recovery move for a Retained volume, because Retain doesn't reattach on its own. When its PVC is deleted, a Retained PV goes to Released, not gone. It won't accept a new claim yet, because it still remembers the old one in a field called claimRef. That's the safety catch. Clear that field and the PV drops back to Available, ready for a fresh PVC to bind. This is the undo button after an accidental delete of a database claim.

$ kubectl patch pv pvc-a31f --type merge -p '{"spec":{"claimRef":null}}'

persistentvolume/pvc-a31f patched

$ kubectl get pv pvc-a31f

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS
pvc-a31f   10Gi       RWX            Retain           Available           nfs

Many cloud disks are RWO only. Sharing one across nodes needs a shared filesystem like NFS, or a premium Multi-Attach disk used as raw block with no filesystem on it.

ReadOnlyMany still needs a provisioner that can publish read-only. The enum alone does not invent capabilities.

Delete a PVC with Retain and you still have a PV full of data. Inventory those orphans.

Try this

Run this on any lab cluster that has a default StorageClass. You create a claim and a Pod that holds it, ask the same class for many writers to see what the backend says, then practise the recovery: protect the volume, delete the claim while the Pod still has it mounted, and bring the volume back. The ReadWriteMany claim is the interesting one. A cloud block class refuses to provision it and says so in the claim's events, while a single-node lab provisioner may simply bind it, because it never has to attach anything to a second node. Both answers make the same point, which is that the backend decides and your YAML only asks. If your class binds late, which is what VOLUMEBINDINGMODE WaitForFirstConsumer means in the listing below, the share-demo claim waits for a Pod before it commits either way, so read its events rather than the STATUS column.

$ kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  12d
 
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mode-demo
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: writer
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "echo hello > /data/keep.txt; sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: mode-demo
EOF
persistentvolumeclaim/mode-demo created
pod/writer created
 
$ kubectl get pvc mode-demo
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mode-demo   Bound    pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2   1Gi        RWO            standard       25s
 
# now ask the same class for many writers, and read the answer off the claim
$ kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: share-demo
spec:
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 1Gi
EOF
persistentvolumeclaim/share-demo created
 
$ kubectl get pvc share-demo
$ kubectl describe pvc share-demo | tail -5
 
# remember the PV behind the RWO claim, then protect it
$ PV=$(kubectl get pvc mode-demo -o jsonpath='{.spec.volumeName}')
$ kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2 patched
 
# delete the claim while the writer Pod still has it mounted
$ kubectl delete pvc mode-demo --wait=false
persistentvolumeclaim "mode-demo" deleted
 
$ kubectl get pvc mode-demo
NAME        STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mode-demo   Terminating   pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2   1Gi        RWO            standard       4m
 
$ kubectl delete pod writer
pod "writer" deleted
 
$ kubectl get pv "$PV"
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM               STORAGECLASS   AGE
pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2   1Gi        RWO            Retain           Released   default/mode-demo   standard       5m
 
# the undo button: clear claimRef and the volume can be claimed again
$ kubectl patch pv "$PV" --type merge -p '{"spec":{"claimRef":null}}'
persistentvolume/pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2 patched
 
$ kubectl get pv "$PV"
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   AGE
pvc-1f0c2a9e-5d3b-4e6a-9c11-8a2f6b0d47e2   1Gi        RWO            Retain           Available           standard       6m
 
# tidy up
$ kubectl delete pvc share-demo && kubectl delete pv "$PV"

Takeaway

Check two columns before you trust a StorageClass with real data: the access modes its backend can actually honor, and RECLAIMPOLICY. If that says Delete on a volume you could not rebuild, patch the live PV to Retain today rather than during the incident.

Related