LVM: flexible storage

Volumes, resizing, and snapshots.

Advanced14 min · lesson 8 of 17

A raw partition is a fixed slice of one disk. Think of it like a concrete wall between two rooms: once it is poured, moving it means demolition. If your /var partition fills up while the one next to it sits half empty, you cannot shove the boundary over to borrow space. You back up, repartition, and restore, usually with the machine offline. LVM (Logical Volume Manager, the set of tools that pool and slice your storage by programming a layer inside the Linux kernel) replaces that concrete wall with a sliding one.

Here is the mental model. Your disks are barrels of water, all poured into one shared tank. From that tank you draw off buckets of whatever size you like, and you can top up or resize a bucket without ever touching the barrels. LVM works in three layers that do exactly that, and once you see them you can read any server's storage at a glance.

The physical volume (PV) is a raw disk or partition you have handed to LVM. Running pvcreate on a device writes a small label and some metadata to the front of it, marking that device as LVM's to use. The volume group (VG) is the tank: one pool that fuses one or more PVs into a single blob of space. The logical volume (LV) is the bucket you draw out, the flexible partition you put a filesystem on and mount. Because every LV draws from the same pool, you can grow it, shrink it (carefully), move it across physical disks while it is in use, and snapshot it.

The LVM stack, bottom to top
1Physical disks & partitions
/dev/sda2, /dev/sdb
2Physical volumes (PVs)
pvcreate labels each device for LVM
3Volume group (VG)
one pool, data-vg, spanning the PVs
4Logical volumes (LVs)
carved from the pool, resizable
5Filesystem + mount
ext4/XFS on /srv/app

See What You Already Have

Before you change anything, know what is there. On a server you inherited, the storage layout is a map of what matters: which volumes hold data, which are half empty, and whether anything exists that should not. An LV you did not create, or a snapshot nobody remembers taking, is worth a hard look. A snapshot can quietly hold a copy of data as it was before someone tampered with it, and it can equally be a place an attacker parked something. Start with lsblk to see the whole block-device tree.

~/secopslog — bash
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 100G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 99G 0 part └─data--vg-app 252:0 0 90G 0 lvm /srv/app sdb 8:16 0 50G 0 disk

That tells you sda2 is an LVM member and sdb is a spare disk doing nothing. For the LVM-native view, three commands mirror the three layers: pvs lists physical volumes, vgs lists pools, lvs lists volumes. Read them together and you know exactly how much room is free and where.

~/secopslog — bash
$ sudo pvs sudo vgs sudo lvs
PV VG Fmt Attr PSize PFree /dev/sda2 data-vg lvm2 a-- <99.00g <9.00g VG #PV #LV #SN Attr VSize VFree data-vg 1 1 0 wz--n- <99.00g <9.00g LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert app data-vg -wi-ao---- 90.00g

The pool (data-vg) has one PV and under 9 GiB free. The single volume app is 90 GiB, and its Attr string shows a then o at the fifth and sixth flags (active, then open), which means it is mounted and in use right now. That last fact matters for the next part.

Grow A Volume Without Downtime

When app fills up, you do not repartition. You take space from the pool and stretch the volume into it, then grow the filesystem sitting on top so it can use the new room. For ext4 (the common default Linux filesystem) and XFS (a high-performance filesystem standard on Red Hat systems and many data volumes), that final step happens while the filesystem stays mounted and serving traffic. No downtime. The pool here is nearly empty though, so first you add the spare disk to it.

~/secopslog — bash
$ sudo pvcreate /dev/sdb # label the raw disk as an LVM physical volume sudo vgextend data-vg /dev/sdb # pour it into the pool
Physical volume "/dev/sdb" successfully created. Volume group "data-vg" successfully extended

Now the pool has room. lvextend -L +50G adds 50 gibibytes to the volume; the plus sign means grow by that much (write -L 140G instead to set an exact target). resize2fs then grows the ext4 filesystem to fill the enlarged volume. On XFS you would run xfs_growfs /srv/app against the mountpoint instead, and remember that XFS can only grow, never shrink. You can fold both steps into one with lvextend -r (also spelled --resizefs), which resizes the filesystem for you.

~/secopslog — bash
$ sudo lvextend -L +50G /dev/data-vg/app sudo resize2fs /dev/data-vg/app # ext4; for XFS: xfs_growfs /srv/app
Size of logical volume data-vg/app changed from 90.00 GiB (23040 extents) to 140.00 GiB (35840 extents). Logical volume data-vg/app successfully resized. resize2fs 1.46.5 (30-Dec-2021) Filesystem at /dev/data-vg/app is mounted on /srv/app; on-line resizing required old_desc_blocks = 12, new_desc_blocks = 18 The filesystem on /dev/data-vg/app is now 36700160 (4k) blocks long.

The extent count in that output is LVM's allocation unit at work: extents are the fixed-size chunks (4 MiB each by default) the pool hands out, so 140 GiB is 35840 of them. Confirm the change the way any storage change should be confirmed, by looking at what the filesystem itself reports, not at what you hoped happened.

~/secopslog — bash
$ df -h /srv/app
Filesystem Size Used Avail Use% Mounted on /dev/mapper/data--vg-app 138G 85G 47G 65% /srv/app

The Device-Mapper Layer Underneath

LVM does not talk to your disks directly. It programs the kernel's device-mapper (dm), the subsystem that builds virtual block devices which translate reads and writes down to the real hardware. The same machinery encrypts disks for LUKS (Linux Unified Key Setup, the standard for full-disk encryption) through a target called dm-crypt, and it runs LVM's own mirror and RAID (redundant array of independent disks, several drives presented as one) modes. Every LV appears under /dev/mapper as VG-LV, with any dash in a name doubled, which is why the volume app in group data-vg shows up as /dev/mapper/data--vg-app. dmsetup shows the live mapping, including which physical devices a volume now spans.

~/secopslog — bash
$ sudo dmsetup ls --tree
data--vg-app (252:0) ├─ (8:16) └─ (8:2)

Read that as: the app device (major 252) is stitched together from sdb (8:16) and sda2 (8:2). One volume, one filesystem, two physical disks, invisible to anything running above it. That is the payoff of the pool. It is also a reminder for defenders that /dev/mapper is part of your attack surface: anyone with root can map a device straight to raw blocks and bypass the filesystem's permissions entirely, so an unexpected entry here deserves the same suspicion as an unexpected process.

Snapshots: A Freeze-Frame Of A Volume

A snapshot is a freeze-frame. It is not a full copy of the volume. A full copy would cost as much time and space as the data itself. Instead, a snapshot keeps a record of the original and stays valid by writing down only what changes after the moment you take it. The trick is copy-on-write (COW): the instant a block on the live volume is about to be overwritten, LVM first copies the old block aside into the snapshot's reserve, then lets the write land. Blocks nobody touches are never copied. Read the snapshot and you get the old block from the reserve if it changed, or straight from the live volume if it did not. It is like photocopying only the ledger pages someone is about to cross out, and leaving the rest in place.

The classic use is a consistent backup. Freeze the volume, back up the still image at your leisure while the live volume keeps changing, then delete the snapshot. You take one with lvcreate -s, giving it a name and a reserve size for the changes.

~/secopslog — bash
$ sudo lvcreate -s -n app-snap -L 5G data-vg/app
Logical volume "app-snap" created.
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert app data-vg owi-aos--- 140.00g app-snap data-vg swi-a-s--- 5.00g app 0.42

Origin app now reads owi-a (the o marks it as a snapshot origin), and app-snap points back at it. The Data% column is the one to watch: it is the percent of the 5 GiB reserve consumed so far, climbing as the live volume changes. Mount the snapshot read-only, take your backup from that frozen point, unmount, and drop it the moment you are done.

~/secopslog — bash
$ sudo mount -o ro /dev/data-vg/app-snap /mnt/snap sudo tar czf /backup/app-$(date +%F).tar.gz -C /mnt/snap . sudo umount /mnt/snap sudo lvremove -f data-vg/app-snap
Logical volume "app-snap" successfully removed.
A snapshot that fills up is destroyed
An LVM snapshot only reserves room for the changes since you took it. If the live volume changes more than the reserve you gave it (5 GiB above), the snapshot fills, is invalidated, and becomes unreadable, silently breaking whatever backup or image depended on it. Size the reserve for the writes you expect during the snapshot's life, watch its Data% in lvs, and delete it promptly. A lingering snapshot also taxes every write to the origin, because each first-time overwrite now costs an extra copy into the reserve.

Snapshots As An Incident-Response Tool

The same freeze-frame is a first move in IR (incident response, the work of investigating and containing a breach). A box is acting strange and you suspect it is compromised. Before evidence changes under your feet, snapshot the volume to capture its exact state, then image that still copy to storage elsewhere while the machine keeps running. A snapshot device is not the size of its reserve: read it and you see the full 140 GiB origin exactly as it looked at the instant you froze it, unchanged blocks served from the origin and changed blocks from the COW store. That is what makes it a clean forensic image. dd (the raw disk-copy tool) reads it block for block, and you hash the result so its integrity can be proven later.

~/secopslog — bash
$ sudo lvcreate -s -n app-ir -L 20G data-vg/app sudo dd if=/dev/data-vg/app-ir of=/evidence/app.img bs=4M status=progress conv=noerror,sync sha256sum /evidence/app.img | sudo tee /evidence/app.img.sha256
35840+0 records in 35840+0 records out 150323855360 bytes (150 GB, 140 GiB) copied, 1432 s, 105 MB/s 4b1f...c7a9 /evidence/app.img
XFS snapshots need nouuid to mount
An XFS snapshot carries the same filesystem UUID (universally unique identifier) as its origin, and the kernel refuses to mount two XFS filesystems with the same UUID at once. Mount an XFS snapshot with mount -o ro,nouuid,norecovery /dev/data-vg/app-snap /mnt/snap. Forget nouuid and you get a confusing 'wrong fs type, bad option' error rather than an obvious hint. ext4 does not hit this.

The attacker's angle on all of this is the mirror image of the defender's. If someone knows your backups run off a small snapshot, flooding the origin with writes fills that reserve, invalidates the snapshot, and blinds your backup without touching a single log line. Newer setups use thin provisioning, where snapshots draw from a shared thin pool rather than a fixed reserve and do not invalidate the same way. They carry their own failure mode: if the thin pool as a whole runs dry, every volume on it can flip to read-only at once. The discipline is identical, watch the pool and alert before it fills.

Put lvs in your monitoring. A snapshot whose Data% is creeping toward 100 is a backup about to break or a forensic image about to be lost, and it stays silent until it is too late. Take snapshots on purpose, size them for the writes you expect during their life, and remove them the second the backup or image is captured.

Quick check
01You take a 5 GiB snapshot of a busy 140 GiB volume to run an overnight backup. By morning lvs shows the snapshot's Data% at 100% and the backup job errored out. What happened?
Incorrect — The origin draws its space from the volume group, not from the snapshot's reserve. A full reserve takes down the snapshot alone while the origin keeps accepting writes as if nothing happened.
Correct — Copy-on-write parks each about-to-be-overwritten block in the reserve before letting the write land. Fill the reserve and there is nowhere to put the next one, so the snapshot turns unreadable and every job reading it fails.
Incorrect — A snapshot that copied everything would cost the same time and space as the data itself. It records only what changes after you take it, which is why a small reserve normally carries a 140 GiB volume fine.
Incorrect — Reading a snapshot never stops the origin. Those writes keep landing, and each first overwrite of a block is exactly what spends the reserve.
02sudo dmsetup ls --tree prints: data--vg-app (252:0) ├─ (8:16) └─ (8:2) What does this tell you about the app volume?
Correct — The 252:0 device is virtual, built by the kernel on top of 8:16 and 8:2. Everything above it sees a single disk while device-mapper spreads the blocks across two, which is what the pool buys you.
Incorrect — Mirroring is a separate device-mapper mode and this tree shows capacity added together rather than duplicated. Lose either device here and you lose the whole volume.
Incorrect — A mountpoint carries one filesystem at a time. The two entries underneath are hardware feeding a single logical volume, not competing mounts.
Incorrect — dm-crypt is its own device-mapper target and would show up as a named mapped device in the tree. Nothing in this output points to encryption.
03You add a disk, vgextend the pool, then run lvextend -L +100G against a mounted ext4 volume. Every command succeeds, but df -h /srv/app still shows the old size. What fixes it?
Incorrect — That is the right instinct aimed at the wrong filesystem. xfs_growfs works against a mounted XFS filesystem, and this volume carries ext4.
Incorrect — df asks the mounted filesystem how large it is, and the filesystem honestly has not grown yet, so a remount reports the same number.
Incorrect — pvcreate labelled the disk once and vgextend already folded its space into the pool. Repeating pvcreate overwrites metadata for no benefit.
Correct — The volume and the filesystem are separate layers, and only the volume grew. resize2fs pushes ext4 out to the new boundary while it stays mounted, and lvextend -r folds both steps together next time.

Try this

Work through “Snapshots As An Incident-Response Tool” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a snapshot that fills up is destroyed. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related