Sysadmin:Layers of abstraction for filesystems

We recently took a painful and convoluted path to understanding management of filesystem abstractions in Linux. I wanted to post that here for reference. Note that these commands should generally not be directly copy-pasted, and should be used advisedly after careful planning.

Let's dive in, shall we?

Filesystem

You've definitely seen this. This is the surface level, the very highest layer of abstraction. If you weren't a sysadmin, there's a good chance this is the only layer you would care about (on your phone, it's likely you don't even care about this one!).

The filesystem is where files are kept and managed. There are tools to mount either the underlying device (/dev/mapper or /dev/vgname) or the filesystem itself to mount points - for example, over NFS. You can also use the filesystem on a logical volume (see below) as the disk for a virtual machine.

This is where ext2, ext3, ext4, xfs, and more come in. This is about abstractions not filesystem implementations, but each works just a little differently. Most of our systems are ext4 but we have some (older) systems with ext2 and others with xfs.

Commands (these may vary by filesystem):

• mount and umount; see /etc/fstab and /etc/exports
• df -h can show you if your filesystem mount is crowded for storage
• fsck (e2fsck, e4fsck, and similar are wrappers around fsck)
• resize2fs /dev/lv/home 512G # resize a filesystem to be 512G, might accompany lvresize below
• xfsdump/xfsrestore for XFS filesystems
• mkfs /dev/lvmdata/device # make a filesystem on a device
• fdisk -l isn't technically a filesystem tool but it operates at a high level of abstraction and you should be aware of it

LVM

LVM stands for "logical volume manager" - Linux's partitioning system.

LVM has three layers of abstraction within itself that each have utilities associated with them. This closely follows the abstraction patterns we'll see as we continue moving closer to hardware.

LVM logical volumes

A volume group can then be organized into logical volumes. The commands here are incredibly powerful and give you the ability to manage disk space with ease (we're grading "easy" on a curve).

If you want to resize a filesystem, there's a good chance you'll want to follow up by resizing the logical volume underneath it.

Commands:

• lvdisplay
• lvscan
• lvcreate -L 20G -n mylv myvg # create a 20GB LVM called mylv in group myvg
• lvresize -L 520G /dev/lv/home # make the LVM on /dev/lv/home 520GB in size

LVM volume groups

A logical volume is created from devices/space within a volume group. It's a collection of one or more LVM "physical" volumes (see below).

Commands:

• vgscan
• vgdisplay
• pvmove /dev/mydevice # to get stuff off of a PV and move it to available free space elsewhere in the VG

LVM physical volumes

At the lowest LVM layer there are "physical" volumes. These might actually correspond to physical volumes (if you have no hardware RAID), or they might be other /dev objects in the OS (/dev/md127 would be a physical volume in this model).

These are the LVM analog to disk partitions.

Commands:

• pvscan
• pvdisplay

Software RAID (optional)

RAID is a system for data management on disk (think redundancy and striping; like filesystem implmentations, too big a topic for this page). There are both `hardware` and `software` implementations of RAID, and software RAID can be managed through programs on the operating systems that are accessible to a (super-)user. Our machine use the popular mdadm, but there are other tools.

Commands:

• mdadm --detail --scan
• mdadm -D /dev/mdXYZ # details
• mdadm -Q /dev/mdXYZ # short, human-readable
• cat /proc/mdstat

Devices in the OS

In UNIX, everything is a file. In Linux that's mostly true as well.

The /dev directory contains the files that correspond to each particular device detected by the OS. I found these useful mostly for reference, because everything refers to them in some way.

If you look closely, things like /dev/mapper/devicename are often symlinks (pointers) to the "real" devices elsewhere in /dev.

You probably won't do that much with these directly. All the other layers provide you better abstractions and more powerful tools for working with devices.

(The astute will observe that /dev is a directory so we've leapt up the layers of abstraction here. True! But it's the best lens you as a user have on the things the OS detects at a much lower level.)

Commands:

• lsblk

Hardware RAID (optional)

If you use software RAID for convenience, you use hardware RAID for performance and information-hiding.

Hardware RAID presents the underlying drives to the OS at boot time by way of a RAID controller on the motherboard. At boot, you can access a tiny bit of software (with a GUI that's probably older than me) to create and modify hardware RAID volumes. In other words, the RAID volume(s), not the physical drives, appear to you as a user.

At least some, and I presume most, RAID controllers have software that you can install on the operating system that will let you get a look at the physical disks that compose the logical volumes.

Relevant software at this level:

• MegaCLI # we have a MegaRAID controller on the server in question
• smartctl --scan
• smartctl -a -d megaraid,15 /dev/bus/6 # substitute the identifying numbers from the scan command above
• not much else - managing hardware RAID carefully requires a reboot; for this reason we tend to keep ours simple

Troubleshooting hardware RAID

If a drive comes offline and the machine uses MegaRAID, you can check by booting the machine into the WebBios. You may find the drive is offline and in an "Unconfigured (bad)" state. This page has the helpful step-by-step guide.

It's always risky playing at the hardware RAID level, so be sure everything is backed up if at all possible.

Physical storage

We have reached the seafloor, where you have some drives - SSD's, spinning disks, etc. Those drives are the very lowest level of abstraction: they are literal, physical machines.

Summary and context

From highest to lowest layers of abstraction:

1. filesystem
2. LVM [lv > vg > pv]
3. software RAID
4. devices in the OS
5. hardware RAID
6. disks

For context, I hope this is useful if we ever run into problems as occurred in February 2020, when we experienced protracted downtime because this knowledge wasn't clear to us yet. If you are reading this of a filesystem problem, I hope this helps organize your thinking and make you aware of the tools that are available to you.