Diskless Cluster Setup

From Earlham CS Department
Jump to navigation Jump to search

This page shows how to setup a diskless cluster that uses UnionFS to consolidate system and cluster software in a single disk image. It is a work in progress.

Choose an operating system

The first version of this project used Debian GNU/Linux. It's intended to be general enough to be deployed on other platforms; but, as Charlie says, "your mileage may vary."

Setup the shared environment

Install a kernel

You must build the kernel locally, otherwise the unionfs install will fail later.

  • Download kernel source v. (or anything 2.6 before 2.6.16) from kernel.org
  • Get a config for your kernel.
    • If you're using the pegasos machines, use pegasos-debian from ppckernel.org
    • If you need to configure your own kernel, make sure the following options are enabled and built in to the kernel (not compiled as modules):
      • all ATA disk drivers
      • a driver for your Ethernet card
      • Packet socket support
      • kernel-level IP autoconfiguration
      • ext2 and ext3 (or whatever file system) drivers
      • NFS support
      • NFS server support
      • Root over NFS support
  • Untar the sources in /usr/src/linux- (should be a more permanent location)
  • Copy the config to /usr/src/linux-
  • Then:
cd /usr/src/linux-
make oldconfig
cp arch/ppc/boot/images/zImage.chrp /boot/vmlinuz-
make modules_install

Install unionfs

You must build unionfs with the same gcc compiler as you used to build the kernel above, otherwise you'll get an error about "Invalid module format."

Now you should be able to modprobe unionfs

See http://www.unionfs.org/ for usage examples.

Setup the server environment

PXE (i386 only)

This section is only required for i386-based machines, not PowerPCs. Download the latest syslinux bootloader: http://www.kernel.org/pub/linux/utils/boot/syslinux/

cp ./pxelinux.0 /boot/
mkdir /boot/pxelinux.cfg
vi /boot/pxelinux.cfg/default

add a line similar to this, but with a reference to a kernel from /boot/:

DEFAULT vmlinuz- init=/linuxrc root=/dev/nfs ip=dhcp


Install a tftp server (Debian's tftpd-hpa is good) and add a line like this to /etc/inetd.conf (or the equivalent in whatever internet "super-server" you use):

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -v -s /boot

This runs tftpd and sets its root directory to /boot


Install ISC's DHCPD (e.g., Debian's dhcp3-server package), and setup dhcpd.conf like so:

option domain-name "little-fe-ppc.net";
option domain-name-servers;
option subnet-mask;
default-lease-time 604800;
max-lease-time 604800;
allow booting;
allow bootp;

subnet netmask {
    option broadcast-address;
    option routers;


    # for client-side union setup
    #   for powerpc:
    filename "vmlinuz- init=/linuxrc root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    #   for i386:
    filename "pxelinux.0"

    option root-path "/client/setup";

    # to use server-side union
    # filename "vmlinuz- root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    # option root-path "/client/root";

    use-host-decl-names on;

host lf1 { hardware ethernet 00:0B:2F:43:61:EB; fixed-address; }
host lf2 { hardware ethernet 00:0b:2f:4f:03:b5; fixed-address; }
#host lf3 { hardware ethernet ; fixed-address; }

subnet netmask {
    option broadcast-address;

    use-host-decl-names on;

host lf1-eth1 { hardware ethernet 00:0b:2f:62:49:d1; fixed-address; }
host lf2-eth1 { hardware ethernet 00:0b:2f:6e:6b:7b; fixed-address; }
#host lf3-eth1 { hardware ethernet ; fixed-address; }


If you're using Debian, use the nfs-kernel-server package (not the nfs-user-server package). (Knowledge about how this translates to other platforms would be helpful here.)

Setup /etc/exports like so:

/     ,insecure_locks,no_root_squash,sync)
/root ,insecure_locks,no_root_squash,sync)
/home ,insecure_locks,no_root_squash,sync)

Setup the client environment


Download and build Busybox:

make defconfig
make menuconfig #make a statically linked binary
make install #this creates the _install directory

Now create some necessary directories:

cd _install
mkdir dev proc sys mnt
cd ..

Now, put the _install directory somewhere that it can be NFS mounted, e.g., /client/setup

Backing up the environment

  • Attach the hdd you wish to back up and the hdd you wish to store the back up to to the machine.
  • Boot in an enviroment that leaves both hdds as read only (i.e. BCCD).
  • Locate where each hdd is in the /dev structure.
  • Mount both drives and ascertain the dircetory locations of both hdd takes special note of the one that is to be backed up.
    • example: mkdir ~/hda0; mount /dev/hda/part0 ~/hda0; ls ~/hda0
  • Umount both drives.
    • umount ~/hda0
  • Use the dd utility to perform the backup. Make sure the drive to be backed up is the input to dd and the drive to back up to is the output. If you want back up hda to hdc, the command would look like:
    • dd if=/dev/hda of=/dev/hdc