Diskless Cluster Setup

This page shows how to setup a diskless cluster that uses UnionFS to consolidate system and cluster software in a single disk image. It is a work in progress.

Choose an operating system

The first version of this project used Debian GNU/Linux. It's intended to be general enough to be deployed on other platforms; but, as Charlie says, "your mileage may vary."

Setup the shared environment

Install a kernel

You must build the kernel locally, otherwise the unionfs install will fail later.

• Download kernel source v. 2.6.15.6 (or anything 2.6 before 2.6.16) from kernel.org
• Get a config for your kernel.
  • If you're using the pegasos machines, use pegasos-debian from ppckernel.org
  • If you need to configure your own kernel, make sure the following options are enabled and built in to the kernel (not compiled as modules):
    • all ATA disk drivers
    • a driver for your Ethernet card
    • Packet socket support
    • kernel-level IP autoconfiguration
    • ext2 and ext3 (or whatever file system) drivers
    • NFS support
    • NFS server support
    • Root over NFS support
• Untar the sources in /usr/src/linux-2.6.15.6 (should be a more permanent location)
• Copy the config to /usr/src/linux-2.6.15.6/.config
• Then:

cd /usr/src/linux-2.6.15.6
make oldconfig
make
cp arch/ppc/boot/images/zImage.chrp /boot/vmlinuz-2.6.15.6
make modules_install

Install unionfs

You must build unionfs with the same gcc compiler as you used to build the kernel above, otherwise you'll get an error about "Invalid module format."

• Download the unionfs sources from http://www.fsl.cs.sunysb.edu/project-unionfs.html
• Untar them, cd into the resulting directory, and then simply type make and then make install

Now you should be able to modprobe unionfs

See http://www.unionfs.org/ for usage examples.

Setup the server environment

PXE (i386 only)

This section is only required for i386-based machines, not PowerPCs. Download the latest syslinux bootloader: http://www.kernel.org/pub/linux/utils/boot/syslinux/

make
cp ./pxelinux.0 /boot/
mkdir /boot/pxelinux.cfg
vi /boot/pxelinux.cfg/default

add a line similar to this, but with a reference to a kernel from /boot/:

DEFAULT vmlinuz-2.6.15.7 init=/linuxrc root=/dev/nfs ip=dhcp

TFTPD

Install a tftp server (Debian's tftpd-hpa is good) and add a line like this to /etc/inetd.conf (or the equivalent in whatever internet "super-server" you use):

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -v -s /boot

This runs tftpd and sets its root directory to /boot

DHCPD

Install ISC's DHCPD (e.g., Debian's dhcp3-server package), and setup dhcpd.conf like so:

option domain-name "little-fe-ppc.net";
option domain-name-servers 192.168.1.100;
option subnet-mask 255.255.255.0;
default-lease-time 604800;
max-lease-time 604800;
allow booting;
allow bootp;

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.200 192.168.1.254;
    option broadcast-address 192.168.1.255;
    option routers 192.168.1.1;

    next-server 192.168.1.100;
    server-identifier 192.168.1.100;

    # for client-side union setup
    #   for powerpc:
    filename "vmlinuz-2.6.15.6 init=/linuxrc root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    #   for i386:
    filename "pxelinux.0"

    option root-path "/client/setup";

    # to use server-side union
    # filename "vmlinuz-2.6.15.6 root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    # option root-path "/client/root";

    use-host-decl-names on;
}

host lf1 { hardware ethernet 00:0B:2F:43:61:EB; fixed-address 192.168.1.101; }
host lf2 { hardware ethernet 00:0b:2f:4f:03:b5; fixed-address 192.168.1.102; }
#host lf3 { hardware ethernet ; fixed-address 192.168.1.103; }

subnet 192.168.10.0 netmask 255.255.255.0 {
    range 192.168.10.200 192.168.10.254;
    option broadcast-address 192.168.10.255;

    use-host-decl-names on;
}

host lf1-eth1 { hardware ethernet 00:0b:2f:62:49:d1; fixed-address 192.168.10.101; }
host lf2-eth1 { hardware ethernet 00:0b:2f:6e:6b:7b; fixed-address 192.168.10.102; }
#host lf3-eth1 { hardware ethernet ; fixed-address 192.168.10.103; }

NFSD

If you're using Debian, use the nfs-kernel-server package (not the nfs-user-server package). (Knowledge about how this translates to other platforms would be helpful here.)

Setup /etc/exports like so:

/               192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/client/setup   192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/client/overlay 192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/root           192.168.1.0/24(rw,insecure_locks,no_root_squash,sync)
/home           192.168.1.0/24(rw,insecure_locks,no_root_squash,sync)

Setup the client environment

Busybox

Download and build Busybox:

make defconfig
make menuconfig #make a statically linked binary
make
make install #this creates the _install directory

Now create some necessary directories:

cd _install
mkdir dev proc sys mnt
cd ..

Now, put the _install directory somewhere that it can be NFS mounted, e.g., /client/setup

Backing up the environment

• Attach the hdd you wish to back up and the hdd you wish to store the back up to to the machine.
• Boot in an enviroment that leaves both hdds as read only (i.e. BCCD).
• Locate where each hdd is in the /dev structure.
• Mount both drives and ascertain the dircetory locations of both hdd takes special note of the one that is to be backed up.
  • example: mkdir ~/hda0; mount /dev/hda/part0 ~/hda0; ls ~/hda0
• Umount both drives.
  • umount ~/hda0
• Use the dd utility to perform the backup. Make sure the drive to be backed up is the input to dd and the drive to back up to is the output. If you want back up hda to hdc, the command would look like:
  • dd if=/dev/hda of=/dev/hdc