Diskless Cluster Setup

This page shows how to setup a diskless cluster that uses UnionFS to consolidate system and cluster software in a single disk image. It is a work in progress.

Choose an operating system

The first version of this project used Debian GNU/Linux. It's intended to be general enough to be deployed on other platforms; but, as Charlie says, "your mileage may vary."

Setup the shared environment

Install a kernel

You must build the kernel locally, otherwise the unionfs install will fail later.

• Download kernel source v. 2.6.15.6 (or anything 2.6 before 2.6.16) from kernel.org
• Get a config for your kernel.
  • If you're using the pegasos machines, use pegasos-debian from ppckernel.org
  • If you need to configure your own kernel, make sure the following options are enabled and built in to the kernel (not compiled as modules):
    • all ATA disk drivers
    • a driver for your Ethernet card
    • Packet socket support
    • kernel-level IP autoconfiguration
    • ext2 and ext3 (or whatever file system) drivers
    • NFS support
    • NFS server support
    • Root over NFS support
• Untar the sources in /usr/src/linux-2.6.15.6 (should be a more permanent location)
• Copy the config to /usr/src/linux-2.6.15.6/.config
• Then:

cd /usr/src/linux-2.6.15.6
make oldconfig
make
cp arch/ppc/boot/images/zImage.chrp /boot/vmlinuz-2.6.15.6
make modules_install

Install unionfs

You must build unionfs with the same gcc compiler as you used to build the kernel above, otherwise you'll get an error about "Invalid module format."

• Download the unionfs sources from http://www.fsl.cs.sunysb.edu/project-unionfs.html
• Untar them, cd into the resulting directory, and then simply type make and then make install

Now you should be able to modprobe unionfs

See http://www.unionfs.org/ for usage examples.

Setup the server environment

PXE (i386 only)

This section is only required for i386-based machines, not PowerPCs. Download the latest syslinux bootloader: http://www.kernel.org/pub/linux/utils/boot/syslinux/

make
cp ./pxelinux.0 /boot/
mkdir /boot/pxelinux.cfg
vi /boot/pxelinux.cfg/default

add a line similar to this, but with a reference to a kernel from /boot/:

DEFAULT vmlinuz-2.6.15.7 init=/linuxrc root=/dev/nfs ip=dhcp

TFTPD

Install a tftp server (Debian's tftpd-hpa is good) and add a line like this to /etc/inetd.conf (or the equivalent in whatever internet "super-server" you use):

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -v -s /boot

This runs tftpd and sets its root directory to /boot

DHCPD

Install ISC's DHCPD (e.g., Debian's dhcp3-server package), and setup dhcpd.conf like so:

option domain-name "little-fe-ppc.net";
option domain-name-servers 192.168.1.100;
option subnet-mask 255.255.255.0;
default-lease-time 604800;
max-lease-time 604800;
allow booting;
allow bootp;

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.200 192.168.1.254;
    option broadcast-address 192.168.1.255;
    option routers 192.168.1.1;

    next-server 192.168.1.100;
    server-identifier 192.168.1.100;

    # for client-side union setup
    filename "vmlinuz-2.6.15.6 init=/linuxrc root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    option root-path "/client/setup";

    # to use server-side union
    # filename "vmlinuz-2.6.15.6 root=/dev/nfs ip=dhcp console=ttyS1,115200n1";
    # option root-path "/client/root";

    use-host-decl-names on;
}

host lf1 { hardware ethernet 00:0B:2F:43:61:EB; fixed-address 192.168.1.101; }
host lf2 { hardware ethernet 00:0b:2f:4f:03:b5; fixed-address 192.168.1.102; }
#host lf3 { hardware ethernet ; fixed-address 192.168.1.103; }

subnet 192.168.10.0 netmask 255.255.255.0 {
    range 192.168.10.200 192.168.10.254;
    option broadcast-address 192.168.10.255;

    use-host-decl-names on;
}

host lf1-eth1 { hardware ethernet 00:0b:2f:62:49:d1; fixed-address 192.168.10.101; }
host lf2-eth1 { hardware ethernet 00:0b:2f:6e:6b:7b; fixed-address 192.168.10.102; }
#host lf3-eth1 { hardware ethernet ; fixed-address 192.168.10.103; }

NFSD

If you're using Debian, use the nfs-kernel-server package (not the nfs-user-server package). (Knowledge about how this translates to other platforms would be helpful here.)

Setup /etc/exports like so:

/               192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/client/setup   192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/client/overlay 192.168.1.0/24(ro,insecure_locks,no_root_squash,sync)
/root           192.168.1.0/24(rw,insecure_locks,no_root_squash,sync)
/home           192.168.1.0/24(rw,insecure_locks,no_root_squash,sync)

Setup the client environment

Busybox

Download and build Busybox:

make defconfig
make menuconfig #make a statically linked binary
make
make install #this creates the _install directory

Now create some necessary directories:

cd _install
mkdir dev proc sys mnt
cd ..

Now, put the _install directory somewhere that it can be NFS mounted, e.g., /client/setup