BCCD:Fossilizing

This page is now stale

Your mileage may vary, and we are not updating this page any longer. We use it internally for reference, but we are now working on Liberating the BCCD (see the main Cluster Computing Group page).

Fossilizing the BCCD

This section outlines the steps required to disassemble a BCCD ISO, manifest it on a hard disk drive, and boot from that hard drive. Most or all of this must be done as root.

Mount the Images

These scripts, used for the lnx-bbc project, might prove to be helpful in working with the BCCD images: FossilScripts

The Basic Images

cd /mnt # or where ever
mkdir bccd
mount -t iso9660 -o loop bccd-ppc-2005-08-30T00-0500.iso bccd

# on PPC
mkdir initrd
gunzip < bccd/boot/root.bin > initrd.ext2
mount -t ext2 -o loop initrd.ext2 initrd

# on x86
mkdir lnx
mount -o loop bccd/lnx.img lnx
mkdir root
gunzip < lnx/root.bin > root.ext2
mount -o loop root.ext2 root

The singularity

First, decompress the singularity with the cloop utility extract_compressed_fs:

wget http://developer.linuxtag.net/knoppix/sources/cloop_0.66-1.tar.gz
tar xzf cloop_0.66-1.tar.gz
cd cloop-0.66
vim Makefile # add APPSONLY=1 at the top
make zcode
make extract_compressed_fs
./extract_compressed_fs ../bccd/singularity > ../singularity.romfs
cd ..

The latest currently-available version of cloop (2.01) doesn't work for this purpose; others might (I didn't experiment), but 0.66 definitely does.

Next, mount the singularity (you must have romfs support compiled into the kernel):

mkdir singularity
mount -t romfs -o loop singularity.romfs singularity

Extract the singularity

cd singularity
tar cf - . | (cd /path/to/destination/partition;tar xvf -)

Create a working initrd

Create an initrd for fossilized booting with the linuxrc at http://ppckernel.org/~tobias/bccd/linuxrc:

cd /mnt/root # or where ever you mounted root.ext2 (from root.bin)
wget http://ppckernel.org/~tobias/bccd/linuxrc # replace the existing linuxrc
chmod a+x linuxrc
cd ..
umount root
gzip < root.ext2 > /path/to/destination/partition/boot/root.bin

Edit singularity-init

Add / remount read-write hook

Edit /sbin/singularity-init to remount / read-write during init, using the following command:

debug "Remounting / read-write..."
mount -o rw,remount /dev/root /

This can be placed somewhere around the proc mount command.

Prepare for Fossilization of /mnt/rw

Comment out lines concerning /mnt/rw

# mount -n -t tmpfs none /mnt/rw

Add network setup to singularity-init

ifconfig eth0 inet 192.168.10.1 netmask 255.255.255.0 broadcast 192.168.10.255 up
route add default gw 192.168.10.1 eth0

Configure the bootloader

Configure your bootloader (e.g., yaboot, lilo, or grub) as follows:

• boot the kernel /boot/vmlinux on PowerPC or /boot/bzImage on x86
• use the initrd /boot/root.bin
• execute the init script /linuxrc.

Here is a sample lilo.conf.

Setup Compatibility Nodes

Add the following to /linuxrc:

• /sbin/devfsd /dev

De-Obfuscation

Remove Unneeded Symlinks

The deal is that the BCCD is now on a different (read/writeable) medium: a harddisk. Let's un-obfuscate some of the workings. An ls -l on / will reveal a few symlinks: /etc, /home, /local, /tmp, and /var. All of these point to an appropriate directory in /mnt/rw. What happens is that since the CD is not writeable, it creates a ramdisk, copies files from /etc.ro/ to /mnt/rw/etc/ (change etc accordingly), and then the /etc symlink becomes a writeable medium.

Here's the works:

rm /etc /home /local /tmp /var
mkdir /etc /home /local /tmp /var
cd /etc.ro   && tar cf - . | (cd /etc/;   tar vf -)
cd /home.ro  && tar cf - . | (cd /home/;  tar vf -)
cd /local.ro && tar cf - . | (cd /local/; tar vf -)
cd /tmp.ro   && tar cf - . | (cd /tmp/;   tar vf -)
cd /var.ro   && tar cf - . | (cd /var/;   tar vf -)

You're almost done, except you should remove the place in the scripts where the bootup copies the files from /<dir>.ro/. Just comment out the lines in /sbin/singularity-init that do the copying (around line 105):

# cp -a /etc.ro /mnt/rw/etc
# cp -a /var.ro /mnt/rw/var

While you're editing /sbin/singularity-init, also comment out these lines:

# rsync -plarv /lib/mozilla-1.6/plugins.ro/ /mnt/rw/plugins/
# chmod 1777 /mnt/rw/tmp
# debug "Making /mnt/rw/tmp/build links"
# mkdir -p /mnt/rw/tmp/build/
# mkdir -p /mnt/rw/tmp/build/staging
# mkdir -p /mnt/rw/tmp/build/staging/singularity
# mkdir -p /mnt/rw/tmp/build/staging/singularity/image
# ln -s /lib /mnt/rw/tmp/build/staging/singularity/image/lib

Configure gcc Environment

Though the BCCD is now fossilized onto the harddrive, the gcc environment does not know this as it was compiled for the CD. It will look for files in (effectively) /tmp/build/staging/singularity/image/lib ... the directories and symlink creation that we just commented out. Since /tmp is a fossilized directory, just create a symlink inside of it:

mkdir -p /tmp/build/staging/singularity/image
cd /tmp/build/staging/singularity/image/
ln -s /lib

TODO

• fix the mounting commands so that / is only mounted once (?)
• decide how to handle directories like /etc that are mounted in ram at /dev/rw/etc and populated with items from /etc.ro (leave as is, or create a script to simplify the setup for hard disk booting?)
  • Kevin's done this, we just need to document
    • DONE
• modify init scripts to make them appropriate for hard disk booting (e.g., remove the "Enter a password for the default user" prompt)
  • This appears to be done
• finish setting up networking
• create a patch against the original singularity image for /sbin/singularity-init and other modified configuration files for automating the fossilize process
• package up any binary additions with list-packages (see the package instructions in the wiki)
• last but not least, keep track of all the changes we make!

Good luck! Direct questions and comments to tobias@cs.earlham.edu.