|
|
| (5 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | The BCCD, or Bootable Cluster CD, is a diskless cluster solution "created to facilitate ''instruction'' of parallel computing aspects and paradigms." More information about the BCCD can be obtained from [http://bccd.cs.uni.edu/ bccd.cs.uni.edu], as well as from the BCCD's [http://bccd.cs.uni.edu/cgi-bin/twiki/view main wiki], which this page is meant to complement. This page serves generally as an ongoing summation of [http://cs.earlham.edu/~tmcnulty/ Tobias McNulty]'s work on the BCCD for PowerPC. | + | The BCCD, or Bootable Cluster CD, is a diskless cluster solution "created to facilitate ''instruction'' of parallel computing aspects and paradigms." More information about the BCCD can be obtained from [http://bccd.cs.uni.edu/ bccd.cs.uni.edu], as well as from the BCCD's [http://bccd.cs.uni.edu/cgi-bin/twiki/view main wiki], which this page is meant to complement. |
| | | | |
| | =Building the BCCD on PowerPC= | | =Building the BCCD on PowerPC= |
| Line 5: |
Line 5: |
| | '''See also:''' [http://hermes.osuosl.org/nucleus/bccd.php BCCD weblog] | | '''See also:''' [http://hermes.osuosl.org/nucleus/bccd.php BCCD weblog] |
| | | | |
| − | ==Setting Up Your Build Environment==
| + | The instructions that formerly lived here have been replaced by the [[Building the BCCD|generic build instructions]]. |
| − | Base System: [http://www.debian.org/ Debian GNU/Linux] testing with development tools. The Debian installer can be netbooted on most NewWorld PowerMacs. See the PowerPC Kernel Archives' [http://hermes.ppckernel.org/wiki/Mac_Netboot Mac netboot] page for instructions.
| |
| − | | |
| − | To setup your Debian environment, download [http://hermes.osuosl.org/~tmcnulty/bccd-dpkg-selections <code>bccd-dpkg-selections</code>] and run <code>dpkg --set-selections < bccd-dpkg-selections</code> then <code>apt-get dselect-upgrade</code>:
| |
| − | | |
| − | <pre>
| |
| − | su -
| |
| − | /* enter root pass */
| |
| − | wget http://hermes.osuosl.org/~tmcnulty/bccd-dpkg-selections
| |
| − | dpkg --set-selections < bccd-dpkg-selections
| |
| − | apt-get dselect-upgrade
| |
| − | </pre>
| |
| − | | |
| − | ==Building the BBC-PPC ISO==
| |
| − | <pre>
| |
| − | /* Check out the cvs tree anonymously */
| |
| − | cvs -d:pserver:anonymous@bccd.cs.uni.edu:/ login
| |
| − | /* hit return when prompted for a password */
| |
| − | cvs -d:pserver:anonymous@bccd.cs.uni.edu:/ co bccd
| |
| − | | |
| − | /* OR: Check out the cvs tree with a real user account */
| |
| − | export CVS_RSH=ssh
| |
| − | export CVSROOT=username@bccd.cs.uni.edu:/var/lib/cvs
| |
| − | cvs co bccd
| |
| − | | |
| − | /* execute the build */
| |
| − | export singularity_GARCH=ppc
| |
| − | cd bccd/meta/lnx-bbc
| |
| − | make build
| |
| − | /* wait a long, long, long time */
| |
| − | </pre>
| |
| − | | |
| − | =Building a Package and Adding it to the Repository=
| |
| − | The BCCD has a set of downloadable packages, available via the <code>list-packages</code> command (as root) on a running copy of the BCCD. The instructions that follow explain the process of adding a package to the repository that <code>list-packages</code> uses. The BCCD package system simply unpacks the specified tarball in /usr/local, so a "package" can contain anything from a set of binaries to a collection of documentation files in text, html, or some other format (e.g., a curriculum module).
| |
| − | ==On a running BCCD==
| |
| − | These instructions assume that you '''start as root''' in ~, on a running copy of BCCD/PowerPC.
| |
| − | * Build the source:
| |
| − | <pre>
| |
| − | tar xzf somesource-*.tar.gz
| |
| − | cd somesource-*
| |
| − | ./configure --prefix=/usr/local
| |
| − | make
| |
| − | </pre>
| |
| − | * Install the binaries in <code>/usr/local</code>, but archive its current contents first so they don't get packaged up too:
| |
| − | <pre>
| |
| − | mkdir ~/local
| |
| − | mv /usr/local/* ~/local
| |
| − | make install
| |
| − | </pre>
| |
| − | * Package up <code>/usr/local</code>, clean up, and move the old contents of <code>/usr/local</code> back where they belong:
| |
| − | <pre>
| |
| − | cd /usr/local
| |
| − | tar czf ~/somepkg.tar.gz .
| |
| − | rm -rf /usr/local/*
| |
| − | mv ~/local/* /usr/local
| |
| − | cd
| |
| − | </pre>
| |
| − | * Sign the package with GPG (assuming you have access to a [[BCCD#Generating_a_GPG_Key_to_Sign_Packages|valid private key]]):
| |
| − | <pre>
| |
| − | gpg --detach somepkg.tar.gz
| |
| − | gpg --verify somepkg.tar.gz.sig # does this do what I think it does?
| |
| − | </pre>
| |
| − | * Create the package directory on <code>bccd.cs.uni.edu</code>, upload the package and signature, and add the package to <code>packages.txt</code>:
| |
| − | <pre>
| |
| − | ssh bccd.cs.uni.edu mkdir /var/www/packages/ppc/2.2/somepkg
| |
| − | scp somesource.tar.gz* bccd.cs.uni.edu:/var/www/packages/ppc/2.2/somepkg
| |
| − | ssh bccd.cs.uni.edu echo "somepkg some package description" >> /var/www/packages/ppc/2.2/packages.txt
| |
| − | </pre>
| |
| − | (notice the tab between the package name and description on the last line)
| |
| − | | |
| − | ===Example: Building GROMACS===
| |
| − | *Setup your environment:
| |
| − | <pre>
| |
| − | export LDFLAGS="-L/lib/gcc-lib/powerpc-lnxbbc-linux -L/usr/local/lib -L/mpich/lib"
| |
| − | export CPPFLAGS=-I/usr/local/include
| |
| − | export LD_LIBRARY_PATH=/lib/gcc-lib/powerpc-lnxbbc-linux/:/mpich/lib
| |
| − | export PATH=$PATH:/mpich/bin
| |
| − | </pre>
| |
| − | *Fix BCCD lib bug (necessary as of 6/20/2005):
| |
| − | <pre>
| |
| − | mkdir -p /tmp/snapshot.build/staging/singularity/image
| |
| − | ln -s /lib /tmp/snapshot.build/staging/singularity/image/lib
| |
| − | </pre>
| |
| − | *Install necessary headers:
| |
| − | <pre>
| |
| − | list-packages #select fixpackages and hit OK
| |
| − | </pre>
| |
| − | *Build FFTW and GROMACS:
| |
| − | <pre>
| |
| − | cd fftw-* # 2.1.5 works
| |
| − | ./configure --enable-float --enable-type-prefix --enable-mpi --prefix=/usr/local
| |
| − | make -j2
| |
| − | make install
| |
| − | make distclean
| |
| − | ./configure --enable-type-prefix --enable-mpi --prefix=/usr/local
| |
| − | make -j2
| |
| − | make install
| |
| − | cd ../gromacs-* # 3.2.1 works
| |
| − | make distclean
| |
| − | ./configure --enable-mpi --enable-shared --prefix=/usr/local --exec-prefix=/usr/local
| |
| − | make -j2
| |
| − | make install
| |
| − | </pre>
| |
| − | Now you should have a working copy of GROMACS in <code>/usr/local</code>. Enjoy!
| |
| − | | |
| − | ==Generating a GPG Key to Sign Packages==
| |
| − | ===Generating the Key===
| |
| − | '''Quick Guide.''' For the far more detailed instructions from which these commands have been extracted, see [http://www.nmlug.org/faqs/gen-gpg-key.html How to Generate a GPG key].
| |
| − | <pre>
| |
| − | gpg --gen-key
| |
| − | gpg --fingerprint
| |
| − | gpg --export --armor user@host.org > key.gpg.asc
| |
| − | gpg --keyserver wwwkeys.us.pgp.net --send-keys user@host.org
| |
| − | gpg --gen-revoke user@host.org > revoke.gpg.asc
| |
| − | chmod 400 revoke.gpg.asc
| |
| − | </pre>
| |
| − | When finished, <code>key.gpg.asc</code> is your public key (if you're going to sign packages with this key in the distribution BCCD, add it to the image as a trusted key), and <code>revoke.gpg.asc</code> is your revocation certificate, should you need to revoke this key pair.
| |
| − | ===Making the Key Authoritative===
| |
| − | To make your new key authoritative in the BCCD image, you need to append it to <code>bccd/archive/garpkg/files/gar-keys.asc</code>.
| |
| − | * First, grab a copy of your key fingerprint:
| |
| − | <pre>
| |
| − | tmcnulty@bccd:~$ gpg --fingerprint
| |
| − | /home/tmcnulty/.gnupg/pubring.gpg
| |
| − | ---------------------------------
| |
| − | pub 1024R/825ADB6E 2005-06-14 Tobias McNulty (BCCD) <tmcnulty@ppckernel.org>
| |
| − | Key fingerprint = 47BB 9126 6FE0 866D 11C0 AB3E 499D 4547 825A DB6E
| |
| − | </pre>
| |
| − | * Next, append the line starting with "pub" to <code>gar-keys.asc</code>:
| |
| − | <pre>
| |
| − | tmcnulty@c15:~/bccd/archive/garpkg/files$ cat >> gar-keys.asc
| |
| − | | |
| − | pub 1024R/825ADB6E 2005-06-14 Tobias McNulty (BCCD) <tmcnulty@ppckernel.org>
| |
| − | | |
| − | <control-d>
| |
| − | </pre>
| |
| − | * Now, append <code>gar-keys.asc</code> with the contents of <code>key.gpg.asc</code>:
| |
| − | <pre>
| |
| − | tmcnulty@c15:~/bccd/archive/garpkg/files$ cat ~/key.gpg.asc >> gar-keys.asc
| |
| − | </pre>
| |
| − | * You'll need to update the checksum for gar-keys.asc:
| |
| − | <pre>
| |
| − | tmcnulty@c15:~/bccd/archive/garpkg/files$ md5sum gar-keys.asc >> ../checksums
| |
| − | tmcnulty@c15:~/bccd/archive/garpkg/files$ vim ../checksums #remove the old checksum line
| |
| − | </pre>
| |
| − | Now, [[BCCD#Building_the_BBC-PPC_ISO|rebuild the ISO]], [[BCCD#Building_a_Package_and_Adding_it_to_the_Repository|upload some signed packages to the repository]], and give <code>list-packages</code> a try!
| |
| − | | |
| − | =Regression Testing=
| |
| − | ==Executing the Tests==
| |
| − | To execute the regression tests included with bccdrt on a running copy of the BCCD, simply checkout the <code>bccdrt</code> module from CVS on <code>bccd.cs.uni.edu</code> and start <code>runtests.py</code>:
| |
| − | <pre>
| |
| − | export CVSROOT=username@bccd.cs.uni.edu:/var/lib/cvs/var/lib/cvs # you aren't seeing double and this isn't a typo!
| |
| − | cvs co bccdrt
| |
| − | cd bccdrt
| |
| − | ./runtests.py # run list-packages and install python first if need be
| |
| − | </pre>
| |
| − | | |
| − | ==System Overview==
| |
| − | This section exposes the architecture behind bccdrt, which refers generally to the collection of Python and PHP scripts than handle the execution and data extraction (<code>bccdrt.py</code>), recording (<code>post-results.php</code>), and analysis (<code>show-results.php</code>) of BCCD regression tests (the scripts were written and tested on PPC, but can be used on any architecture the BCCD supports).
| |
| − | | |
| − | ===htdocs===
| |
| − | This directory houses the PHP scripts that record and analyze test data.
| |
| − | | |
| − | ===rt.d===
| |
| − | This directory houses individual test scripts and the bccdrt library, which provides several routines to the individual tests (see [[#bccdrt.py|bccdrt.py]] below).
| |
| − | | |
| − | ===runtests.py===
| |
| − | This script calls some generic bccd cluster initialization routines (<code>bccd-allowall</code>, <code>bccd-snarfhosts</code>, and <code>bccd-checkem machines</code>) and then executes each of the tests in the <code>rt.d</code> directory. No per-test modification of <code>runtests.py</code> is necessary.
| |
| − | | |
| − | ===bccdrt.py===
| |
| − | This library provides several routines to individal tests, including <code>system(...)</code>, <code>runcmd(...)</code>, and <code>postresult(...)</code>. It resides in the <code>rt.d</code> directory, but contains no actual test code itself.
| |
| − | | |
| − | *<code>system(...)</code> takes a single argument in the form of a shell command, executes that command, and allows the output to print to the console.
| |
| − | *<code>runcmd(...)</code> does the same thing, but returns the output (both stdout and stderr) as a string.
| |
| − | *<code>postresult(...)</code> takes two arguments: the test name and an associative array of your test data (column) names and values. <code>postresult(...)</code> adds the generic test data, like machine name, BCCD release, etc., encodes the data in XML, and posts it to <code>post-results.php</code> via HTTP.
| |
| − | | |
| − | To define a new test, write a new script in Python that executes whatever programs on the BCCD the test involves and then extracts the relevant data from them (e.g., with regular expressions). There is an sample test in <code>rt.d</code> named <code>gromacs.py</code>. The only call your test function must make is to <code>postresult(...)</code>.
| |
| − | | |
| − | '''Example: Creating the associative array and calling <code>postresult(...)</code>'''
| |
| − | <pre>
| |
| − | import bccdrt
| |
| − | | |
| − | result={}
| |
| − | | |
| − | result["molecule"] = os.path.basename(moleculedir)
| |
| − | result["np"] = np
| |
| − | result["walltime"] = Real
| |
| − | result["psNODEhr"] = psNODEhr
| |
| − | | |
| − | postresult("gromacs",result)
| |
| − | </pre>
| |
| − | | |
| − | ===post-results.php===
| |
| − | This script is generally responsible for recording data as posted by <code>bccdrt.py</code>. This involves the following:
| |
| − | * receiving the XML-encoded data from <code>bccdrt.py</code>
| |
| − | * determining the SQL table format from the given data names and values
| |
| − | * creating the SQL table if necessary
| |
| − | * inserting the given data into the table
| |
| − | | |
| − | '''What does this mean for you?''' <code>post-results.php</code> is entirely automatic; that is, it requires no per-test modification to operate.
| |
| − | | |
| − | ===show-results.php===
| |
| − | This script handles the analysis and display of the regression test data that resides in the SQL tables created by <code>post-results.php</code>. Per regression test, it operates with the folllowing parameters: (a) a table name; (b) an array of check columns, i.e., columns that contain the actual test data that we're concerned with; and (c) an array of control columns, i.e., columns whose values determine the grouping of data within each test table, e.g., if one's control columns are 'processor' and 'node_count', then <code>show-results.php</code> will discover every possible combination (between columns) of distinct column cell values and use that set of combinations to group test data prior to analysis and display. Each grouping combination includes exactly one value from every control column. This is easier understood when seen than when described (due in a large part to my inability to describe the system at hand), so visit the [http://cluster.earlham.edu/~tmcnulty/bccdrt/ actual script] and see what it does for yourself!
| |
| − | | |
| − | So, to make <code>show-results.php</code> aware of your newly-defined regression test, simply create an associative array containing your table name (<code>tbname</code>), array of check columns (<code>check_cols</code>), and array of control columns (<code>control_cols</code>) and append it to the <code>$tests</code> array. The table name is the test name prefixed by <code>rt_</code>, and the column names are exactly the same as those passed to <code>postresult(...)</code> in the original Python.
| |
| − | | |
| − | '''Example: Defining a test in show-results.php'''
| |
| − | <pre>
| |
| − | $tests = array( array( "tbname" => "rt_gromacs",
| |
| − | "check_cols" => array("walltime","psNODEhr"),
| |
| − | "control_cols" => array("machine","molecule","np") )
| |
| − | );
| |
| − | </pre>
| |
| − | | |
| − | <code>show-results.php</code> then iterates through <code>$tests</code> in the body of the HTML document, calling <code>showResults(...)</code> with the parameters defined in each element.
| |
| − | | |
| − | ==Recap: Creating a New Test==
| |
| − | That's a lot of talk for a relatively simple process. Here's the basic outline once more of what creating a new regression test involves:
| |
| − | | |
| − | * 1. write the actual test in Python and call <code>postresult(...)</code> to send the test data to the result server
| |
| − | * 2. drop your test in the <code>rt.d</code> directory
| |
| − | * 3. define your test's table name, check columns, and control columns in <code>show-results.php</code>
| |
| − | * 4. commit your changes to CVS, update the CVS module on the result server, and [[#Executing_the_Tests|execute the tests]].
| |
| − | | |
| − | ==FAQ==
| |
| − | ===How does show-results.php analyze test data?===
| |
| − | The analysis script calculates the standard deviation of each grouping of each check column and from that determines, on a per-cell basis, outliers in the data set, i.e., possible test failures (1*S). Cells that pass the test are given the CSS class <code>rt_pass</code>, and cells that fail <code>rt_fail</code>. Currently, the stylesheet marks passes green and potential failures red.
| |
| − | | |
| − | ==TODO==
| |
| − | * Implement early failure detection and reporting (e.g., "this command doesn't even run")
| |
| − | | |
| − | ==Conclusion==
| |
| − | That brings a close to our discussion of BCCD regression testing. Please direct any questions, comments, or suggestions to the system's author, [mailto:tmcnulty@ppckernel.org Tobias McNulty].
| |
| − | | |
| − | =Fossilizing the BCCD=
| |
| − | This section outlines the steps required to disassemble a BCCD ISO, manifest it on a hard disk drive, and boot from that hard drive. Most or all of this '''must be done as root'''.
| |
| − | | |
| − | ==Mount the Images==
| |
| − | ===The Basic Images===
| |
| − | <pre>
| |
| − | cd /mnt # or where ever
| |
| − | mkdir bccd
| |
| − | mount -t iso9660 -o loop bccd-ppc-2005-08-30T00-0500.iso bccd
| |
| − | | |
| − | # on PPC
| |
| − | mkdir initrd
| |
| − | gunzip < bccd/boot/root.bin > initrd.ext2
| |
| − | mount -t ext2 -o loop initrd.ext2 initrd
| |
| − | | |
| − | # on x86
| |
| − | mkdir lnx
| |
| − | mount -o loop bccd/lnx.img lnx
| |
| − | mkdir root
| |
| − | gunzip < lnx/root.bin > root.ext2
| |
| − | mount -o loop root.ext2 root
| |
| − | </pre>
| |
| − | | |
| − | ===The singularity===
| |
| − | '''First, decompress the singularity with the cloop utility <code>extract_compressed_fs</code>:'''
| |
| − | <pre>
| |
| − | wget http://developer.linuxtag.net/knoppix/sources/cloop_0.66-1.tar.gz
| |
| − | tar xzf cloop_0.66-1.tar.gz
| |
| − | cd cloop-0.66
| |
| − | vim Makefile # add APPSONLY=1 at the top
| |
| − | make zcode
| |
| − | make extract_compressed_fs
| |
| − | ./extract_compressed_fs ../bccd/singularity > ../singularity.romfs
| |
| − | cd ..
| |
| − | </pre>
| |
| − | The latest currently-available version of cloop (2.01) doesn't work for this purpose; others might (I didn't experiment), but 0.66 definitely does.
| |
| − | | |
| − | '''Next, mount the singularity (you must have romfs support compiled into the kernel):'''
| |
| − | <pre>
| |
| − | mkdir singularity
| |
| − | mount -t romfs -o loop singularity.romfs singularity
| |
| − | </pre>
| |
| − | | |
| − | ==Extract the singularity==
| |
| − | <pre>
| |
| − | cd singularity
| |
| − | tar cf - . | (cd /path/to/destination/partition;tar xvf -)
| |
| − | </pre>
| |
| − | | |
| − | ==Create a working initrd==
| |
| − | Create an initrd for fossilized booting with the linuxrc at http://ppckernel.org/~tmcnulty/bccd/linuxrc:
| |
| − | <pre>
| |
| − | cd /mnt/root # or where ever you mounted root.ext2 (from root.bin)
| |
| − | wget http://ppckernel.org/~tmcnulty/bccd/linuxrc # replace the existing linuxrc
| |
| − | chmod a+x linuxrc
| |
| − | cd ..
| |
| − | umount root
| |
| − | gzip < root.ext2 > /path/to/destination/partition/boot/root.bin
| |
| − | </pre>
| |
| − | | |
| − | ==Add / remount read-write hook==
| |
| − | Edit <code>/sbin/singularity-init</code> to remount / read-write during init, using the following command:
| |
| − | <pre>
| |
| − | debug "Remounting / read-write..."
| |
| − | mount -o rw,remount /dev/root /
| |
| − | </pre>
| |
| − | This can be placed somewhere around the proc mount command.
| |
| − | | |
| − | ==Configure the bootloader==
| |
| − | Configure your bootloader (e.g., yaboot, lilo, or grub) as follows:
| |
| − | * boot the kernel <code>/boot/vmlinux</code> on PowerPC or <code>/boot/bzImage</code> on x86
| |
| − | * use the initrd <code>/boot/root.bin</code>
| |
| − | * execute the init script <code>/linuxrc</code>.
| |
| − | | |
| − | Here is a sample [http://ppckernel.org/~tmcnulty/bccd/lilo.conf lilo.conf].
| |
| − | | |
| − | ==TODO==
| |
| − | * fix the mounting commands so that / is only mounted once
| |
| − | * decide how to handle directories like /etc that are mounted in ram at /dev/rw/etc and populated with items from /etc.ro (leave as is, or create a script to simplify the setup for hard disk booting?)
| |
| − | * modify init scripts to make them appropriate for hard disk booting (e.g., remove the "Enter a password for the default user" prompt)
| |
| − | | |
| − | Good luck! Direct questions and comments to [mailto:tmcnulty@ppckernel.org tmcnulty@ppckernel.org].
| |
| | | | |
| | =TODO List for BCCD/PowerPC= | | =TODO List for BCCD/PowerPC= |
The BCCD, or Bootable Cluster CD, is a diskless cluster solution "created to facilitate instruction of parallel computing aspects and paradigms." More information about the BCCD can be obtained from bccd.cs.uni.edu, as well as from the BCCD's main wiki, which this page is meant to complement.
Building the BCCD on PowerPC
See also: BCCD weblog
The instructions that formerly lived here have been replaced by the generic build instructions.
TODO List for BCCD/PowerPC
Fix Runtime Build Environment
1) /etc/ld.so.conf
change /lib/gcc-lib/i386-lnxbbc-linux
to /lib/gcc-lib/powerpc-lnxbbc-linux
bccd/special/bbc-provided/files/ld.so.conf
2) LD_LIBRARY_PATH in /etc/bashrc
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib/gcc-lib/i386-lnxbbc-linux/
should be:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib/gcc-lib/powerpc-lnxbbc-linux/
bccd/special/bbc-provided/files/bashrc
3) Fix library install dir:
ld looks at "/tmp/${USER}.build/staging/singularity/image/lib" to find the libraries that are in "/lib":
/bin/../lib/gcc-lib/powerpc-lnxbbc-linux/3.2.3/../../../../powerpc-lnxbbc-linux/bin/ld: cannot find
//tmp/gray.build/staging/singularity/image//lib/libc.so.6
collect2: ld returned 1 exit status
Not sure what needs to be changed here.. (the quick fix on the image is mkdir -p /tmp/gray.build/staging/singularity/image && ln -sf /lib /tmp/gray.build/staging/singularity/image/lib)
4) CPP defaults to /lib/cpp, but cpp is at /bin/cpp and /usr/bin/cpp
5) MPICC
export MPICC=/lam-mpi/bin/mpicc
so mpicc is found?
6) LDFLAGS - may be fixed once above issues are addressed
Maybe this will come automatically once LD_LIBRARY_PATH and/or ld.so.conf is fixed, but:
export LDFLAGS="-L/lib/gcc-lib/powerpc-lnxbbc-linux"
is currently necessary while building FFTW:
checking whether the Fortran 77 compiler (f77 ) works... no
configure: error: installation or configuration problem: Fortran 77 compiler cannot create executables.
(can't find -lfrtbegin)
Build and Test Packages
To Build and Test
perfex
To Test
gromacs - test multinode runs
