Difference between revisions of "Cluster: New BobSCEd Install Log"

From Earlham CS Department
Jump to navigation Jump to search
(→‎Cluster Image Review)
 
(76 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
 +
The source code for *anything* installed locally is in <code>/usr/local/src</code>.  The source for *anything* installed on NFS is in <code>/mounts/bobsced/usr/local/src</code>.
 +
 +
== Modules Software ==
 +
'''Intel Compilers'''
 +
* [https://wiki.cs.earlham.edu/images/8/83/Release_NotesC_en_US.pdf C/C++ Compiler Release Notes (PDF)]
 +
* Installed both custom and *not from RPM* so that I could put it in a different install location (on NFS):
 +
** /mounts/bobsced/usr/local/modules-sw/intel/cce/11.1/
 +
** /mounts/bobsced/usr/local/modules-sw/intel/fce/11.1/
 +
 +
'''Openmpi'''
 +
* 1.3.1 - configured with <code>./configure --prefix=/mounts/bobsced/usr/local/modules-sw/openmpi/1.3.1/ --enable-mpi-threads --with-openib</code>
 +
* 1.3.3 same
 +
* 1.4.1 same
 +
 +
'''MPICH'''
 +
* MPICH1 installed with <code>./configure --prefix=/mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1</code>
 +
** Make note: to uninstall - /mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/sbin/mpiuninstall
 +
** It looks to /mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/share/machines.ARCH instead of just running on the host machine if run without a machine file.  I removed this file to force people to use a machinesfile... that's going to have to be generated from PBS as part of the qsub script. <font color="green">Gave up and added it back in until I finish writing a qsub script that incorporates it</font>
 +
* MPICH2 installed with
 +
:<code>./configure --prefix=/mounts/bobsced/usr/local/modules-sw/mpich2/2.1.2 --enable-cxx --enable-f90 --enable-f77 --enable-threads=multiple --with-thread-package=posix</code>
 +
** Threw OSC's Torque functionality mpiexec on top of it (originally this was package mpich2, now it's mpich2-osc):
 +
*** ./configure --prefix=/mounts/bobsced/usr/local/modules-sw/mpich2/2.1.2 --with-pbs=/var/spool/pbs/ --with-default-comm=mpich2-pmi
 +
*** See [http://debianclusters.cs.uni.edu/index.php/MPICH_with_Torque_Functionality Debian Clustesr: MPICH with Torque]
 +
 +
=== Using Modules ===
 +
Important commands -
 +
* <code>. /etc/profile.d/modules.sh</code>
 +
* <code>module load modules modules-init modules-bobsced</code>
 +
* <code>module avail</code>
 +
* <code>module load x</code>
 +
 +
== Log ==
 +
 
<font color="green">Green color</font> indicates something that still needs to be done.
 
<font color="green">Green color</font> indicates something that still needs to be done.
  
 +
=== Cloning ===
 +
* Download the udpcast rpm from http://udpcast.linux.lu/source.html
 +
** Install with <code>yum --nogpgcheck localinstall udpcast-20081213-1.i386.rpm</code>
 +
** On hopper, installed the syslinux and tftpd-hpa ports
 +
*** Enable tftpd in <code>/etc/inetd.conf</code> by removing the comments and restart inetd with <code>/etc/rc.d/inetd restart</code>, and then also run the command listed on that line to start tftpd
 +
*** The following lines were already in <code>/usr/local/etc/dhcpd.conf: allow booting; allow bootp;</code>, put the filename in the particular group (see [http://debianclusters.cs.uni.edu/index.php/Udpcast_Cloning:_Preparing_the_DHCP_Server_for_PXE_Booting Debian Clusters])
 +
*** <code>cp /usr/local/share/syslinux/pxelinux.0 /tftpboot/</code> (the /tftpboot directory needs to be created)
 +
*** Download linux, initrd, and default from [http://udpcast.linux.lu/bootloader.html#pxe the udpcast site] into /tftpboot
 +
*** Move default into /tftpboot/pxelinux.cfg
 +
*** Restart dhcpd (<code>killall -KILL dhcpd</code> and <code>/usr/local/sbin/dhcpd -q -cf /usr/local/etc/dhcpd.conf -lf /var/db/dhcpd/dhcpd.leases -pf /var/run/dhcpd/dhcpd.pid -user dhcpd -group dhcpd</code>
 +
 +
=== Head Node ===
 
'''Yum installed:'''
 
'''Yum installed:'''
−
* gcc.x86_64
+
* gcc.x86_64, gcc-c++.x86_64
 
* for Ganglia:
 
* for Ganglia:
 
** apr.x86_64 and apr-devel.x86_64
 
** apr.x86_64 and apr-devel.x86_64
−
** libconfuse-2.5-4.el5.x86_64.rpm and libconfuse-devel-2.5-4.el5.x86_64.rpm (from [http://download.fedora.redhat.com/pub/epel/5/x86_64/ Fedora repositories]
+
** libconfuse-2.5-4.el5.x86_64.rpm, libconfuse-devel-2.5-4.el5.x86_64.rpm (from [http://download.fedora.redhat.com/pub/epel/5/x86_64/ Fedora repositories])
 
** expat-devel.x86_64
 
** expat-devel.x86_64
 +
* for Intel updates:
 +
** compat-libstdc++-33.i386
 +
* blas.x86_64 (on all nodes)
  
 
''' Install C3 tools''' from http://www.csm.ornl.gov/torc/C3/C3softwarepage.shtml
 
''' Install C3 tools''' from http://www.csm.ornl.gov/torc/C3/C3softwarepage.shtml
 
* Downloaded full install rpm on bs0, installed with <code>yum --nogpgcheck localinstall c3-4.0.1-1.noarch.rpm</code>
 
* Downloaded full install rpm on bs0, installed with <code>yum --nogpgcheck localinstall c3-4.0.1-1.noarch.rpm</code>
 
** See [[Cluster: C3 Tools README | C3 Tools README]] and [[Cluster: C3 Tools INSTALL | C3 Tools INSTALL]]
 
** See [[Cluster: C3 Tools README | C3 Tools README]] and [[Cluster: C3 Tools INSTALL | C3 Tools INSTALL]]
−
* <font color="green">/etc/c3.conf still needs to be created and configured</font>
+
** Put root's keys in the home directory and authorized itself, then copied that to the worker node image
  
 
'''Ganglia'''
 
'''Ganglia'''
Line 18: Line 67:
 
** See [[Cluster: Ganglia README | Ganglia README]]
 
** See [[Cluster: Ganglia README | Ganglia README]]
 
** <code>./configure --prefix=/cluster</code>
 
** <code>./configure --prefix=/cluster</code>
 +
** The head node uses a different Ganglia gmond.conf in /etc/ganglia/gmond.conf and the workers just have theirs symlinked to /cluster/etc/gmond.conf
 +
** By default, iptables is running on the CentOS install and blocks hopper's Ganglia requests
 +
*** Turned off by clearing it and then running <code>/sbin/service iptables save</code>
 +
 +
'''Networking'''
 +
* Shorewall, see <code>/etc/shorewall/params</code> for almost all of the important definitions
 +
** Natting is done through <code>/etc/shorewall/masq</code>
 +
* DHCP relay, added to boot with <code>chkconfig on</code>, set for hopper (installed as part of dhcp yum package)
 +
** See <code>/etc/sysconfig/dhcrelay</code>
 +
** This means that a dhcp server is also installed, but it is not set to run and is not configured, either
 +
** Hopper needs to have a static route added in order to have the responses return, these are in <code>/etc/rc.conf</code>:
 +
:<code>static_routes="bs0"</code>
 +
:<code>route_bs0="192.168.0.1 159.28.234.200"</code>
 +
 +
'''Modules'''
 +
* Installed environment-modules from http://download.fedora.redhat.com/pub/epel/5/x86_64/repoview/environment-modules.html
 +
** Important directories: <code>/usr/share/Modules/</code>
 +
 +
'''Torque'''
 +
* Installed from source with <code>./configure --with-default-server=bs0.bobsced.loc --with-rc=scp --disable-mom --with-server-home=/var/spool/pbs</code>  ("clients" is what installs qmgr)
 +
* Installs to /usr/local/
 +
* Set up according to [http://debianclusters.cs.uni.edu/index.php/Resource_Manager:_Torque Debian Clusters setup]
 +
* Reran the ./configure but without --disable-moms, then ran <code>make packages</code>, copied this to worker node
 +
 +
'''Maui'''
 +
* installed Maui according to same link as above
 +
 +
'''Intel Firmware Updates'''
 +
* downloaded from http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/17974/eng/D36234-010cd.zip&agr=&ProductID=2451&DwnldId=17974&strOSs=&OSFullName=&lang=eng
 +
 +
'''Mail'''
 +
* Configured sendmail by adding bs0.bobsced.loc to /etc/mail/local-host-names
 +
 +
'''NFS'''
 +
* The actual filesystem on bs0-new is at /mounts/bobsced.  The nodes all mount this in the same place.
 +
* It's mounted on hopper at /mounts/bobsced, with the symlink (currently) at /cluster/bobscednew
 +
 +
'''WebMO'''
 +
* yum installed httpd
 +
* Installed on bs0 with the following params:
 +
<pre>
 +
Path to perl:        /usr/bin/perl
 +
Webserver name:      bs0-new.cluster.earlham.edu
 +
HTML directory:      /var/www/webmo
 +
HTML URL:            /webmo
 +
CGI script directory: /var/www/cgi-bin
 +
CGI script URL:      /cgi-bin
 +
User files directory: /mounts/bobsced/WebMO</pre>
 +
* Get this error when authing with LDAP: <code>Can't locate Authen/Simple/LDAP.pm</code>
 +
* yum installed perl-LDAP.noarch, didn't work, so used CPAN to install Authen::Simple::LDAP
 +
* edited /var/www/cgi-bin/interfaces/authen.conf for our LDAP settings
 +
* Before externally authenticated users can use it, you have to go in as administrator and check the box to allow them in the Webmo group (or whatever other group)
 +
* Gamess:
 +
** yum install compat-gcc-34-g77.x86_64 and gfortran
 +
** Followed directions from [http://www.webmo.net/support/gamess_linux.html Webmo site]
 +
* Added the following line to httpd.conf:
 +
:<code>SuexecUserGroup bob users </code>
 +
* Gaussian 09 not supported, though it's installed in /mounts/bobsced/usr/local/g09
 +
* Installed g03, except get errors:
 +
<pre>Erroneous write during file extend. write 160 instead of 4096
 +
Probably out of disk space.
 +
Write error in NtrExt1: No such file or directory
 +
</pre>
 +
or
 +
<pre>Write error in NtrExt1: Bad address</pre>
 +
** To fix this, do <code>echo 0 > /proc/sys/kernel/randomize_va_space</code>
 +
** <font color="green">This needs to be set to happen all the time on boot</font>
 +
 +
== Infiniband ==
 +
* Drivers downloaded from [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=26&menu_section=34#tab-three here] - the Red Hat 5.3 ones
 +
** mount the ISO as a loopback somewhere (ie <code>mount -o loop /mounts/bobsced/usr/src/MLNX_OFED_LINUX-1.4-rhel5.3.iso /media/</code>)
 +
** run with -msm (ie <code>/media/mlnxofedinstall --msm</code>
 +
* Need to boot into the kernel that came in the original install (2.6.18-128.el5), otherwise get a message like this:
 +
<pre>
 +
The 2.6.18-164.el5 kernel is installed, but do not have drivers available.
 +
Cannot continue.</pre>
 +
* Then (before rebooting back to old kernel), run <code>mst start</code>
 +
* Then, sym link to current kernel version, like this: (see [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2009-May/039758.html Rocks discussion here about it])
 +
** You should get the version number in the error mst start will give you
 +
:<code>ln -s /usr/mst/lib/2.6.18-128.el5/ /usr/mst/lib/2.6.18-164.2.1.e15.plus</code>
 +
* Success looks like this:
 +
<pre>
 +
[root@bs0-new ~]# mst start
 +
    Starting MST (Mellanox Software Tools) driver set:
 +
Loading MST PCI module                                    [  OK  ]
 +
Loading MST PCI configuration module                      [  OK  ]
 +
Saving configuration for PCI device 01:00.0                [  OK  ]
 +
Create devices
 +
</pre>
 +
* <font color="green">Does this need to be done every time?  It looks like yes.</font>
 +
 +
== Cluster Image Review ==
 +
* LDAP needs to be installed on hopper
 +
* Need to test if these are working
 +
** mpich1
 +
** mpich2
 +
** openmpi
 +
* Intel MPI needs to be installed (maybe?)
 +
* Init scripts for pbs and maui need to be setup
 +
 +
{| {{table}}
 +
| align="center" style="background:#f0f0f0;"|''''''
 +
| align="center" style="background:#f0f0f0;"|'''Command Line -np 8'''
 +
| align="center" style="background:#f0f0f0;"|'''Command Line -np 9'''
 +
| align="center" style="background:#f0f0f0;"|'''Torque -np 8'''
 +
| align="center" style="background:#f0f0f0;"|'''Torque -np 9'''
 +
| align="center" style="background:#f0f0f0;"|'''Machinefile'''
 +
| align="center" style="background:#f0f0f0;"|'''Notes'''
 +
|-
 +
| mpich1||OK, except allocates 1 process to bs0||OK, except allocates 1 process to bs0||OK||OK||/mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/share/machines.LINUX||Creates temporary file PIxxxxx while running under qsub; MPI does not respect qsub nodes given, could get around this by creating machinefile at runtime
 +
|-
 +
| mpich2-osc||N/A||N/A||OK, MUST specify # nodes, ppn||OK, MUST specify # nodes, ppn||Generated by PBS||Currently uses OSC's pbs-specific mpiexec (cannot be run outside of qsub)
 +
|-
 +
| mpich2 ||OK, must setup mpd ring||OK, must set up mdp ring||OK, must set up mdp ring||OK, must set up mdp ring||N/A (mpd ring)||Requires chmod 600 .mpd.conf in home directory with MPD_SECRETWORD=somevalue; setup ring with <code><nowiki>sort $PBS_NODEFILE | uniq -c | awk '{print $2":"$1}' > /cluster/home/kwanous/tmp/mpd.nodes; mpdboot -f /cluster/home/kwanous/tmp/mpd.nodes -n 2</nowiki></code>
 +
|-
 +
| openmpi 1.*||OK for running on one node||OK for running on one node||Uses PBS_NODES automatically||Uses PBS_NODES automatically||
 +
|}
 +
 +
Submission scripts:<br>
 +
'''Mpich1'''
 +
 +
'''Mpich2'''
 +
<pre>
 +
#!/bin/bash
 +
#PBS -N testmpich2
 +
#PBS -l cput=00:60:00
 +
#PBS -l nodes=2:ppn=4
 +
 +
. /etc/profile.d/modules.sh
 +
module load modules modules-init modules-bobsced
 +
module load mpich2
 +
 +
sort $PBS_NODEFILE | uniq -c | awk '{print $2":"$1}' > /cluster/home/kwanous/tmp/mpd.nodes
 +
 +
mpdboot -f /cluster/home/kwanous/tmp/mpd.nodes -n 2
 +
mpiexec -np 9 /cluster/home/kwanous/a.out
 +
mpdallexit
 +
 +
rm -rf /cluster/home/kwanous/tmp/mpd.nodes
 +
</pre>
 +
 +
'''Mpich2 - OSC'''
 +
<pre>
 +
#!/bin/bash
 +
#PBS -N testmpich1
 +
#PBS -l cput=00:60:00
 +
#PBS -l nodes=2:ppn=4
 +
 +
hostname
 +
. /etc/profile.d/modules.sh
 +
module load modules modules-init modules-bobsced
 +
module load mpich2
 +
mpirun -np 8 /cluster/home/kwanous/a.out</pre>
 +
 +
'''Openmpi'''
 +
<pre>#!/bin/bash
 +
#PBS -N testopenmpi
 +
#PBS -l cput=00:60:00
 +
#PBS -l nodes=2:ppn=4
 +
 +
hostname
 +
. /etc/profile.d/modules.sh
 +
module load modules modules-init modules-bobsced
 +
module load openmpi/1.3.1
 +
mpirun -np 9 /cluster/home/kwanous/a.out</pre>

Latest revision as of 19:16, 21 April 2010

The source code for *anything* installed locally is in /usr/local/src. The source for *anything* installed on NFS is in /mounts/bobsced/usr/local/src.

Modules Software

Intel Compilers

  • C/C++ Compiler Release Notes (PDF)
  • Installed both custom and *not from RPM* so that I could put it in a different install location (on NFS):
    • /mounts/bobsced/usr/local/modules-sw/intel/cce/11.1/
    • /mounts/bobsced/usr/local/modules-sw/intel/fce/11.1/

Openmpi

  • 1.3.1 - configured with ./configure --prefix=/mounts/bobsced/usr/local/modules-sw/openmpi/1.3.1/ --enable-mpi-threads --with-openib
  • 1.3.3 same
  • 1.4.1 same

MPICH

  • MPICH1 installed with ./configure --prefix=/mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1
    • Make note: to uninstall - /mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/sbin/mpiuninstall
    • It looks to /mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/share/machines.ARCH instead of just running on the host machine if run without a machine file. I removed this file to force people to use a machinesfile... that's going to have to be generated from PBS as part of the qsub script. Gave up and added it back in until I finish writing a qsub script that incorporates it
  • MPICH2 installed with
./configure --prefix=/mounts/bobsced/usr/local/modules-sw/mpich2/2.1.2 --enable-cxx --enable-f90 --enable-f77 --enable-threads=multiple --with-thread-package=posix
    • Threw OSC's Torque functionality mpiexec on top of it (originally this was package mpich2, now it's mpich2-osc):

Using Modules

Important commands -

  • . /etc/profile.d/modules.sh
  • module load modules modules-init modules-bobsced
  • module avail
  • module load x

Log

Green color indicates something that still needs to be done.

Cloning

  • Download the udpcast rpm from http://udpcast.linux.lu/source.html
    • Install with yum --nogpgcheck localinstall udpcast-20081213-1.i386.rpm
    • On hopper, installed the syslinux and tftpd-hpa ports
      • Enable tftpd in /etc/inetd.conf by removing the comments and restart inetd with /etc/rc.d/inetd restart, and then also run the command listed on that line to start tftpd
      • The following lines were already in /usr/local/etc/dhcpd.conf: allow booting; allow bootp;, put the filename in the particular group (see Debian Clusters)
      • cp /usr/local/share/syslinux/pxelinux.0 /tftpboot/ (the /tftpboot directory needs to be created)
      • Download linux, initrd, and default from the udpcast site into /tftpboot
      • Move default into /tftpboot/pxelinux.cfg
      • Restart dhcpd (killall -KILL dhcpd and /usr/local/sbin/dhcpd -q -cf /usr/local/etc/dhcpd.conf -lf /var/db/dhcpd/dhcpd.leases -pf /var/run/dhcpd/dhcpd.pid -user dhcpd -group dhcpd

Head Node

Yum installed:

  • gcc.x86_64, gcc-c++.x86_64
  • for Ganglia:
    • apr.x86_64 and apr-devel.x86_64
    • libconfuse-2.5-4.el5.x86_64.rpm, libconfuse-devel-2.5-4.el5.x86_64.rpm (from Fedora repositories)
    • expat-devel.x86_64
  • for Intel updates:
    • compat-libstdc++-33.i386
  • blas.x86_64 (on all nodes)

Install C3 tools from http://www.csm.ornl.gov/torc/C3/C3softwarepage.shtml

  • Downloaded full install rpm on bs0, installed with yum --nogpgcheck localinstall c3-4.0.1-1.noarch.rpm

Ganglia

  • On hopper, added the data_source line for bs0 to /usr/local/etc/gmetad.conf and restarted it with /usr/local/etc/rc.d/gmetad restart
  • Downloaded tar ball from http://sourceforge.net/projects/ganglia/
    • See Ganglia README
    • ./configure --prefix=/cluster
    • The head node uses a different Ganglia gmond.conf in /etc/ganglia/gmond.conf and the workers just have theirs symlinked to /cluster/etc/gmond.conf
    • By default, iptables is running on the CentOS install and blocks hopper's Ganglia requests
      • Turned off by clearing it and then running /sbin/service iptables save

Networking

  • Shorewall, see /etc/shorewall/params for almost all of the important definitions
    • Natting is done through /etc/shorewall/masq
  • DHCP relay, added to boot with chkconfig on, set for hopper (installed as part of dhcp yum package)
    • See /etc/sysconfig/dhcrelay
    • This means that a dhcp server is also installed, but it is not set to run and is not configured, either
    • Hopper needs to have a static route added in order to have the responses return, these are in /etc/rc.conf:
static_routes="bs0"
route_bs0="192.168.0.1 159.28.234.200"

Modules

Torque

  • Installed from source with ./configure --with-default-server=bs0.bobsced.loc --with-rc=scp --disable-mom --with-server-home=/var/spool/pbs ("clients" is what installs qmgr)
  • Installs to /usr/local/
  • Set up according to Debian Clusters setup
  • Reran the ./configure but without --disable-moms, then ran make packages, copied this to worker node

Maui

  • installed Maui according to same link as above

Intel Firmware Updates

Mail

  • Configured sendmail by adding bs0.bobsced.loc to /etc/mail/local-host-names

NFS

  • The actual filesystem on bs0-new is at /mounts/bobsced. The nodes all mount this in the same place.
  • It's mounted on hopper at /mounts/bobsced, with the symlink (currently) at /cluster/bobscednew

WebMO

  • yum installed httpd
  • Installed on bs0 with the following params:
Path to perl:         /usr/bin/perl
Webserver name:       bs0-new.cluster.earlham.edu
HTML directory:       /var/www/webmo
HTML URL:             /webmo
CGI script directory: /var/www/cgi-bin
CGI script URL:       /cgi-bin
User files directory: /mounts/bobsced/WebMO
  • Get this error when authing with LDAP: Can't locate Authen/Simple/LDAP.pm
  • yum installed perl-LDAP.noarch, didn't work, so used CPAN to install Authen::Simple::LDAP
  • edited /var/www/cgi-bin/interfaces/authen.conf for our LDAP settings
  • Before externally authenticated users can use it, you have to go in as administrator and check the box to allow them in the Webmo group (or whatever other group)
  • Gamess:
    • yum install compat-gcc-34-g77.x86_64 and gfortran
    • Followed directions from Webmo site
  • Added the following line to httpd.conf:
SuexecUserGroup bob users
  • Gaussian 09 not supported, though it's installed in /mounts/bobsced/usr/local/g09
  • Installed g03, except get errors:
Erroneous write during file extend. write 160 instead of 4096
Probably out of disk space.
Write error in NtrExt1: No such file or directory

or 

Write error in NtrExt1: Bad address
    • To fix this, do echo 0 > /proc/sys/kernel/randomize_va_space
    • This needs to be set to happen all the time on boot

Infiniband

  • Drivers downloaded from here - the Red Hat 5.3 ones
    • mount the ISO as a loopback somewhere (ie mount -o loop /mounts/bobsced/usr/src/MLNX_OFED_LINUX-1.4-rhel5.3.iso /media/)
    • run with -msm (ie /media/mlnxofedinstall --msm
  • Need to boot into the kernel that came in the original install (2.6.18-128.el5), otherwise get a message like this:
The 2.6.18-164.el5 kernel is installed, but do not have drivers available. 
Cannot continue.
  • Then (before rebooting back to old kernel), run mst start
  • Then, sym link to current kernel version, like this: (see Rocks discussion here about it)
    • You should get the version number in the error mst start will give you
ln -s /usr/mst/lib/2.6.18-128.el5/ /usr/mst/lib/2.6.18-164.2.1.e15.plus
  • Success looks like this:
[root@bs0-new ~]# mst start
    Starting MST (Mellanox Software Tools) driver set: 
Loading MST PCI module                                     [  OK  ]
Loading MST PCI configuration module                       [  OK  ]
Saving configuration for PCI device 01:00.0                [  OK  ]
Create devices
  • Does this need to be done every time? It looks like yes.

Cluster Image Review

  • LDAP needs to be installed on hopper
  • Need to test if these are working
    • mpich1
    • mpich2
    • openmpi
  • Intel MPI needs to be installed (maybe?)
  • Init scripts for pbs and maui need to be setup
' Command Line -np 8 Command Line -np 9 Torque -np 8 Torque -np 9 Machinefile Notes
mpich1 OK, except allocates 1 process to bs0 OK, except allocates 1 process to bs0 OK OK /mounts/bobsced/usr/local/modules-sw/mpich1/1.2.7p1/share/machines.LINUX Creates temporary file PIxxxxx while running under qsub; MPI does not respect qsub nodes given, could get around this by creating machinefile at runtime
mpich2-osc N/A N/A OK, MUST specify # nodes, ppn OK, MUST specify # nodes, ppn Generated by PBS Currently uses OSC's pbs-specific mpiexec (cannot be run outside of qsub)
mpich2 OK, must setup mpd ring OK, must set up mdp ring OK, must set up mdp ring OK, must set up mdp ring N/A (mpd ring) Requires chmod 600 .mpd.conf in home directory with MPD_SECRETWORD=somevalue; setup ring with sort $PBS_NODEFILE | uniq -c | awk '{print $2":"$1}' > /cluster/home/kwanous/tmp/mpd.nodes; mpdboot -f /cluster/home/kwanous/tmp/mpd.nodes -n 2
openmpi 1.* OK for running on one node OK for running on one node Uses PBS_NODES automatically Uses PBS_NODES automatically

Submission scripts:
Mpich1

Mpich2 

#!/bin/bash
#PBS -N testmpich2
#PBS -l cput=00:60:00
#PBS -l nodes=2:ppn=4

. /etc/profile.d/modules.sh
module load modules modules-init modules-bobsced
module load mpich2

sort $PBS_NODEFILE | uniq -c | awk '{print $2":"$1}' > /cluster/home/kwanous/tmp/mpd.nodes

mpdboot -f /cluster/home/kwanous/tmp/mpd.nodes -n 2
mpiexec -np 9 /cluster/home/kwanous/a.out
mpdallexit

rm -rf /cluster/home/kwanous/tmp/mpd.nodes

Mpich2 - OSC 

#!/bin/bash
#PBS -N testmpich1
#PBS -l cput=00:60:00
#PBS -l nodes=2:ppn=4

hostname
. /etc/profile.d/modules.sh
module load modules modules-init modules-bobsced
module load mpich2
mpirun -np 8 /cluster/home/kwanous/a.out

Openmpi 

#!/bin/bash
#PBS -N testopenmpi
#PBS -l cput=00:60:00
#PBS -l nodes=2:ppn=4

hostname
. /etc/profile.d/modules.sh
module load modules modules-init modules-bobsced
module load openmpi/1.3.1
mpirun -np 9 /cluster/home/kwanous/a.out
Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Cluster:_New_BobSCEd_Install_Log&oldid=11175"