Cluster: New BobSCEd Install Log

Log

Green color indicates something that still needs to be done.

Cloning

• Download the udpcast rpm from http://udpcast.linux.lu/source.html
  • Install with yum --nogpgcheck localinstall udpcast-20081213-1.i386.rpm
  • On hopper, installed the syslinux and tftpd-hpa ports
    • Enable tftpd in /etc/inetd.conf by removing the comments and restart inetd with /etc/rc.d/inetd restart, and then also run the command listed on that line to start tftpd
    • The following lines were already in /usr/local/etc/dhcpd.conf: allow booting; allow bootp;, put the filename in the particular group (see Debian Clusters)
    • cp /usr/local/share/syslinux/pxelinux.0 /tftpboot/ (the /tftpboot directory needs to be created)
    • Download linux, initrd, and default from the udpcast site into /tftpboot
    • Move default into /tftpboot/pxelinux.cfg
    • Restart dhcpd (killall -KILL dhcpd and /usr/local/sbin/dhcpd -q -cf /usr/local/etc/dhcpd.conf -lf /var/db/dhcpd/dhcpd.leases -pf /var/run/dhcpd/dhcpd.pid -user dhcpd -group dhcpd

Head Node

Yum installed:

• gcc.x86_64, gcc-c++.x86_64
• for Ganglia:
  • apr.x86_64 and apr-devel.x86_64
  • libconfuse-2.5-4.el5.x86_64.rpm, libconfuse-devel-2.5-4.el5.x86_64.rpm (from Fedora repositories)
  • expat-devel.x86_64
• for Intel updates:
  • compat-libstdc++-33.i386

Install C3 tools from http://www.csm.ornl.gov/torc/C3/C3softwarepage.shtml

• Downloaded full install rpm on bs0, installed with yum --nogpgcheck localinstall c3-4.0.1-1.noarch.rpm
  • See C3 Tools README and C3 Tools INSTALL
  • Put root's keys in the home directory and authorized itself, then copied that to the worker node image

Ganglia

• On hopper, added the data_source line for bs0 to /usr/local/etc/gmetad.conf and restarted it with /usr/local/etc/rc.d/gmetad restart
• Downloaded tar ball from http://sourceforge.net/projects/ganglia/
  • See Ganglia README
  • ./configure --prefix=/cluster
  • The head node uses a different Ganglia gmond.conf in /etc/ganglia/gmond.conf and the workers just have theirs symlinked to /cluster/etc/gmond.conf
  • By default, iptables is running on the CentOS install and blocks hopper's Ganglia requests
    • Turned off by clearing it and then running /sbin/service iptables save

Networking

• Shorewall, see /etc/shorewall/params for almost all of the important definitions
  • Natting is done through /etc/shorewall/masq
• DHCP relay, added to boot with chkconfig on, set for hopper (installed as part of dhcp yum package)
  • See /etc/sysconfig/dhcrelay
  • This means that a dhcp server is also installed, but it is not set to run and is not configured, either
  • Hopper needs to have a static route added in order to have the responses return, these are in /etc/rc.conf:

static_routes="bs0"

route_bs0="192.168.0.1 159.28.234.200"

Modules

• Installed environment-modules from http://download.fedora.redhat.com/pub/epel/5/x86_64/repoview/environment-modules.html
  • Important directories: /usr/share/Modules/

Torque

• Installed from source with ./configure --with-default-server=bs0.bobsced.loc --with-rc=scp --disable-mom --with-server-home=/var/spool/pbs ("clients" is what installs qmgr)
• Installs to /usr/local/
• Set up according to Debian Clusters setup
• Reran the ./configure but without --disable-moms, then ran make packages, copied this to worker node

Maui

• installed Maui according to same link as above

LDAP

• Head node: yum installed openldap-servers.x86_64
  • Edited /etc/openldap/slapd.conf
  • Ran slappasswd to set the ldap root password
  • Follow quickstart guide: http://www.openldap.org/doc/admin24/quickstart.html
• Workers and head node need to auth from it:
  • yum install openldap-clients
  • http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-ldap-pam.html
  • Make sure /etc/pam.d/sysauth has these ldap entries:

auth       required     pam_env.so
auth       sufficient   pam_unix.so likeauth nullok
auth       sufficient   pam_ldap.so use_first_pass
auth       required     pam_deny.so

account    sufficient   pam_unix.so
account    sufficient   pam_ldap.so
account    required     pam_ldap.so

password   required     pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password   sufficient   pam_unix.so nullok md5 shadow use_authtok
password   sufficient   pam_ldap.so use_first_pass
password   required     pam_deny.so

session    required     pam_limits.so
session    required     pam_unix.so
session    optional     pam_ldap.so

Intel Firmware Updates

• downloaded from http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/17974/eng/D36234-010cd.zip&agr=&ProductID=2451&DwnldId=17974&strOSs=&OSFullName=&lang=eng