Bobsced Cluster - Revision history

Leemasa: /* RLIMIT_MEMLOCK */

2010-01-04T16:51:04Z

RLIMIT_MEMLOCK

← Older revision		Revision as of 16:51, 4 January 2010
Line 82:		Line 82:
	Please refer here if you encounter an error message on BobSCEd that you cannot handle.		Please refer here if you encounter an error message on BobSCEd that you cannot handle.
	==RLIMIT_MEMLOCK==		==RLIMIT_MEMLOCK==
			$./area_mpi
	libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.		libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
	This will severely limit memory registrations.--------------------------------------------------------------------------		This will severely limit memory registrations.--------------------------------------------------------------------------
Line 89:		Line 90:
	memlock limits should be set to "unlimited". The failure occured		memlock limits should be set to "unlimited". The failure occured
	here:		here:

	Local host: bobsced0		Local host: bobsced0
	OMPI source: btl_openib_component.c:1040		OMPI source: btl_openib_component.c:1040
Line 95:		Line 95:
	Device: mthca0		Device: mthca0
	Memlock limit: 32768		Memlock limit: 32768

	You may need to consult with your system administrator to get this		You may need to consult with your system administrator to get this
	problem fixed. This FAQ entry on the Open MPI web site may also be		problem fixed. This FAQ entry on the Open MPI web site may also be
	helpful:		helpful:

	http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages		http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
	--------------------------------------------------------------------------		--------------------------------------------------------------------------
	--------------------------------------------------------------------------		--------------------------------------------------------------------------
	WARNING: There was an error initializing an OpenFabrics device.		WARNING: There was an error initializing an OpenFabrics device.

	Local host: bobsced0		Local host: bobsced0
	Local device: mthca0		Local device: mthca0
	--------------------------------------------------------------------------		--------------------------------------------------------------------------

			This error is specifically related to infiniband on BobSCEd. If you're using ethernet, specify as such by creating a file,
			~/.openmpi/mca-params.conf
			with the first line being
			btl = ^openib
			This will tell BobSCEd not to try infiniband, and will stop the error.
			If you are using Infiniband and find a solution to this error, please place it in the wiki.

	=References=		=References=

Leemasa at 16:42, 4 January 2010

2010-01-04T16:42:35Z

← Older revision		Revision as of 16:42, 4 January 2010
Line 79:		Line 79:
	* What's in here? Things for client or bobsced0?		* What's in here? Things for client or bobsced0?

			=Known Error Messages=
			Please refer here if you encounter an error message on BobSCEd that you cannot handle.
			==RLIMIT_MEMLOCK==
			libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
			This will severely limit memory registrations.--------------------------------------------------------------------------
			The OpenFabrics (openib) BTL failed to initialize while trying to
			allocate some locked memory. This typically can indicate that the
			memlock limits are set too low. For most HPC installations, the
			memlock limits should be set to "unlimited". The failure occured
			here:

			Local host: bobsced0
			OMPI source: btl_openib_component.c:1040
			Function: ompi_free_list_init_ex_new()
			Device: mthca0
			Memlock limit: 32768

			You may need to consult with your system administrator to get this
			problem fixed. This FAQ entry on the Open MPI web site may also be
			helpful:

			http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
			--------------------------------------------------------------------------
			--------------------------------------------------------------------------
			WARNING: There was an error initializing an OpenFabrics device.

			Local host: bobsced0
			Local device: mthca0
			--------------------------------------------------------------------------

	=References=		=References=

Leemasa: /* Generating an up-to-date machinefile */

2010-01-03T19:28:21Z

Generating an up-to-date machinefile

← Older revision		Revision as of 19:28, 3 January 2010
Line 62:		Line 62:
	== Generating an up-to-date machinefile ==		== Generating an up-to-date machinefile ==
	Ethernet:		Ethernet:
	cluster-fork /sbin/ifconfig -a \| grep -1 Ethernet \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > ~~bobsced.eth_hosts~~		cluster-fork /sbin/ifconfig -a \| grep -1 Ethernet \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bs-eth-hosts
	Infiniband:		Infiniband:
	cluster-fork /sbin/ifconfig -a \| grep -1 UNSPEC \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > ~~bobsced.eth_hosts~~		cluster-fork /sbin/ifconfig -a \| grep -1 UNSPEC \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bs-ib-hosts

	Notice that the only difference is the search field in the first grep command. UNSPEC here refers to Infiniband.		Notice that the only difference is the search field in the first grep command. UNSPEC here refers to Infiniband.

Leemasa: /* Generating an up-to-date machinefile for BobSCEd over Ethernet and Infiniband. */

2010-01-03T19:24:52Z

Generating an up-to-date machinefile for BobSCEd over Ethernet and Infiniband.

← Older revision		Revision as of 19:24, 3 January 2010
Line 60:		Line 60:
	* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]		* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]

	== Generating an up-to-date machinefile ~~for BobSCEd over Ethernet and Infiniband.~~ ==		== Generating an up-to-date machinefile ==
	Ethernet:		Ethernet:
	cluster-fork /sbin/ifconfig -a \| grep -1 Ethernet \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bobsced.eth_hosts		cluster-fork /sbin/ifconfig -a \| grep -1 Ethernet \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bobsced.eth_hosts

Leemasa at 19:24, 3 January 2010

2010-01-03T19:24:22Z

← Older revision		Revision as of 19:24, 3 January 2010
Line 59:		Line 59:
	* [http://www.rocksclusters.org/rocks-documentation/4.2.1/faq-configuration.html#DISABLE-REINSTALL Official documentation]		* [http://www.rocksclusters.org/rocks-documentation/4.2.1/faq-configuration.html#DISABLE-REINSTALL Official documentation]
	* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]		* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]

			== Generating an up-to-date machinefile for BobSCEd over Ethernet and Infiniband. ==
			Ethernet:
			cluster-fork /sbin/ifconfig -a \| grep -1 Ethernet \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bobsced.eth_hosts
			Infiniband:
			cluster-fork /sbin/ifconfig -a \| grep -1 UNSPEC \| awk '{printf("%s slots=4\n",$2)}' \| cut -d : -f 2 > bobsced.eth_hosts

			Notice that the only difference is the search field in the first grep command. UNSPEC here refers to Infiniband.

	=General Info=		=General Info=

Lemanal: bobsced0 as dns server

2007-06-01T17:28:55Z

bobsced0 as dns server

← Older revision		Revision as of 17:28, 1 June 2007
Line 20:		Line 20:
	*Setup and test Infiniband fabric		*Setup and test Infiniband fabric
	*Why does hopper see an interface flap from bobsced0?		*Why does hopper see an interface flap from bobsced0?
			*bosced0 wants to be the DNS server for the compute nodes

	=Howtos=		=Howtos=

Charliep: /* Todo */

2007-06-01T17:14:57Z

Todo

← Older revision		Revision as of 17:14, 1 June 2007
Line 1:		Line 1:
	=Todo=		=Todo=
	==411 tools==		* 411 tools
	* fix ganglia to recognize broadcasts & update		** fix ganglia to recognize broadcasts & update
	==Naming scheme==		* Naming scheme
	* bs* vs compute-- vs c-		** bs* vs compute-- vs c-
	* This is terrible it needs work		** This is terrible it needs work
	==Updating bobsced0's RPM repo==		*Updating bobsced0's RPM repo
	* yum-- free		** yum-- free
	* up2date-- RHEL		** up2date-- RHEL
	* "Aborting the rocks-update tool while the tool is downloading RPMs might produce corrupted RPM packages (SDSC Toolkit)" from pr_troubleshooting.doc		** "Aborting the rocks-update tool while the tool is downloading RPMs might produce corrupted RPM packages (SDSC Toolkit)" from pr_troubleshooting.doc
	==NIS map==		*NIS map
	*<code>/etc/passwd & /etc/group</code> permissions		**<code>/etc/passwd & /etc/group</code> permissions
	==What broke <code>cluster-fork</code>?==		**An architecture without a variable amount of delay before BobSCEd is updated would be nice.
	==NAT & NFS==		*What broke <code>cluster-fork</code>?
	* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-October/021958.html Mailing list]		*NAT & NFS
	*[https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-May/018503.html Mailing list]		**[https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-October/021958.html Mailing list]
			**[https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-May/018503.html Mailing list]
			**We should consider flattening the network, that is moving everything into the 159.28.234/24 subnet.
			*Stop the hourly cron from producing output on stdout unless there is an error
			*Setup and test Infiniband fabric
			*Why does hopper see an interface flap from bobsced0?

	=Howtos=		=Howtos=

Lemanal: NAT & NFS added to todo

2007-06-01T16:44:43Z

NAT & NFS added to todo

← Older revision		Revision as of 16:44, 1 June 2007
Line 12:		Line 12:
	*<code>/etc/passwd & /etc/group</code> permissions		*<code>/etc/passwd & /etc/group</code> permissions
	==What broke <code>cluster-fork</code>?==		==What broke <code>cluster-fork</code>?==
			==NAT & NFS==
			* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-October/021958.html Mailing list]
			*[https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-May/018503.html Mailing list]

	=Howtos=		=Howtos=

Lemanal: cleanup

2007-06-01T15:07:32Z

cleanup

← Older revision		Revision as of 15:07, 1 June 2007
Line 1:		Line 1:
	=~~Bobsced~~=		=Todo=
			==411 tools==
			* fix ganglia to recognize broadcasts & update
			==Naming scheme==
			* bs* vs compute-- vs c-
			* This is terrible it needs work
			==Updating bobsced0's RPM repo==
			* yum-- free
			* up2date-- RHEL
			* "Aborting the rocks-update tool while the tool is downloading RPMs might produce corrupted RPM packages (SDSC Toolkit)" from pr_troubleshooting.doc
			==NIS map==
			*<code>/etc/passwd & /etc/group</code> permissions
			==What broke <code>cluster-fork</code>?==

	==~~Todo==~~		=Howtos=
	* 411 tools-- fix ganglia to recognize broadcasts & update
	~~* Naming scheme bs* vs compute-- vs c-~~
	** This is terrible it needs work
	* Updating bobsced0's RPM repo:
	** yum-- free
	** up2date-- RHEL
	** "Aborting the rocks-update tool while the tool is downloading RPMs might produce corrupted RPM packages (SDSC Toolkit)" from pr_troubleshooting.doc
	* NIS map & <code>/etc/passwd & /etc/group</code> permissions
	* What broke <code>cluster-fork</code>?

	~~==Howtos==~~		==Updating nodes to be kickstarted & adding new packages==

	===Updating nodes to be kickstarted & adding new packages===
	* bobsced0 can be updated by just installing rpms		* bobsced0 can be updated by just installing rpms
	* Check for an RPM in: <code>/state/partition1/home/install/rocks-dist/lan/x86_64/RedHat/RPMS/</code>		* Check for an RPM in: <code>/state/partition1/home/install/rocks-dist/lan/x86_64/RedHat/RPMS/</code>
Line 29:		Line 30:
	** <code>ssh â€“p 2200 compute-x-x</code>		** <code>ssh â€“p 2200 compute-x-x</code>

	=== Adding post install scripts to kickstart ===		== Adding post install scripts to kickstart ==
	* Edit <code>/home/install/site-profiles/4.1.1/nodes/extend-compute.xml</code>		* Edit <code>/home/install/site-profiles/4.1.1/nodes/extend-compute.xml</code>
	* Add a <code><post arch="x86_64"></code> entry i.e.:		* Add a <code><post arch="x86_64"></code> entry i.e.:
	** <code><post arch="x86_64">cp /cluster/ganglia/gmond.conf /etc/gmond.conf</post></code>		** <code><post arch="x86_64">cp /cluster/ganglia/gmond.conf /etc/gmond.conf</post></code>
	=== Using 411 tools ===
			== Using 411 tools ==
	* make -C /var/411 on bobsced0		* make -C /var/411 on bobsced0
	** Copies the files to /etc/411.d/ using 411put		** Copies the files to /etc/411.d/ using 411put
Line 39:		Line 41:
	** The files that are watched can be updated by changing the makefiles in /var/411/		** The files that are watched can be updated by changing the makefiles in /var/411/
	* <code>cluster-fork /opt/rocks/bin/411get --all</code>		* <code>cluster-fork /opt/rocks/bin/411get --all</code>
	=== cluster-fork ===
			== cluster-fork ==
	* Used to run commands on all cluster nodes like the c3tools		* Used to run commands on all cluster nodes like the c3tools
	** Broken, see todo		** Broken, see todo
	** Temporary fix: <code>cluster-fork --nodes="compute-0-%d:0-14" <command> </code>		** Temporary fix: <code>cluster-fork --nodes="compute-0-%d:0-14" <command> </code>

	=== disabling reinstall (kickstart) after hard reset ===		== disabling reinstall (kickstart) after hard reset ==
	* [http://www.rocksclusters.org/rocks-documentation/4.2.1/faq-configuration.html#DISABLE-REINSTALL Official documentation]		* [http://www.rocksclusters.org/rocks-documentation/4.2.1/faq-configuration.html#DISABLE-REINSTALL Official documentation]
	* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]		* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]

	==General Info==		=General Info=
	* NIS Importing		==NIS Importing==
	** <code>/etc/cron.hourly/importNIS.sh</code>		* <code>/etc/cron.hourly/importNIS.sh</code>
	** This comes from the rocks users guide & a mailing list thread.		* This comes from the rocks users guide & a mailing list thread.
	* http		==http==
	** <code>/cluster/www/bobsced/</code>		* <code>/cluster/www/bobsced/</code>
	* <code>/cluster</code>		==<code>/cluster</code>==
	** Mounted using <code>/etc/rc.local</code>		* Mounted using <code>/etc/rc.local</code>
	* <code>/cluster/bobsced/etc/</code>		==<code>/cluster/bobsced/etc/</code>==
	** What's in here? Things for client or bobsced0?		* What's in here? Things for client or bobsced0?


	==~~References~~==		=References=
	* Rocks ~~user's guide~~		==Rocks Documentation==
	** [http://www.rocksclusters.org/rocks-documentation/4.1/rocks-usersguide-4.1.pdf Rocks users guide pdf]		* [http://www.rocksclusters.org/rocks-documentation/4.1/rocks-usersguide-4.1.pdf Rocks users guide pdf]
	** [http://www.rocksclusters.org/rocks-documentation/4.1/ Online version]		* [http://www.rocksclusters.org/rocks-documentation/4.1/ Online version]
	* [http://www.dell.com/downloads/global/power/ps4q05-20050227-Ali.pdf Platform rocks]		* [http://www.dell.com/downloads/global/power/ps4q05-20050227-Ali.pdf Platform rocks]
			==Troubleshooting Platform Open Cluster Stack (OCS) and Platform Lava==
	* pr_troubleshooting.doc		* pr_troubleshooting.doc
			==411 Tools==
	* [http://www.rocksclusters.org/rocks-doc/papers/hpdc2005/hpdc2005-411.pdf 411tools]		* [http://www.rocksclusters.org/rocks-doc/papers/hpdc2005/hpdc2005-411.pdf 411tools]
	* [http://www.centos.org/docs/4/pdf/rhel-ig-x8664-multi-en.pdf RHEL] & [http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/pdf/rhel-isa-en.pdf More RHEL]		==RHEL==
			* [http://www.centos.org/docs/4/pdf/rhel-ig-x8664-multi-en.pdf RHEL]
			* [http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/pdf/rhel-isa-en.pdf More RHEL]

Lemanal: added info on disabling kickstart after hard reset

2007-06-01T14:14:25Z

added info on disabling kickstart after hard reset

← Older revision		Revision as of 14:14, 1 June 2007
Line 43:		Line 43:
	** Broken, see todo		** Broken, see todo
	** Temporary fix: <code>cluster-fork --nodes="compute-0-%d:0-14" <command> </code>		** Temporary fix: <code>cluster-fork --nodes="compute-0-%d:0-14" <command> </code>

			=== disabling reinstall (kickstart) after hard reset ===
			* [http://www.rocksclusters.org/rocks-documentation/4.2.1/faq-configuration.html#DISABLE-REINSTALL Official documentation]
			* [https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2006-December/022969.html From the mailing list]

	==General Info==		==General Info==