ShutdownProcedure - Revision history

Craigje: /* File system mounting */

2019-08-19T14:04:41Z

File system mounting

← Older revision		Revision as of 14:04, 19 August 2019
Line 125:		Line 125:

	# Make sure /smiley-home-disk is mounted on smiley. (If not, run <code>mount --source=/dev/vmdata/eccs-home-disk --target=/smiley-eccs-home-disk</code>)		# Make sure /smiley-home-disk is mounted on smiley. (If not, run <code>mount --source=/dev/vmdata/eccs-home-disk --target=/smiley-eccs-home-disk</code>)
			## If you get the message <tt>requested nfs version or transport protocol not supported</tt> at any point when running <code>mount -a</code>, run <code>service nfs-server start</code> on smiley and try again.
	# Launch net (see below). On the Xen console, check that all file systems are correct there. Do not move on to other VM's until net is up, running, and mounting file systems.		# Launch net (see below). On the Xen console, check that all file systems are correct there. Do not move on to other VM's until net is up, running, and mounting file systems.
	# Bring up tools and web. Make sure they're stable and have each mounted file systems.		# Bring up tools and web. Make sure they're stable and have each mounted file systems.

Craigje at 18:43, 25 April 2019

2019-04-25T18:43:06Z

← Older revision		Revision as of 18:43, 25 April 2019
Line 124:		Line 124:
	In order:		In order:

	# Make sure /smiley-home-disk is mounted on smiley.		# Make sure /smiley-home-disk is mounted on smiley. (If not, run <code>mount --source=/dev/vmdata/eccs-home-disk --target=/smiley-eccs-home-disk</code>)
	# Launch net (see below). On the Xen console, check that all file systems are correct there. Do not move on to other VM's until net is up, running, and mounting file systems.		# Launch net (see below). On the Xen console, check that all file systems are correct there. Do not move on to other VM's until net is up, running, and mounting file systems.
	# Bring up tools and web. Make sure they're stable and have each mounted file systems.		# Bring up tools and web. Make sure they're stable and have each mounted file systems.

Craigje: /* Mounting Logical Volumes */

2019-04-23T13:20:23Z

Mounting Logical Volumes

← Older revision		Revision as of 13:20, 23 April 2019
Line 116:		Line 116:
	console-# vgchange -a y		console-# vgchange -a y
	</pre>		</pre>
	This should be done at boot using <code>/etc~~/init.d~~/rc.~~sysinit~~</code> ~~but there still might be some subtleties there~~.		This should be done at boot using <code>/etc/rc.local</code>. Verify this by running lvdisplay and creating the VM's.

	==== File system mounting ====		==== File system mounting ====

Craigje: /* File system mounting */

2019-04-10T14:34:46Z

File system mounting

← Older revision		Revision as of 14:34, 10 April 2019
Line 123:		Line 123:

	In order:		In order:
	1. Make sure /smiley-home-disk is mounted on smiley.
	2. Launch net (see below). On the Xen console, check that ~~everything is right~~ there. Do not move on to other VM's until net is up, running, and mounting file systems.		# Make sure /smiley-home-disk is mounted on smiley.
	3. Bring up tools and web. Make sure they're stable and have each mounted file systems.		# Launch net (see below). On the Xen console, check that all file systems are correct there. Do not move on to other VM's until net is up, running, and mounting file systems.
			# Bring up tools and web. Make sure they're stable and have each mounted file systems.

	==== Starting VMs ====		==== Starting VMs ====

Craigje at 18:46, 8 April 2019

2019-04-08T18:46:59Z

← Older revision		Revision as of 18:46, 8 April 2019
Line 29:		Line 29:
	* to get to <code>sysadmin@smiley</code> first <code>ssh</code> into <code>home</code> or <code>hopper</code>.		* to get to <code>sysadmin@smiley</code> first <code>ssh</code> into <code>home</code> or <code>hopper</code>.
	* '''Make sure all virtual machines are shut down before restarting the bare metal hardware'''		* '''Make sure all virtual machines are shut down before restarting the bare metal hardware'''
	* This is a good time to test scripts relating to power, which are otherwise testable mostly in a local environment made for that purpose.

	== Cluster ==		== Cluster ==
Line 54:		Line 53:

	The reverse of shutdown, again make sure to wait before proceeding at the appropriate steps.		The reverse of shutdown, again make sure to wait before proceeding at the appropriate steps.

			It is important for file system reasons to make sure hopper is up and stable before proceeding to the other machines. Likewise, make sure dali and kahlo are stable before proceeding to clusters, utilities, and phat nodes.

	=== Hadoop ===		=== Hadoop ===
Line 87:		Line 88:
	=== Order of shutdowns ===		=== Order of shutdowns ===

	~~# proto (lives seperatly, <code>ssh admin@proto.cs.earlham.edu</code>)~~
	# tools		# tools
	# web		# web
Line 124:		Line 124:
	In order:		In order:
	1. Make sure /smiley-home-disk is mounted on smiley.		1. Make sure /smiley-home-disk is mounted on smiley.
	2. Launch net (see below). On the console, ~~make sure it~~'s ~~stable~~ and ~~has mounted~~ file systems.		2. Launch net (see below). On the Xen console, check that everything is right there. Do not move on to other VM's until net is up, running, and mounting file systems.
	3. Bring up tools and web. Make sure they're stable and have each mounted file systems.		3. Bring up tools and web. Make sure they're stable and have each mounted file systems.

Craigje at 18:43, 8 April 2019

2019-04-08T18:43:06Z

← Older revision		Revision as of 18:43, 8 April 2019
Line 46:		Line 46:
	# wait until everything up to this point has shutdown		# wait until everything up to this point has shutdown
	# hopper		# hopper

			=== File system mounting ===

			We consistently run into file system mounting problems on shutdown day. Mostly this relates to mounts from dali and kahlo to hopper, and from hopper to dali or kahlo.

	=== Order to bring up ===		=== Order to bring up ===
Line 98:		Line 102:
	# xl destroy net.cs.earlham.edu		# xl destroy net.cs.earlham.edu
	</pre>		</pre>


	=== Start up again ===		=== Start up again ===
Line 112:		Line 117:
	</pre>		</pre>
	This should be done at boot using <code>/etc/init.d/rc.sysinit</code> but there still might be some subtleties there.		This should be done at boot using <code>/etc/init.d/rc.sysinit</code> but there still might be some subtleties there.

			==== File system mounting ====

			File system mounting is sometimes a problem on the CS side too, but for different reasons. The problem here only appears when we start up again.

			In order:
			1. Make sure /smiley-home-disk is mounted on smiley.
			2. Launch net (see below). On the console, make sure it's stable and has mounted file systems.
			3. Bring up tools and web. Make sure they're stable and have each mounted file systems.

	==== Starting VMs ====		==== Starting VMs ====

Lcruberl16: /* Order of shutdowns */

2019-04-07T14:11:33Z

Order of shutdowns

← Older revision		Revision as of 14:11, 7 April 2019
Line 39:		Line 39:
	# all head nodes: (layout, alsalam, whedon)		# all head nodes: (layout, alsalam, whedon)
	# pollock		# pollock
			# shinken
	# bronte + disk array		# bronte + disk array
	# wait until everything up to this point has shutdown		# wait until everything up to this point has shutdown

Craigje: /* During */

2019-04-04T13:48:02Z

During

← Older revision		Revision as of 13:48, 4 April 2019
Line 22:		Line 22:
	# Tidy things up.		# Tidy things up.
	# Bring everything back up.		# Bring everything back up.
	Suggestion: Don't do OS upgrades or ~~anything~~ during this process. Don't entangle startup issues with upgrade issues.		Suggestion: Don't do OS upgrades or similar during this process. Don't entangle startup issues with upgrade issues.

	== General Info ==		== General Info ==

Craigje: /* During */

2019-04-04T13:45:22Z

During

← Older revision		Revision as of 13:45, 4 April 2019
Line 22:		Line 22:
	# Tidy things up.		# Tidy things up.
	# Bring everything back up.		# Bring everything back up.
	~~# Eat lunch.~~
	Suggestion: Don't do OS upgrades or anything during this process. Don't entangle startup issues with upgrade issues.		Suggestion: Don't do OS upgrades or anything during this process. Don't entangle startup issues with upgrade issues.

Craigje: /* General Info */

2019-01-24T17:53:24Z

General Info

← Older revision		Revision as of 17:53, 24 January 2019
Line 30:		Line 30:
	* to get to <code>sysadmin@smiley</code> first <code>ssh</code> into <code>home</code> or <code>hopper</code>.		* to get to <code>sysadmin@smiley</code> first <code>ssh</code> into <code>home</code> or <code>hopper</code>.
	* '''Make sure all virtual machines are shut down before restarting the bare metal hardware'''		* '''Make sure all virtual machines are shut down before restarting the bare metal hardware'''
			* This is a good time to test scripts relating to power, which are otherwise testable mostly in a local environment made for that purpose.

	== Cluster ==		== Cluster ==