ShutdownProcedure
These are the shutdown and boot up instructions for CS and Cluster servers.
Originally published: 2017-04-18 Updated: 2017-06-24
Contents
General Info
- babbage should be the very last machine brought down
- to get to
sysadmin@control
firstssh
intohome
orhopper
. - Make sure all virtual machines are shut down before restarting the bare metal hardware
CS virtual machies
The different VMs mount from eachother, so just be patient and everything hopefully will work out.
Tools
We may have to restart nginx
, jupyter
, and sage
by hand. Using history | grep <command>
is helpful here. (make sure to grab the entire command including ampersand)
Jupyter
eccs-tools# nohup su -c "/mnt/lovelace/software/anaconda/envs/py35/bin/jupyterhub -f /etc/jupyterhub/jupyterhub_config.py --no-ssl" &
Sage
eccs-tools# nohup /home/sage/sage-6.8/sage --notebook=sagenb accounts=False automatic_login=False interface= port=8080 &
Cluster
order of shutdowns
- all compute nodes: (layout, alsalam, whedon) and t-voc, bigfe, elwood
- all head nodes: (layout, alsalam, whedon)
- pollock
- bronte + disk array
- wait until everything up to this point has shutdown
- dali
- kahlo
- wait until everything up to this point has shutdown
- hopper
order to bring up
The reverse of shutdown, again make sure to wait before proceeding at the appropriate steps.
CS
order of shutdowns
If hopper
is back online, ssh sysadmin@cluster.cs.earlham.edu
and then ssh sysadmin@control.cs.earlham.edu
. This way we can shutdown all the VMs directly without being knocked off line or being in the machine room.
Example for shutting down a machine on control
.
ssh sysadmin@home.cs.earlham.edu ssh sysadmin@control.cs.earlham.edu sudo su - control-# xm destroy <hostname>.cs.earlham.edu
List running VMs
control-# xm list
- proto (lives seperatly,
ssh admin@proto.cs.earlham.edu
) - murphy (
ssh admin@home.cs.earlham.edu
thenssh admin@murphy.cs.earlham.edu
) - tools
- web
- home
- net
- control (where xen runs net, home, web, tools as VMs)
- babbage (firewall)
order of bring up
Bring up control
first.
- Make sure all virtual machines are shut down before restarting the bare metal hardware*
Mounting Logical Volumes
When you reboot, the LVM volume groups and logical volumes may not be automatically enabled. To bring them back do
console-# vgchange -a y
This should be done at boot using /etc/init.d/rc.sysinit
but there still might be some subtleties there.
And then check the status of the volumes with
console-# vgscan console-# lvscan
You can use vgchange -a y <volume group name>
to enable volume groups and lvchange -a y <logical volume name>
to enable logical volumes.
If it's an issue, you'll need to enable the vmstore-shared
volume group as well as any logical volumes in there.
Starting VMs
The VMs on control
should be brought up in the reverse order they were shutdown. It is very important to bring up net
first.
control-# xm create -c /mnt/vmdata-shared/config/eccs-<hostname>.cfg # To exit to the hypervisor shell you can press Ctrl + ] # To start VM without going into console for boot messages, forego the -c
The configuration files used for starting the virtual machines are in /mnt/vmdata-shared/config/
. There are 4: eccs-home.cfg
, eccs-web.cfg
, eccs-net.cfg
, and eccs-tools.cfg
. There are backup copies in ~sysadmin/xen-configs
.
Connect to VM console
console-# xm console <hostname>.cs.earlham.edu
You'll need to make sure that their DNS resolver settings are correctly set. I've had trouble with them using the incorrect DNS server settings sometimes and I'm not sure if the issues are resolved or not.
-Eamon
Update 24/6/2017: The new procedure for starting and shutting down net and web are as follow.
To shut down:
# xl destroy net.cs.earlham.edu # xl destroy web.cs.earlham.edu
Idealy though, the VM's should be shutdown from inside (by ssh'ing into them). After that, run "xl list" to see if they're still listed as domains, then run the "xl destroy" commands as above if needed.
To start them up:
# xl create ~sysadmin/xen-configs/eccs-net.cfg # xl create ~sysadmin/xen-configs/eccs-web.cfg