Sysadmin:ImportantInfo:PowerFailure

this page has been deprecated

Before a planned shutdown

1. Send an email to content at least a week in advance and get them to update the news section of the web page with the pertinent information (where/when/why).
2. Send an email to clients informing them of the shutdown at least two days in advance.
3. Schedule a shutdown of the ACL machines. Preferably this would be a day in advance, but certainly leave no less than 15 minutes of warning. Have the ACLs shutdown between 15 and 30 minutes prior to the planned shutdown. You can use the c3 tools on image to do this for the entire cluster.
4. Schedule a shutdown of the servers. This includes:
   1. quark
   2. quarkprime
   3. image
   4. backup
   5. logger
   6. millie

Again, for quark, this would preferably be at least a day in advance, but certainly no less than 15 minutes. Cut the planned shutdown of the servers as close to the planned shutdown time as possible, but be careful not to cut it too close. 5--10 minutes should be good.

Check list

1. Shut off the breakers by the CS rack to control the behavior of the network when power returns.

Notes

1. The UPS in the bottom of the CS rack provides the only backup power in the room. There are two rack-mounted PDUs backed by the UPS, one in the 4-post rack and one in the 2-post rack. These should not be shut off except in an emergency. These are labeled on the front to distinguish them from the mains-backed PDUs.
2. To conserve battery power, backup, image, and quarkprime shut themselves off automatically a few minutes after a loss of mains power. You should not turn these back on except in an emergency.

After power comes back on

1. Power up these servers in this order:
   1. quark
   2. quarkprime
   3. backup
   4. image

You can come back in the other servers later. These ones have special services (NIS and NFS in particular) that the other machines require.

1. Power up the ACLs.
2. Go back around and power up the other servers.
   1. millie
   2. bagend
   3. proto

Things to check

1. Sendmail: Make sure sendmail and MailScanner on quark are both running. If they need prodding, run /usr/local/etc/rc.d/mta.sh restart and /usr/local/etc/rc.d/mailscanner.sh restart.
2. Apache: Make sure Apache is running on quark. If it needs restarting, stop it with apachectl stop and restart it with apachectl startssl. Make sure to restart it with SSL enabled!

Problems you will run into

1. backup will need you to push enter when it comes up to clear an "error" in one of its SCSI drives.

Checklists (new!)

Shutdown

1. Shut off the breakers by the CS rack to control the behavior of the network when power returns.
2. Shut off the switches on the mains-backed (not UPS backed!) rack-mounted PDUs in all four racks.
3. Shut off the HVAC system.

Startup

1. Turn on all breakers.
2. Turn on HVAC system.
3. Turn on mains-backed CS PDUs.
   1. Turn on image if it's not on.
4. Turn on the PDU labeled "cluster" in the 4-post cluster rack.
   1. Turn on hopper and wait for it to settle.
   2. Turn on admin.
5. Turn on cairo PDUs.
6. Turn on bazaar PDUs.

Notes

1. The UPS in the bottom of the CS rack provides the only backup power in the room. There are two rack-mounted PDUs backed by the UPS, one in the 4-post rack and one in the 2-post rack. These should not be shut off except in an emergency. These are labeled on the front to distinguish them from the mains-backed PDUs.
2. To conserve battery power, backup, image, and quarkprime shut themselves off automatically a few minutes after a loss of mains power. You should not turn these back on except in an emergency.