Sysadmin

Machines and Brief Descriptions of Services

CS Machines

Server layout as of May 2017

NET (vm1)

LDAP Server DNS DHCP Backup to Dali: etc, var

WEB (vm2)

Mailman Mail Stack Apache2 PostgresQL MySQL Wiki Backup to Dali: etc, var

TOOLS (vm3)

SageNB Server Jupyterhub Server Software Modules NginX SSH Users Backup to Dali: etc, var, mnts, sage

BABBAGE

Firewall

PROTO

Weather Monitoring GPS/NTP Energy Monitoring Backup to Dali: etc, var

CONTROL

Users SSH HOME TOOLS Backup to Dali: etc, var

SMILEY

XenDocs NET WEB NFS Backup to Dali: etc, var

SHINKEN

Users SSH Add machines

MURPHY

Elderly email stack Users SSH

HOME (vm0)

SSH NFS Backup to Dali: eccs, etc, var deprecated 07-2018

Cluster Machines

HOPPER

Users SSH NFS server LDAP server Software Modules PostgreSQL Wiki Apache2 DNS DHCP Backup to Dali: etc, var, cluster

INDIANA

New Storage Server

DALI

Storage Server Gitlab Backups NginX Backup to Dali (/media/r10_vol/backups/): etc, var/opt/gitlab/backups

AL-SALAM

WebMO Software Modules Apache2 Backup to Dali: etc, var

LAYOUT

Jupyterhub Server Software Modules NginX Apache2 WebMO Backup to Dali: etc, var

BRONTE

Software Modules Backup to Dali: etc, var, nbserver

POLLOCK

Software Modules WebMO NginX Backup to Dali: etc, var

KAHLO

Storage Server Backups NginX Backup to Dali: etc, var

BIGFE

Software Modules Hosts BCCD related repositories and distributions.

T-VOC

Software Modules

ELWOOD

Software Modules Used by BCCD to host www.bccd.net and www.littlefe.net. Will be deprecated when BCCD project offloads their sites onto cloud-based hosting platforms.

krasner

Docker platform on an old lovelace machine upgraded to have 16GB of RAM.

Switches

SG538SF02J

Model: HP Procurve 3400cl Ports: 24 Backplane bandwidth: 88 Gbps 64 million pps Memory: 2MB packet buffer 16 MB dual flash 128 MB SDRAM Cut-through switching: No Unused as of May 12, 2017

CN63FP762S

Model: HP 2530-24G Ports: 24 Switching Capacity: 56 Gbps 41.6 million pps Memory: 1.5 MB packet buffer 256 MB flash 128 MB DDR3 DIMM Cut-through switching: No Connected to Al-Salam as of May 12, 2017

SG525SG025

Model: HP Procurve 3400cl Ports: 24 Backplane bandwidth: 88 Gbps 64 million pps Memory: 2MB packet buffer 16 MB dual flash 128 MB SDRAM Cut-through switching: No Connected to layout and whedon as of May 12, 2017

Netgear JGS524

Current cluster head-node Unmanaged (no console/configuration) Ports: 24 Switching bandwidth: 48 Gbps 1.5 million pps Memory: 2MB packet buffer Cut-through switching: No Connected to Al-Salam, Hopper, Pollock, Nagios, Dali, Kahlo, Bronte as of May 12, 2017

cs-main

Model: HP 5920AF-24XG Ports: 24 Backplane bandwidth: 480 Gbps 367 million pps Memory: 3.6 GB packet buffer 256 MB dual flash 2 GB SDRAM Cut-through switching: Yes IP Address: 159.28.31.66 Connected to layout, kahlo, and dali as of May 12, 2017

5500denniscs-sw1

Model: HP 5500 JG542A Ports: 24 Backplane bandwidth: 224 Gbps 166.6 million pps Memory: 6 MB packet buffer 512 MB dual flash 1 GB SDRAM Cut-through switching: No IP Address: 159.28.31.67 Connected to Babbage, Control, Nagios, and the cluster's netgear switch (via port 14) as of May 12, 2017

Systems Administration Documentation

For old documentation, see: Old Wiki Information

Admin Tasks Nagios Monitoring Shinken Monitoring Upgrading SSL Certificates User Management Installing software under modules Backup Contacting all users Welcoming a new sysadmin to the fold Add a computer Setting up Lovelace Lab Machines Reset password

Services Cluster Overview Apache2 Databases DNS and DHCP Virtualization Xen Server

Miscellaneous Shutdown and Boot up Contact Information Phone Numbers Authentication Information UPS Generating SSL Certificates Power draws Working with Sun Hardware Sysadmin:Passwords Patching Linux Kernel Patching Cable Ends NEW Virtualization Comparison

Networking Network Layout (as of 08/2006) D224 cable plant Fiber plans Switches Rack notes Public Network Old Network Topo Figures Network layout (May 2007) Alt Network path Sysadmin:UPS Setup

Current Projects

This is the list we will work from in addition to service requests.

• Possibly: update CentOS on Pollock and Bronte
• Mailing list archives are gone
• Graceful shutdown during outage
• Web logins
• Reinstall OS's on Lovelace and Hamilton machines
  • Hamilton: installed, needs config

16 July 2018

• 159.28.23.26 is a ghost machine - it responds to ping and purportedly exists, but we are unsure where it is or what it is. - ping -f
• Power map additions and updates
• nbgrader - setup continues
• indiana config
• shinken - didn't boot, and the whole two sets of processes bit
• File:Manual TeraStation5010.pdf

Spring 2018

• Adam and Ahsan will be there for Noyes room tour on Friday, April 20 at 08:00
• Setup and install new machines in Lovelace - Eli
  • New machine has Debian installed, needs ethernet driver installed though
• 10 Gbps for dali and kahlo, hopper, etc (nfs mounts in /etc/fstab) - Adam
  • real close, reboot test 10a 8 April
• Chau & Laurence will be brought up to speed for Layout - Adam
  • "Layout needs to be brought out of surgery" - Charlie
  • Layout - head node swap, check disk space on /scratch - Adam
• APC is on Shinken now - Alek
• Postgres connection monitoring on Shinken - Vitalli
• Figure out the cs web production and test solution - Chau
• Upgrade t-voc - ?

• -------------------------
• Backup - max disk capacity, scripts on all machines backing-up at least /etc; babbage - Chau
• Ganglia on Hopper - Ahsan
• UPS load monitoring - Eli
  • One UPS is being monitored, more can be added and stuff can still be cleared up
• Temperature and humidity monitoring in the machine room (new item)
• Fixing the user create script
• Password auditing script

• -------------------------
• FIFO for requests rather than ad-hoc
• Accounting for hours logged
• PBS Shinken monitoring
• Power layout - legend, color-code servers by type, how are servers with 2x power supplies plumbed?

• -------------------------
• Shinken - Vitalii and Aleks (documentation, monitoring webmo and pm8)
• Hadoop on Whedon - Vitalii and Adam (stuck on ?)
• Layout - Adam (stuck on ldap)
• Gaussian & WebMO on Whedon - Ahsan and Eli (stuck on firewall)
• Backup - Châu (moving along, setup backup.cs.e.e next)
  • Needs to setup a machine with CentOs in Noyes basement, eventually run it over to Lilly - Laurence
• Installing power monitor, etc. and rack cleanup - TO BE ASSIGNED (eli and charlie)
  • switch switch (charlie to check inventory)
• Mothur - Ahsan
• password policy, force change and random initial
  • for now notify people with default and then change after a couple of days; script will generate random string
• Talk about at next meeting:
  • Spring break and summer people (important)
  • Jon's user and Postgres database
  • investigate tools /clients/ directory with what looks like duplicate user directories

(list from 2017-10-26)

• Finish migrating tools and home to smiley migrate web and net back to control
  • Record consistent & thorough documentation, especially concerning the startup and shutdown of the VMs
• Setup graceful shutdown when we detect to be running solely off UPS
  • Additionally, setup clean shutdown and startup for VMs on smiley and control (?)
• Fix reverse lookup error for mail.cs.earlham.edu
  • Should consistently refer to 159.28.22.2 (web.cs.earlham.edu)
  • It's possible that this isn't actually broken.
• Layout infiniband subnet manager
• Layout disk swap, new lo0
  • Redo /scratch for mglerner group on /media/r10_vol ?
• Migrate Elwood, BigFe, t-voc to repurposed Lovelace Machines (Eli)
• HP Al-Salam switch enable jumboframes ?
• Strike unused lovelace machine addresses from CS DNS file
  • Perhaps there's a python file in root's home somewhere that checks for unused DNS/DHCP addresses?

TODO

• EMAILING ALL THE USERS https://wiki.cs.earlham.edu/index.php/Sysadmin:Old:Contacting_All_Users
• SHUTDOWN SCHEDULED FOR SUNDAY (APRIL 16)
  • Check/update instructions - one version is at https://wiki.cs.earlham.edu/index.php/Sysadmin:ImportantInfo:PowerFailure, there are others too
  • Notify users
• Fix certs for gitlab, etc.
• Secure 1-2 admins for the summer
• Prep layout for May-June usage
• Practice shutdown-startup procedure (with Michael)
• Nsswitch consistency across all machines
• Document tools: startup / shutdown - Charlie
• Use Sysadmin namespace for all our pages - All
  • Testing usefulness of documentation - Dave
• Al Salam: configure switch, re-rack. - Vitalii
  • HP switch should be reset and tested.
• LDAP cleanup of system users / old groups - James
• Layout - Nirdesh
  • Lo0 RAID (mdadm)
  • 10GB from Dali to lo0 (adding rules on compute node routing tables as a possible fix)
  • BIOS reset
• 10Gb, perfsonar, ...
• Monitoring: (Ganglia, Shinken)
  • Getting consistency among all the machines(check_nrpe regularly stops working).
• Whedon: configured and available
• Change passwords (on everything). Postgres, shenken, ...
• Webcam on office whiteboard (new office location?)
• Learn virtual machine architecture and modules - Dave
  • Document in a format for future admin training?
  • Find existing introduction material
• Mirror control for testing, swapping, etc.

DONE (19 Jan 2017)

• Examine extra "layout" node. - Adam
  • Differences are: Single PSU, Single GPGPU, No VGA.
  • It has Infiniband and 10GB cards installed.
• Networking - Adam, Charlie
  • IP over Infiniband working on layout
    • Resolved by resetting IB switch configuration: ibwarn: [3349] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 1)

FUTURE

• Centralized password database / manager / location

Current Projects (updated 13 Oct 16)

• Groups and LDAP and sudo - James
• Amber - James
• Edward's setup - Vitalli
• WebDev access - Nirdesh
• Puppet - James and Vitalii
• Bacula - Nirdesh
• SSL certificate upgrade and documentation - Kristin
• Listserv merging with archives preserved - Nirdesh
• Ganglia - Bret
• Shenken - Vitalii
  • latency, UPS
• New Layout node - ? and ?
• Provision Sappho (compute) - after Puppet
• Provision Kahlo (storage) -
  • replace broken drive
• I2 setup
  • DTN, storage nodes, head nodes, ports in CST
• Provision Whedon (compute) - after Puppet
• Shutdown and startup test - scheduled for Sunday 27 November
• Disk cleaning - Charlie
• Password changing in the CS and cluster domains - Vitalii and James
• Proto setup and maintenance with HIP/Green Science