Sysadmin

Machines and Brief Descriptions of Services

CS Machines

HOME (vm0)

Users SSH NFS

NET (vm1)

LDAP server DNS DHCP

WEB (vm2)

Mailman Mail Stack Apache2 PostgresQL MySQL Wiki

TOOLS (vm3)

SageNB Server Jupyterhub Server Software Modules NginX

BABBAGE

Firewall

PROTO

Weather Monitoring GPS/NTP Energy Monitoring

CONTROL

Users SSH

SHINKEN

Users SSH Add machines

MURPHY

Users SSH

Cluster Machines

HOPPER

Users SSH NFS Software Modules PostgresQL Wiki Apache2 DNS DHCP

DALI

Gitlab Backups NginX

AL-SALAM

WebMO Software Modules Apache2

LAYOUT

Jupyterhub Server Software Modules NginX Apache2 WebMO

BRONTE

Software Modules

POLLOCK

Software Modules WebMO NginX

KAHLO

Backups NginX

BIGFE

Software Modules

T-VOC

Software Modules

ELWOOD

Software Modules

Switches

SG538SF02J

HP Procurve 3400cl Ports: 24 Backplane bandwidth: 88 Gbps 64 million pps Memory: 2MB packet buffer 16 MB dual flash 128 MB SDRAM Cut-through switching: No Unused as of May 12, 2017

CN63FP762S

HP 2530-24G Ports: 24 Switching Capacity: 56 Gbps 41.6 million pps Memory: 1.5 MB packet buffer 256 MB flash 128 MB DDR3 DIMM Cut-through switching: No On Al-Salam as of May 12, 2017

SG525SG025

HP Procurve 3400cl Ports: 24 Backplane bandwidth: 88 Gbps 64 million pps Memory: 2MB packet buffer 16 MB dual flash 128 MB SDRAM Cut-through switching: No On layout and whedon as of May 12, 2017

Netgear JGS524

Current cluster head-node Unmanaged (no console/configuration) Ports: 24 Switching bandwidth: 48 Gbps 1.5 million pps Memory: 2MB packet buffer Cut-through switching: No Connected to Al-Salam, Hopper, Pollock, Nagios, Dali, Kahlo, Bronte as of May 12, 2017

HP 5920

Ports: 24 Backplane bandwidth: 480 Gbps 367 million pps Memory: 3.6 GB packet buffer 256 MB dual flash 2 GB SDRAM Cut-through switching: Yes Connected to layout, kahlo, and dali as of May 12, 2017

HP 5500

Model: JG542A Ports: 24 Backplane bandwidth: 224 Gbps 166.6 million pps Memory: 6 MB packet buffer 512 MB dual flash 1 GB SDRAM Cut-through switching: No Connected to Babbage, Control, Nagios, and the netgear switch as of May 12, 2017

Systems Administration Documentation

For old documentation, see: Old Wiki Information

Admin Tasks Nagios Monitoring Shinken Monitoring Upgrading SSL Certificates User Management Installing software under modules Backup Contacting all users Welcoming a new sysadmin to the fold Add a computer Setting up Lovelace Lab Machines Reset password

Services Apache2 Databases DNS and DHCP Virtualization Xen Server

Miscellaneous Shutdown and Boot up Contact Information Phone Numbers Authentication Information UPS Generating SSL Certificates Power draws Working with Sun Hardware Sysadmin:Passwords Patching Linux Kernel Patching Cable Ends NEW Virtualization Comparison

Networking Network Layout (as of 08/2006) D224 cable plant Fiber plans Switches Rack notes Public Network Old Network Topo Figures Network layout (May 2007) Alt Network path Sysadmin:UPS Setup

Current Projects (updated 2017-04-27)

TODO

• Layout infiniband subnet manager
• Layout disk swap, new lo0
• HP Al-Salam switch enable jumboframes

Ongoing Projects (Spring 2017)

TODO

• EMAILING ALL THE USERS https://wiki.cs.earlham.edu/index.php/Sysadmin:Old:Contacting_All_Users
• SHUTDOWN SCHEDULED FOR SUNDAY (APRIL 16)
  • Check/update instructions - one version is at https://wiki.cs.earlham.edu/index.php/Sysadmin:ImportantInfo:PowerFailure, there are others too
  • Notify users
• Fix certs for gitlab, etc.
• Secure 1-2 admins for the summer
• Prep layout for May-June usage
• Practice shutdown-startup procedure (with Michael)
• Nsswitch consistency across all machines
• Document tools: startup / shutdown - Charlie
• Use Sysadmin namespace for all our pages - All
  • Testing usefulness of documentation - Dave
• Al Salam: configure switch, re-rack. - Vitalii
  • HP switch should be reset and tested.
• LDAP cleanup of system users / old groups - James
• Layout - Nirdesh
  • Lo0 RAID (mdadm)
  • 10GB from Dali to lo0 (adding rules on compute node routing tables as a possible fix)
  • BIOS reset
• 10Gb, perfsonar, ...
• Monitoring: (Ganglia, Shinken)
  • Getting consistency among all the machines(check_nrpe regularly stops working).
• Whedon: configured and available
• Change passwords (on everything). Postgres, shenken, ...
• Webcam on office whiteboard (new office location?)
• Learn virtual machine architecture and modules - Dave
  • Document in a format for future admin training?
  • Find existing introduction material
• Mirror control for testing, swapping, etc.

DONE (19 Jan 2017)

• Examine extra "layout" node. - Adam
  • Differences are: Single PSU, Single GPGPU, No VGA.
  • It has Infiniband and 10GB cards installed.
• Networking - Adam, Charlie
  • IP over Infiniband working on layout
    • Resolved by resetting IB switch configuration: ibwarn: [3349] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 1)

FUTURE

• Centralized password database / manager / location

Current Projects (updated 13 Oct 16)

• Groups and LDAP and sudo - James
• Amber - James
• Edward's setup - Vitalli
• WebDev access - Nirdesh
• Puppet - James and Vitalii
• Bacula - Nirdesh
• SSL certificate upgrade and documentation - Kristin
• Listserv merging with archives preserved - Nirdesh
• Ganglia - Bret
• Shenken - Vitalii
  • latency, UPS
• New Layout node - ? and ?
• Provision Sappho (compute) - after Puppet
• Provision Kahlo (storage) -
  • replace broken drive
• I2 setup
  • DTN, storage nodes, head nodes, ports in CST
• Provision Whedon (compute) - after Puppet
• Shutdown and startup test - scheduled for Sunday 27 November
• Disk cleaning - Charlie
• Password changing in the CS and cluster domains - Vitalii and James
• Proto setup and maintenance with HIP/Green Science