Sysadmin: Difference between revisions

From Earlham CS Department
Jump to navigation Jump to search
← Older editNewer edit →
Craigje (talk | contribs)
administrator, Bureaucrats, Administrators
1,085 edits
Craigje (talk | contribs)
administrator, Bureaucrats, Administrators
1,085 edits
Line 365: Line 365:
** Hamilton: installed, can ssh, so can do whatever else we need
** Hamilton: installed, can ssh, so can do whatever else we need
** Lovelace machine: still no Internet
** Lovelace machine: still no Internet
* shinken: web is back, need to check status
* layout node outside machine room - Eli interested
* layout: make different node the head node?
* passwords
* email to users (automated)
* check Lovelace DNS file - check what machines we have vs. what's in the DNS file?
* check backup statuses of machines, see the wiki page on this - also backup.cs.e.e (indiana?)
* Password auditing script
* FIFO for requests rather than ad-hoc
* Accounting for hours logged


=== Elderly, to be sorted through ===
=== From the summer, possible followup ===
===16 July 2018===
* 159.28.23.26 is a ghost machine - it responds to ping and purportedly exists, but we are unsure where it is or what it is. - ping -f - Eli looking into it in the context of al-salam
* 159.28.23.26 is a ghost machine - it responds to ping and purportedly exists, but we are unsure where it is or what it is. - ping -f  
* Power map additions and updates  
* Power map additions and updates  
* nbgrader - setup continues  
* nbgrader - setup continues  
* indiana config  
* indiana config  
* shinken - didn't boot, and the whole two sets of processes bit
* Hadoop on Whedon - Vitalii and Adam (stuck on ?) - talk to Ajit
* [[File:Manual TeraStation5010.pdf]]
 
===Spring 2018===
* Adam and Ahsan will be there for Noyes room tour on Friday, April 20 at 08:00
* Setup and install new machines in Lovelace - Eli
** New machine has Debian installed, needs ethernet driver installed though
* 10 Gbps for dali and kahlo, hopper, etc (nfs mounts in /etc/fstab) - Adam
** real close, reboot test 10a 8 April
* Chau & Laurence will be brought up to speed for Layout - Adam
** "Layout needs to be brought out of surgery" - Charlie
** Layout - head node swap, check disk space on /scratch - Adam
* APC is on Shinken now - Alek
* Postgres connection monitoring on Shinken - Vitalli
* Figure out the cs web production and test solution - Chau
* Upgrade t-voc - ?
 
* -------------------------
* Backup - max disk capacity, scripts on all machines backing-up at least /etc; babbage - Chau
* Ganglia on Hopper - Ahsan
* <s> UPS load monitoring </s> - Eli
** One UPS is being monitored, more can be added and stuff can still be cleared up
* Temperature and humidity monitoring in the machine room (new item)
* Fixing the user create script
* Password auditing script
 
* -------------------------
* FIFO for requests rather than ad-hoc
* Accounting for hours logged
* PBS Shinken monitoring
* Power layout - legend, color-code servers by type, how are servers with 2x power supplies plumbed?
 
* -------------------------
* Shinken - Vitalii and Aleks (documentation, monitoring webmo and pm8)
* Hadoop on Whedon - Vitalii and Adam (stuck on ?)
* Layout - Adam (stuck on ldap)
* Gaussian & WebMO on Whedon - Ahsan and Eli (stuck on firewall)
** Needs to setup a machine with CentOs in Noyes basement, eventually run it over to Lilly - Laurence
* <s> Installing power monitor, etc. and rack cleanup - TO BE ASSIGNED (eli and charlie) </s>
** <s> switch switch (charlie to check inventory) </s>
* Mothur - Ahsan
* <s> password policy, force change and random initial </s>
** <s> for now notify people with default and then change after a couple of days; script will generate random string </s>
* Talk about at next meeting:
** Spring break and summer people (important)
** Jon's user and Postgres database
** investigate tools /clients/ directory with what looks like duplicate user directories
 
=== (list from 2017-10-26) ===
* <s>Finish migrating tools and home to smiley</s> migrate web and net back to control
** Record consistent & thorough documentation, especially concerning the startup and shutdown of the VMs
* Setup graceful shutdown when we detect to be running solely off UPS
** Additionally, setup clean shutdown and startup for VMs on smiley and control (?)
* <s> Fix reverse lookup error for mail.cs.earlham.edu
** Should consistently refer to 159.28.22.2 (web.cs.earlham.edu)
** It's possible that this isn't actually broken. </s>
* Layout infiniband subnet manager
* Layout disk swap, new lo0
** <s>Redo /scratch for mglerner group on /media/r10_vol</s> ?
* Migrate Elwood, BigFe, t-voc to repurposed Lovelace Machines (Eli)
* <s>HP Al-Salam switch enable jumboframes</s> ?
* Strike unused lovelace machine addresses from CS DNS file
** Perhaps there's a python file in root's home somewhere that checks for unused DNS/DHCP addresses?
 
=== Ongoing Projects (Spring 2017) ===
=== TODO ===
* EMAILING ALL THE USERS https://wiki.cs.earlham.edu/index.php/Sysadmin:Old:Contacting_All_Users
* SHUTDOWN SCHEDULED FOR SUNDAY (APRIL 16)
** Check/update instructions - one version is at https://wiki.cs.earlham.edu/index.php/Sysadmin:ImportantInfo:PowerFailure, there are others too
** Notify users
* Fix certs for gitlab, etc.
* Secure 1-2 admins for the summer
* Prep layout for May-June usage
* Practice shutdown-startup procedure (with Michael)
* Nsswitch consistency across all machines
* Document tools: startup / shutdown - Charlie
* Use Sysadmin namespace for all our pages - All
** Testing usefulness of documentation - Dave
* Al Salam: configure switch, re-rack. - Vitalii
** HP switch should be reset and tested.
* LDAP cleanup of system users / old groups - James
* Layout - Nirdesh
** Lo0 RAID (mdadm)
** 10GB from Dali to lo0 (adding rules on compute node routing tables as a possible fix)
** BIOS reset
* 10Gb, perfsonar, ...
* Monitoring: (Ganglia, Shinken)
** Getting consistency among all the machines(check_nrpe regularly stops working).
* Whedon: configured and available
* Change passwords (on everything). Postgres, shenken, ...
* Webcam on office whiteboard (new office location?)
* Learn virtual machine architecture and modules - Dave
** Document in a format for future admin training?
** Find existing introduction material
* Mirror ''control'' for testing, swapping, etc.
 
=== DONE (19 Jan 2017) ===
* Examine extra "layout" node. - Adam
** Differences are: Single PSU, Single GPGPU, No VGA.
** It has Infiniband and 10GB cards installed.
* Networking - Adam, Charlie
** IP over Infiniband working on layout
*** Resolved by resetting IB switch configuration: <code>ibwarn: [3349] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 1)</code>
 
=== FUTURE ===
* Centralized password database / manager / location
 
=== Current Projects (updated 13 Oct 16) ===
* '''Groups and LDAP and sudo - James'''
* <s>Amber - James</s>
* <s>Edward's setup - Vitalli</s>
* <s>WebDev access - Nirdesh<s>
* Puppet - James and Vitalii
* '''Bacula - Nirdesh'''
* SSL certificate upgrade and documentation - Kristin
* <s>Listserv merging with archives preserved - Nirdesh </s>
* '''Ganglia - Bret'''
* '''Shenken - Vitalii'''
** latency, UPS
* New Layout node - ? and ?
* Provision Sappho (compute) - after Puppet
* Provision Kahlo (storage) -
** replace broken drive
* I2 setup
** DTN, storage nodes, head nodes, ports in CST
* [[Sysadmin:WhedonProvisioning|Provision Whedon]] (compute) - after Puppet
* '''Shutdown and startup test - scheduled for Sunday 27 November'''
* Disk cleaning - Charlie
* <s>Password changing in the CS and cluster domains - Vitalii and James</s>
* Proto setup and maintenance with HIP/Green Science

Revision as of 18:52, 6 August 2018


Machines and Brief Descriptions of Services

CS Machines

Server layout as of May 2017
NET
(vm1)
LDAP Server
DNS
DHCP

Backup to Dali: etc, var
WEB
(vm2)
Mailman
Mail Stack
Apache2
PostgresQL
MySQL
Wiki

Backup to Dali: etc, var
TOOLS
(vm3)
SageNB Server
Jupyterhub Server
Software Modules
NginX
SSH
Users

Backup to Dali: etc, var, mnts, sage
BABBAGE
Firewall
PROTO
Weather Monitoring
GPS/NTP
Energy Monitoring

Backup to Dali: etc, var
CONTROL
Users
SSH
HOME
TOOLS

Backup to Dali: etc, var
SMILEY
XenDocs
NET
WEB
NFS

Backup to Dali: etc, var
SHINKEN
Users
SSH
Add machines
MURPHY
Elderly email stack
Users
SSH
HOME
(vm0)
SSH
NFS

Backup to Dali: eccs, etc, var

deprecated 07-2018














Cluster Machines

HOPPER
Users
SSH
NFS server
LDAP server
Software Modules
PostgreSQL
Wiki
Apache2
DNS
DHCP

Backup to Dali: etc, var, cluster
INDIANA
New Storage Server
DALI
Storage Server
Gitlab
Backups
NginX

Backup to Dali (/media/r10_vol/backups/): etc, var/opt/gitlab/backups
AL-SALAM
WebMO
Software Modules
Apache2

Backup to Dali: etc, var
LAYOUT
Jupyterhub Server
Software Modules
NginX
Apache2
WebMO

Backup to Dali: etc, var
BRONTE
Software Modules

Backup to Dali: etc, var, nbserver
POLLOCK
Software Modules
WebMO
NginX

Backup to Dali: etc, var
KAHLO
Storage Server
Backups
NginX

Backup to Dali: etc, var
BIGFE
Software Modules

Hosts BCCD related repositories and distributions.
T-VOC
Software Modules
ELWOOD
Software Modules

Used by BCCD to host www.bccd.net and www.littlefe.net. Will be deprecated when BCCD project offloads their sites onto cloud-based hosting platforms.
krasner
Docker platform on an old lovelace machine upgraded to have 16GB of RAM.


































Switches

SG538SF02J
  • Model: HP Procurve 3400cl
  • Ports: 24
  • Backplane bandwidth:
    • 88 Gbps
    • 64 million pps
  • Memory:
    • 2MB packet buffer
    • 16 MB dual flash
    • 128 MB SDRAM
  • Cut-through switching: No
  • Unused as of May 12, 2017
CN63FP762S
  • Model: HP 2530-24G
  • Ports: 24
  • Switching Capacity:
    • 56 Gbps
    • 41.6 million pps
  • Memory:
    • 1.5 MB packet buffer
    • 256 MB flash
    • 128 MB DDR3 DIMM
  • Cut-through switching: No
  • Connected to Al-Salam as of May 12, 2017
SG525SG025
  • Model: HP Procurve 3400cl
  • Ports: 24
  • Backplane bandwidth:
    • 88 Gbps
    • 64 million pps
  • Memory:
    • 2MB packet buffer
    • 16 MB dual flash
    • 128 MB SDRAM
  • Cut-through switching: No
  • Connected to layout and whedon as of May 12, 2017
Netgear JGS524
  • Current cluster head-node
  • Unmanaged (no console/configuration)
  • Ports: 24
  • Switching bandwidth:
    • 48 Gbps
    • 1.5 million pps
  • Memory:
    • 2MB packet buffer
  • Cut-through switching: No
  • Connected to Al-Salam, Hopper, Pollock, Nagios, Dali, Kahlo, Bronte as of May 12, 2017
cs-main
  • Model: HP 5920AF-24XG
  • Ports: 24
  • Backplane bandwidth:
    • 480 Gbps
    • 367 million pps
  • Memory:
    • 3.6 GB packet buffer
    • 256 MB dual flash
    • 2 GB SDRAM
  • Cut-through switching: Yes
  • IP Address: 159.28.31.66
  • Connected to layout, kahlo, and dali as of May 12, 2017
5500denniscs-sw1
  • Model: HP 5500 JG542A
  • Ports: 24
  • Backplane bandwidth:
    • 224 Gbps
    • 166.6 million pps
  • Memory:
    • 6 MB packet buffer
    • 512 MB dual flash
    • 1 GB SDRAM
  • Cut-through switching: No
  • IP Address: 159.28.31.67
  • Connected to Babbage, Control, Nagios, and the cluster's netgear switch (via port 14) as of May 12, 2017

























Systems Administration Documentation

For old documentation, see: Old Wiki Information

Current Projects

This is the list we will work from in addition to service requests.

  • Possibly: update CentOS on Pollock and Bronte
  • Mailing list archives are gone
  • Graceful shutdown during outage
  • Web logins
  • Reinstall OS's on Lovelace and Hamilton machines
    • Hamilton: installed, can ssh, so can do whatever else we need
    • Lovelace machine: still no Internet
  • shinken: web is back, need to check status
  • layout node outside machine room - Eli interested
  • layout: make different node the head node?
  • passwords
  • email to users (automated)
  • check Lovelace DNS file - check what machines we have vs. what's in the DNS file?
  • check backup statuses of machines, see the wiki page on this - also backup.cs.e.e (indiana?)
  • Password auditing script
  • FIFO for requests rather than ad-hoc
  • Accounting for hours logged

From the summer, possible followup

  • 159.28.23.26 is a ghost machine - it responds to ping and purportedly exists, but we are unsure where it is or what it is. - ping -f - Eli looking into it in the context of al-salam
  • Power map additions and updates
  • nbgrader - setup continues
  • indiana config
  • Hadoop on Whedon - Vitalii and Adam (stuck on ?) - talk to Ajit
Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Sysadmin&oldid=16651"