Difference between revisions of "Sysadmin"

From Earlham CS Department
Jump to navigation Jump to search
 
(252 intermediate revisions by 15 users not shown)
Line 1: Line 1:
−
__NOTOC__
+
This is the hub for the CS sysadmins on the wiki.
  
−
= Machines and Brief Descriptions of Services =
+
= Overview =
−
== CS Machines ==
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:40px; width:150px; text-align:center; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-top:solid 5px #ADDFFF; border-bottom:solid 1px white; border-right:solid 5px      #ADDFFF; font-size:120%;" | HOME <br> (vm0)
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-bottom:solid 5px #ADDFFF; border-right:solid 5px #ADDFFF;" | Users <br> SSH <br> NFS
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
[https://docs.google.com/drawings/d/1XaULz5IxXV_BZQjrko3QJ8wV5aXsSTYcSWxxT49OyZk/edit If you're visually inclined, we have a colorful and easy-to-edit map of our servers here!]
−
| style="height:40px; width:150px; text-align:center; background-color:#54C571; border-left:solid 5px #54C571; border-top:solid 5px #54C571; border-bottom:solid 1px white; border-right:solid 5px #54C571; font-size:120%;" | NET <br> (vm1)
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#54C571; border-left:solid 5px #54C571; border-bottom:solid 5px #54C571; border-right:solid 5px #54C571;" | LDAP server <br> [[Sysadmin:DNS & DHCP | DNS]] <br> [[Sysadmin:DNS & DHCP | DHCP]]
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== Server room ==
−
| style="height:40px; width:150px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px #E77471; font-size:120%;" | WEB <br> (vm2)
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;" | Mailman <br> [[Sysadmin:Mail Stack | Mail Stack]]<br> Apache2 <br> PostgresQL <br> MySQL <br> Wiki
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
Our servers are in Noyes, the science building that predates the CST. For general information about the server room and how to use it, check out [[Sysadmin:Server Room|this page]].
−
| style="height:40px; width:150px; text-align:center; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-top:solid 5px #C38EC7; border-bottom:solid 1px white; border-right:solid 5px #C38EC7; font-size:120%;" | TOOLS <br> (vm3)
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-bottom:solid 5px #C38EC7; border-right:solid 5px #C38EC7;" | [[Sysadmin:SageNB Server | SageNB Server]] <br> [[Sysadmin:Jupyterhub Notebook Server | Jupyterhub Server]] <br> [[Sysadmin:Software Modules | Software Modules]] <br> NginX
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
Columns: machine name, IPs, type (virtual, metal), purpose, dies, cores, RAM
−
| style="height:55px; width:150px; text-align:center; background-color:#E3A869; border-left:solid 5px #E3A869; border-top:solid 5px #E3A869; border-bottom:solid 1px white; border-right:solid 5px #E3A869; font-size:120%;" | BABBAGE
 
−
|-
 
−
| style="height:135px; width:150px; background-color: #E3A869; border-left:solid 5px #E3A869; border-bottom:solid 5px #E3A869; border-right:solid 5px #E3A869;" | [[Sysadmin:Firewall | Firewall]]
 
−
|}
 
  
−
{|
+
== Compute Resources ==
−
| style="height:55px; width:150px; text-align:center; background-color:#EEDC82; border-left:solid 5px #EEDC82; border-top:solid 5px #EEDC82; border-bottom:solid 1px white; border-right:solid 5px #EEDC82; font-size:120%;" | [[Sysadmin:Servers:Proto | PROTO]]
 
−
|-
 
−
| style="height:135px; width:150px; background-color: #EEDC82; border-left:solid 5px #EEDC82; border-bottom:solid 5px #EEDC82; border-right:solid 5px #EEDC82;" | Weather Monitoring <br> GPS/NTP <br> Energy Monitoring
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
[https://wiki.cs.earlham.edu/index.php/Sysadmin:Computer_Resources Machines and VMs related information here!]
−
| style="height:40px; width:150px; text-align:center; background-color:#54C571; border-left:solid 5px #54C571; border-top:solid 5px #54C571; border-bottom:solid 1px white; border-right:solid 5px      #54C571; font-size:120%;" | CONTROL
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#54C571; border-left:solid 5px #54C571; border-bottom:solid 5px #54C571; border-right:solid 5px #54C571;" | Users <br> SSH
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== Network ==
−
| style="height:40px; width:150px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px      #E77471; font-size:120%;" | SHINKEN
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;" | Users <br> SSH <br> Add machines
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
We have two network fabrics linking the machines together. There are three subdomains.
−
| style="height:40px; width:150px; text-align:center; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-top:solid 5px #C38EC7; border-bottom:solid 1px white; border-right:solid 5px      #C38EC7; font-size:120%;" |MURPHY
 
−
|-
 
−
| style="height:135px; width:150px; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-bottom:solid 5px #C38EC7; border-right:solid 5px #C38EC7;" | Users <br> SSH
 
−
|}
 
  
 +
=== 10 Gb ===
  
 +
We have 10Gb fabric to mount files over NFS. Machines with 10Gb support have an IP address in the class C range 10.10.10.0/24 and we want to add DNS to these addresses.
  
 +
=== 1 Gb (cluster, cs) ===
  
 +
We have two class C subnets on the 1Gb fabric: 159.28.22.0/24 (CS) and 159.28.23.0/24 (cluster). This means we have double the IP addresses on the 1Gb fabric that we have on the 10Gb fabric.
  
 +
Any user accessing *.cluster.earlham.edu and *.cs.earlham.edu is making calls on a 1Gb network.
  
 +
=== Intra-cluster fabrics ===
  
 +
The layout cluster has an Infiniband infrastructure. Wachowski has only a 1Gb infrastructure.
  
 +
== Power ==
  
 +
We have a backup power supply, with batteries last upgraded in 2019 (?). We’ve had a few outages since then and power has held up well.
  
 +
== HVAC ==
  
 +
HVAC systems are static and are largely managed by Facilities.
  
 +
[[Topology|See full topology diagrams here.]]
  
 +
[[Sysadmin:Layers of abstraction for filesystems|A word about what's happening between files and the drives they live on.]]
  
−
== Cluster Machines ==
+
= New sysadmins =
  
−
{| style="float:left; margin-right:2px;"
+
These pages will be helpful for you if you're just starting in the group:
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px      #0099cc; font-size:120%;" | HOPPER
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" | Users <br> SSH <br> NFS <br> [[Sysadmin:Software Modules | Software Modules]] <br> PostgresQL <br> Wiki <br> Apache2 <br> [[Sysadmin:DNS & DHCP | DNS]] <br> [[Sysadmin:DNS & DHCP | DHCP]]
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
* [[Sysadmin:New Sysadmins | Welcoming a new sysadmin ]]
−
| style="height:55px; width:150px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | DALI
+
* [[Sysadmin:Troubleshooting|General troubleshooting tips for admins]]
−
|-
+
* [[Sandbox Notes|Sandbox Notes]]
−
| style="height:200px; width:150px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;" | [[Sysadmin:Gitlab | Gitlab]] <br> Backups <br> NginX
+
* [[Password managers]]
−
|}
+
* [[Server safety]]
 +
* [https://code.cs.earlham.edu/sysadmin/ticket-tracker Ticket tracking for current projects]
  
−
{| style="float:left; margin-right:2px;"
+
Note: you'll need to log in with wiki credentials to see most Sysadmin pages.
−
| style="height:55px; width:150px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | AL-SALAM
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;" | WebMO <br> [[Sysadmin:Software Modules | Software Modules]] <br> Apache2
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
= Additional information =
−
| style="height:55px; width:150px; text-align:center; background-color:#39ad39; border-left:solid 5px #39ad39; border-top:solid 5px #39ad39; border-bottom:solid 1px white; border-right:solid 5px #39ad39; font-size:120%;" | LAYOUT
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#39ad39; border-left:solid 5px #39ad39; border-bottom:solid 5px #39ad39; border-right:solid 5px #39ad39;" | [[Sysadmin:Jupyterhub Notebook Server | Jupyterhub Server]] <br> [[Sysadmin:Software Modules | Software Modules]] <br> NginX <br> Apache2 <br> WebMO
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
These pages contain a lot of the most important information about our systems and how we operate.
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | BRONTE
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]]
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
===Handy Tools===
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | POLLOCK
+
* [http://monitor.cluster.earlham.edu:8088/packages Porter's Package Explorer]
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]] <br> WebMO <br> NginX
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
===Technical docs===
−
| style="height:55px; width:150px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | KAHLO
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;" | Backups <br> NginX
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
* [https://code.cs.earlham.edu/sysadmin/ticket-tracker Ticket tracking for current projects]
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | BIGFE
+
* [[Server safety]]
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]]
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | T-VOC
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]]
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | ELWOOD
 
−
|-
 
−
| style="height:200px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]]
 
−
|}
 
−
 
 
−
 
 
−
 
 
−
 
 
−
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
== Switches ==
 
−
 
 
−
 
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px      #0099cc; font-size:120%;" | SG538SF02J
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc; font-size:80%;" |
 
−
*Model: HP Procurve 3400cl
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**88 Gbps
 
−
**64 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
**16 MB dual flash
 
−
**128 MB SDRAM
 
−
*Cut-through switching: No
 
−
*Unused as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | CN63FP762S
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;font-size:80%;" |
 
−
*Model: HP 2530-24G
 
−
*Ports: 24
 
−
*Switching Capacity:
 
−
**56 Gbps
 
−
**41.6 million pps
 
−
*Memory:
 
−
**1.5 MB packet buffer
 
−
**256 MB  flash
 
−
**128 MB DDR3 DIMM
 
−
*Cut-through switching: No
 
−
*Connected to Al-Salam as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | SG525SG025
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;font-size:80%;" |
 
−
*Model: HP Procurve 3400cl
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**88 Gbps
 
−
**64 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
**16 MB dual flash
 
−
**128 MB SDRAM
 
−
*Cut-through switching: No
 
−
*Connected to layout and whedon as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#39ad39; border-left:solid 5px #39ad39; border-top:solid 5px #39ad39; border-bottom:solid 1px white; border-right:solid 5px #39ad39; font-size:120%;" | Netgear JGS524
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#39ad39; border-left:solid 5px #39ad39; border-bottom:solid 5px #39ad39; border-right:solid 5px #39ad39;font-size:80%;" |
 
−
*Current cluster head-node
 
−
*Unmanaged (no console/configuration)
 
−
*Ports: 24
 
−
*Switching bandwidth:
 
−
**48 Gbps
 
−
**1.5 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
*Cut-through switching: No
 
−
*Connected to Al-Salam, Hopper, Pollock, Nagios, Dali, Kahlo, Bronte as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px #E77471; font-size:120%;" | HP 5920
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;font-size:80%;" |
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**480 Gbps
 
−
**367 million pps
 
−
*Memory:
 
−
**3.6 GB packet buffer
 
−
**256 MB dual flash
 
−
**2 GB SDRAM
 
−
*Cut-through switching: Yes
 
−
*Connected to layout, kahlo, and dali as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-top:solid 5px #ADDFFF; border-bottom:solid 1px white; border-right:solid 5px #ADDFFF; font-size:120%;" | HP 5500
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-bottom:solid 5px #ADDFFF; border-right:solid 5px #ADDFFF;font-size:80%;" |
 
−
*Model: JG542A
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**224 Gbps
 
−
**166.6 million pps
 
−
*Memory:
 
−
**6 MB packet buffer
 
−
**512 MB dual flash
 
−
**1 GB SDRAM
 
−
*Cut-through switching: No
 
−
*Connected to Babbage, Control, Nagios, and the netgear switch as of May 12, 2017
 
−
|}
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
 
 
−
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
 
−
<br><br><br><br><br><br>
 
−
 
 
−
= Systems Administration Documentation =
 
−
For old documentation, see: [[Sysadmin:Old | Old Wiki Information]]
 
−
 
 
−
{|
 
−
|- valign:"top"
 
−
|
 
−
<div style="border:10px solid #E0EAF8; padding:5px; width:230px; height:500px">
 
−
<div style="background-color:#CEDEF4; padding:5px;">
 
−
 
 
−
=== Admin Tasks ===
 
−
</div>
 
−
* [[Sysadmin:Nagios | Nagios Monitoring ]]
 
−
* [[Sysadmin:Shinken | Shinken Monitoring ]]
 
−
* [[Sysadmin:Upgrading SSL Certificate | Upgrading SSL Certificates ]]
 
−
* [[Sysadmin:User Management | User Management]]
 
−
* [[Newmodules | Installing software under modules ]]  
 
 
* [[Sysadmin:Backup|Backup]]
 
* [[Sysadmin:Backup|Backup]]
−
* [[Sysadmin:Contacting all users|Contacting all users]]
+
* [[Sysadmin:Monitoring | Monitoring ]]
−
* [[Sysadmin:New Sysadmins | Welcoming a new sysadmin to the fold]]
+
* [[Sysadmin:SSH|SSH info relevant to admins]]
−
* [[Sysadmin:AddComputer|Add a computer]]
+
* [[Sysadmin:User Management | User Management]] and [[Sysadmin:LDAP|LDAP]] generally
−
* [[Sysadmin:Setting up Lovelace Lab Machines | Setting up Lovelace Lab Machines]]
+
* [[Sysadmin:Jupyterhub Notebook Server|Jupyterhub]] and [[Nbgrader notes|NBGrader]]
−
* [[Reset password]]
+
* [[Sysadmin:MailStack|Email service]]
−
 
+
* [[Sysadmin:XenDocs | Xen Server]]
−
 
+
* [[Sysadmin:NFS|Network File System (NFS)]]
−
<!-- This has to stay as part of the formatting -->
+
* [[Sysadmin:Web Servers|Web Servers and Websites]]
−
</div>
 
−
| style="float:left;" |
 
−
|
 
−
<div style="border:10px solid #FFDFFF; padding:5px; width:230px; height:500px;">
 
−
<div style="background-color:#FFCEFF; padding:5px;">
 
−
 
 
−
=== Services ===
 
−
</div>
 
−
* [[Sysadmin:Services:Apache2|Apache2]]
 
 
* [[Sysadmin:Services:Databases|Databases]]
 
* [[Sysadmin:Services:Databases|Databases]]
 
* [[Sysadmin:DNS & DHCP|DNS and DHCP]]
 
* [[Sysadmin:DNS & DHCP|DNS and DHCP]]
−
* [[Sysadmin:Services:Virtualization | Virtualization]]
+
* [[Sysadmin:AWS|AWS]]
−
* [[Sysadmin:Services:XenServerSetup | Xen Server]]
+
* [[Bash_start_up_script|Bash startup scripts]]
 +
* [[Sysadmin:VirtualBox | VirtualBox]]
 +
* [[X Applications]]
 +
* [[Sysadmin:Services:ClusterOverview|Cluster Overview]] and [[Sysadmin:Ccg-admin|additional details]]
 +
* [[Sysadmin:Firewall|Firewall]] running on babbage.cs.e.e
 +
* [[Sysadmin:Setting_up_Lovelace_Lab_Machines|Setting up Lab Machines]]
  
−
<!-- This has to stay as part of the formatting -->
+
===Common tasks===
−
</div>
+
* [[Sysadmin:Recurring Tasks | Recurring tasks - e.g. software updates, hardware replacements]]
−
| style="float:left;" |
+
* [[Sysadmin:Contacting all users|Contacting all users]]
−
|
+
* [[Reset password]]
−
<div style="border:10px solid #F0DDD5; padding:5px; width:230px; height:500px;">
+
* [[Sysadmin:Software installation | Software installation]]
−
<div style="background-color:#E4C0B1; padding:5px;">
+
* [[Modules | Installing software under modules ]]  
−
 
+
* [[Sysadmin:AddComputer|Add a computer to CS or cluster domains]]
−
=== Miscellaneous ===
+
* [[Senior projects|Supporting senior projects]]
−
</div>
+
* [[ShutdownProcedure|How to do a planned shutdown and reboot of the system]]
−
* [[ShutdownProcedure| Shutdown and Boot up]]
+
** [[Sysadmin:TestingServices | Testing services]] (after a reboot, upgrade, change in the phase of the moon, etc.)
−
* [[SysadminContactInfo| Contact Information]]
+
* [[Sysadmin:Upgrading SSL Certificate | Upgrading SSL Certificates ]]
−
* [[Sysadmin:ImportantInfo:PhoneNumbers| Phone Numbers]]
+
* [[Sysadmin:Launch at startup|Launch a process at startup]]
−
* [[Sysadmin:ImportantInfo:AuthenticationInfo| Authentication Information]]
+
* [[Sysadmin:Psql-setup | setup psql for cs430 students]]
−
* [[Sysadmin:ImportantInfo:UPS| UPS]]
 
−
* [[Sysadmin:ImportantInfo:SSLcerts| Generating SSL Certificates]]
 
−
* [[Sysadmin:Power draws| Power draws]]
 
−
* [[Sysadmin:ImportantInfo:SunHardware|Working with Sun Hardware]]
 
−
* [[Sysadmin:Passwords]]
 
−
* Patching
 
−
** [[LinuxKernelPatching|Linux Kernel Patching]]
 
−
* [[Sysadmin:SerialConsoleCableEnds|Cable Ends]]
 
−
* [[Sysadmin:VirtualizationComparison|NEW Virtualization Comparison]]
 
−
 
 
−
<!-- This has to stay as part of the formatting -->
 
−
</div>
 
−
| style="float:left;" |
 
−
|
 
−
<div style="border:10px solid #D6F8DE; padding:5px; width:230px; height:500px;">
 
−
<div style="background-color:#BDF4CB; padding:5px;">
 
−
 
 
−
=== Networking ===
 
−
</div>
 
−
* [[Sysadmin:Networking:NetworkLayout|Network Layout (as of 08/2006)]]
 
−
* [[Sysadmin:Networking:D224 cable plant|D224 cable plant]]
 
−
* [[Sysadmin:Networking:Fiber plans|Fiber plans]]
 
−
* [[Sysadmin:Networking:Switches|Switches]]
 
−
* [[Sysadmin:Networking:Rack notes|Rack notes]]
 
−
* [[Sysadmin:Networking:Public|Public Network]]
 
−
* [[Sysadmin:Networking:NetworkTopo|Old Network Topo Figures]]
 
−
* [[Sysadmin:Networking:NetworkDiagram|Network layout (May 2007)]]
 
−
* [[Sysadmin:Networking:Alternate Network Path|Alt Network path]]
 
−
* [[Sysadmin:UPS Setup]]
 
−
 
 
−
<!-- This has to stay as part of the formatting -->
 
−
</div>
 
−
|}
 
−
 
 
−
== Current Projects (updated 2017-04-27) ==
 
−
=== TODO ===
 
−
* Layout infiniband subnet manager
 
−
* Layout disk swap, new lo0
 
−
* HP Al-Salam switch enable jumboframes
 
−
 
 
−
== Ongoing Projects (Spring 2017) ==
 
−
=== TODO ===
 
−
* EMAILING ALL THE USERS https://wiki.cs.earlham.edu/index.php/Sysadmin:Old:Contacting_All_Users
 
−
* SHUTDOWN SCHEDULED FOR SUNDAY (APRIL 16)
 
−
** Check/update instructions - one version is at https://wiki.cs.earlham.edu/index.php/Sysadmin:ImportantInfo:PowerFailure, there are others too
 
−
** Notify users
 
−
* Fix certs for gitlab, etc.
 
−
* Secure 1-2 admins for the summer
 
−
* Prep layout for May-June usage
 
−
* Practice shutdown-startup procedure (with Michael)
 
−
* Nsswitch consistency across all machines
 
−
* Document tools: startup / shutdown - Charlie
 
−
* Use Sysadmin namespace for all our pages - All
 
−
** Testing usefulness of documentation - Dave
 
−
* Al Salam: configure switch, re-rack. - Vitalii
 
−
** HP switch should be reset and tested.
 
−
* LDAP cleanup of system users / old groups - James
 
−
* Layout - Nirdesh
 
−
** Lo0 RAID (mdadm)
 
−
** 10GB from Dali to lo0 (adding rules on compute node routing tables as a possible fix)
 
−
** BIOS reset
 
−
* 10Gb, perfsonar, ...
 
−
* Monitoring: (Ganglia, Shinken)
 
−
** Getting consistency among all the machines(check_nrpe regularly stops working).
 
−
* Whedon: configured and available
 
−
* Change passwords (on everything). Postgres, shenken, ...
 
−
* Webcam on office whiteboard (new office location?)
 
−
* Learn virtual machine architecture and modules - Dave
 
−
** Document in a format for future admin training?
 
−
** Find existing introduction material
 
−
* Mirror ''control'' for testing, swapping, etc.
 
−
 
 
−
=== DONE (19 Jan 2017) ===
 
−
* Examine extra "layout" node. - Adam
 
−
** Differences are: Single PSU, Single GPGPU, No VGA.
 
−
** It has Infiniband and 10GB cards installed.
 
−
* Networking - Adam, Charlie
 
−
** IP over Infiniband working on layout
 
−
*** Resolved by resetting IB switch configuration: <code>ibwarn: [3349] mad_rpc_rmpp: _do_madrpc failed; dport (Lid 1)</code>
 
−
 
 
−
=== FUTURE ===
 
−
* Centralized password database / manager / location
 
  
−
== Current Projects (updated 13 Oct 16) ==  
+
===Group and institution information===
−
* '''Groups and LDAP and sudo - James'''
+
* [[Sysadmin:CS-ITS Interoperability|Working with ITS]]
−
* <s>Amber - James</s>
+
* [[Sysadmin:Recurring spending | Recurring spending ]]
−
* <s>Edward's setup - Vitalli</s>
+
* [[Sysadmin:SlackAndGitLab | Slack and GitLab integration]]
−
* <s>WebDev access - Nirdesh<s>
 
−
* Puppet - James and Vitalii
 
−
* '''Bacula - Nirdesh'''
 
−
* SSL certificate upgrade and documentation - Kristin
 
−
* <s>Listserv merging with archives preserved - Nirdesh </s>
 
−
* '''Ganglia - Bret'''
 
−
* '''Shenken - Vitalii'''
 
−
** latency, UPS
 
−
* New Layout node - ? and ?
 
−
* Provision Sappho (compute) - after Puppet
 
−
* Provision Kahlo (storage) -
 
−
** replace broken drive
 
−
* I2 setup
 
−
** DTN, storage nodes, head nodes, ports in CST
 
−
* [[Sysadmin:WhedonProvisioning|Provision Whedon]] (compute) - after Puppet
 
−
* '''Shutdown and startup test - scheduled for Sunday 27 November'''
 
−
* Disk cleaning - Charlie
 
−
* <s>Password changing in the CS and cluster domains - Vitalii and James</s>
 
−
* Proto setup and maintenance with HIP/Green Science
 

Latest revision as of 08:32, 20 March 2024

This is the hub for the CS sysadmins on the wiki.

Overview

If you're visually inclined, we have a colorful and easy-to-edit map of our servers here!

Server room

Our servers are in Noyes, the science building that predates the CST. For general information about the server room and how to use it, check out this page.

Columns: machine name, IPs, type (virtual, metal), purpose, dies, cores, RAM

Compute Resources

Machines and VMs related information here!

Network

We have two network fabrics linking the machines together. There are three subdomains.

10 Gb

We have 10Gb fabric to mount files over NFS. Machines with 10Gb support have an IP address in the class C range 10.10.10.0/24 and we want to add DNS to these addresses.

1 Gb (cluster, cs)

We have two class C subnets on the 1Gb fabric: 159.28.22.0/24 (CS) and 159.28.23.0/24 (cluster). This means we have double the IP addresses on the 1Gb fabric that we have on the 10Gb fabric.

Any user accessing *.cluster.earlham.edu and *.cs.earlham.edu is making calls on a 1Gb network.

Intra-cluster fabrics

The layout cluster has an Infiniband infrastructure. Wachowski has only a 1Gb infrastructure.

Power

We have a backup power supply, with batteries last upgraded in 2019 (?). We’ve had a few outages since then and power has held up well.

HVAC

HVAC systems are static and are largely managed by Facilities.

See full topology diagrams here.

A word about what's happening between files and the drives they live on.

New sysadmins

These pages will be helpful for you if you're just starting in the group:

Note: you'll need to log in with wiki credentials to see most Sysadmin pages.

Additional information

These pages contain a lot of the most important information about our systems and how we operate.

Handy Tools

Technical docs

Common tasks

Group and institution information

Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Sysadmin&oldid=18143"