Difference between revisions of "Sysadmin"

From Earlham CS Department
Jump to navigation Jump to search
(→‎Current Projects)
 
(96 intermediate revisions by 8 users not shown)
Line 1: Line 1:
−
__NOTOC__
+
This is the hub for the CS sysadmins on the wiki.
  
−
= Machines and Brief Descriptions of Services =
+
= Overview =
−
== CS Machines ==
 
−
[[File:Server_layout_summer2017.jpg|thumb|200px|right|Server layout as of May 2017]]
 
  
−
{| style="float:left; margin-right:2px;"
+
[https://docs.google.com/drawings/d/1XaULz5IxXV_BZQjrko3QJ8wV5aXsSTYcSWxxT49OyZk/edit If you're visually inclined, we have a colorful and easy-to-edit map of our servers here!]
−
| style="height:40px; width:150px; text-align:center; background-color:#54C571; border-left:solid 5px #54C571; border-top:solid 5px #54C571; border-bottom:solid 1px white; border-right:solid 5px #54C571; font-size:120%;" | NET <br> (vm1)
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#54C571; border-left:solid 5px #54C571; border-bottom:solid 5px #54C571; border-right:solid 5px #54C571;" | [[LDAP Server]] <br> [[Sysadmin:DNS & DHCP | DNS]] <br> [[Sysadmin:DNS & DHCP | DHCP]] <br><br> Backup to Dali: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== Server room ==
−
| style="height:40px; width:150px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px #E77471; font-size:120%;" | WEB <br> (vm2)
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;" | [[Sysadmin:Email:Mailman | Mailman]] <br> [[Sysadmin:Mail Stack | Mail Stack]]<br> Apache2 <br> PostgresQL <br> MySQL <br> Wiki <br><br> Backup to Dali: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
Our servers are in Noyes, the science building that predates the CST. For general information about the server room and how to use it, check out [[Sysadmin:Server Room|this page]].
−
| style="height:40px; width:150px; text-align:center; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-top:solid 5px #C38EC7; border-bottom:solid 1px white; border-right:solid 5px #C38EC7; font-size:120%;" | TOOLS <br> (vm3)
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-bottom:solid 5px #C38EC7; border-right:solid 5px #C38EC7;" | [[SageNB Server | SageNB Server]] <br> [[Jupyterhub notebook server | Jupyterhub Server]] <br> [[Sysadmin:Software Modules | Software Modules]] <br> NginX  <br>SSH<br>Users<br><br> Backup to Dali: etc, var, mnts, sage
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
Columns: machine name, IPs, type (virtual, metal), purpose, dies, cores, RAM
−
| style="height:55px; width:150px; text-align:center; background-color:#E3A869; border-left:solid 5px #E3A869; border-top:solid 5px #E3A869; border-bottom:solid 1px white; border-right:solid 5px #E3A869; font-size:120%;" | BABBAGE
 
−
|-
 
−
| style="height:210px; width:150px; background-color: #E3A869; border-left:solid 5px #E3A869; border-bottom:solid 5px #E3A869; border-right:solid 5px #E3A869;" | [[Sysadmin:Firewall | Firewall]]
 
−
|}
 
  
−
{|
+
== Compute Resources ==
−
| style="height:55px; width:150px; text-align:center; background-color:#EEDC82; border-left:solid 5px #EEDC82; border-top:solid 5px #EEDC82; border-bottom:solid 1px white; border-right:solid 5px #EEDC82; font-size:120%;" | [[Sysadmin:Servers:Proto | PROTO]]
 
−
|-
 
−
| style="height:210px; width:150px; background-color: #EEDC82; border-left:solid 5px #EEDC82; border-bottom:solid 5px #EEDC82; border-right:solid 5px #EEDC82;" | Weather Monitoring <br> GPS/NTP <br> Energy Monitoring <br><br> Backup to Dali: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
[https://wiki.cs.earlham.edu/index.php/Sysadmin:Computer_Resources Machines and VMs related information here!]
−
| style="height:40px; width:150px; text-align:center; background-color:#FF7E6D; border-left:solid 5px #FF7E6D; border-top:solid 5px #FF7E6D; border-bottom:solid 1px white; border-right:solid 5px      #FF7E6D; font-size:120%;" | CONTROL
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#FF7E6D; border-left:solid 5px #FF7E6D; border-bottom:solid 5px #FF7E6D; border-right:solid 5px #FF7E6D;" | Users <br> SSH <br> HOME <br> TOOLS <br><br> Backup to Dali: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== Network ==
−
| style="height:40px; width:150px; text-align:center; background-color:#54C571; border-left:solid 5px #54C571; border-top:solid 5px #54C571; border-bottom:solid 1px white; border-right:solid 5px      #54C571; font-size:120%;" | SMILEY
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#54C571; border-left:solid 5px #54C571; border-bottom:solid 5px #54C571; border-right:solid 5px #54C571;" | [[XenDocs]] <br> NET <br> WEB <br>[[NFS]]<br><br> Backup to Dali: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
We have two network fabrics linking the machines together. There are three subdomains.
−
| style="height:40px; width:150px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px      #E77471; font-size:120%;" | SHINKEN
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;" | Users <br> SSH <br> Add machines
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
=== 10 Gb ===
−
| style="height:40px; width:150px; text-align:center; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-top:solid 5px #C38EC7; border-bottom:solid 1px white; border-right:solid 5px      #C38EC7; font-size:120%;" |MURPHY
 
−
|-
 
−
| style="height:210px; width:150px; background-color:#C38EC7; border-left:solid 5px #C38EC7; border-bottom:solid 5px #C38EC7; border-right:solid 5px #C38EC7;" | Elderly email stack <br> Users <br> SSH
 
−
|}
 
  
 +
We have 10Gb fabric to mount files over NFS. Machines with 10Gb support have an IP address in the class C range 10.10.10.0/24 and we want to add DNS to these addresses.
  
−
<br> <br> <br> <br> <br> <br><br> <br> <br> <br> <br> <br>
+
=== 1 Gb (cluster, cs) ===
  
−
== Cluster Machines ==
+
We have two class C subnets on the 1Gb fabric: 159.28.22.0/24 (CS) and 159.28.23.0/24 (cluster). This means we have double the IP addresses on the 1Gb fabric that we have on the 10Gb fabric.
  
−
{| style="float:left; margin-right:2px;"
+
Any user accessing *.cluster.earlham.edu and *.cs.earlham.edu is making calls on a 1Gb network.
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px      #0099cc; font-size:120%;" | HOPPER
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" | Users <br> SSH <br> NFS server <br> LDAP server <br> [[Sysadmin:Software Modules | Software Modules]] <br> PostgreSQL <br> Wiki <br> Apache2 <br> [[Sysadmin:DNS & DHCP | DNS]] <br> [[Sysadmin:DNS & DHCP | DHCP]]  <br><br> Backup to Indiana: etc, var, cluster
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
=== Intra-cluster fabrics ===
−
| style="height:55px; width:150px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | INDIANA
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;" | [[Indiana Storage Server|New Storage Server]]
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
The layout cluster has an Infiniband infrastructure. Wachowski has only a 1Gb infrastructure.
−
| style="height:55px; width:150px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | DALI
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;" | Storage Server <br>[[Sysadmin:Gitlab | Gitlab]] <br> Backups <br> NginX <br><br> Backup to Indiana (/media/r10_vol/backups/): etc, var/opt/gitlab/backups
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== Power ==
−
| style="height:55px; width:150px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | AL-SALAM
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;" | [[WebMO]] <br> [[Sysadmin:Software Modules | Software Modules]] <br> Apache2 <br><br> Backup to Indiana: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
We have a backup power supply, with batteries last upgraded in 2019 (?). We’ve had a few outages since then and power has held up well.
−
| style="height:55px; width:150px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | WHEDON
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;" | [[Sysadmin:Software Modules | Software Modules]] <br><br> Backups to Indiana: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
== HVAC ==
−
| style="height:55px; width:150px; text-align:center; background-color:#39ad39; border-left:solid 5px #39ad39; border-top:solid 5px #39ad39; border-bottom:solid 1px white; border-right:solid 5px #39ad39; font-size:120%;" | LAYOUT
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#39ad39; border-left:solid 5px #39ad39; border-bottom:solid 5px #39ad39; border-right:solid 5px #39ad39;" | [[Sysadmin:Jupyterhub Notebook Server | Jupyterhub Server]] <br> [[Sysadmin:Software Modules | Software Modules]] <br> NginX <br> Apache2 <br> [[WebMO]] <br><br> Backup to Indiana: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
HVAC systems are static and are largely managed by Facilities.
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | BRONTE
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]] <br><br> Backup to Indiana: etc, var, nbserver
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
[[Topology|See full topology diagrams here.]]
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | POLLOCK
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]] <br> [[WebMO]] <br> NginX <br><br> Backup to Indiana: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
[[Sysadmin:Layers of abstraction for filesystems|A word about what's happening between files and the drives they live on.]]
−
| style="height:55px; width:150px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | KAHLO
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;" | Storage Server <br>Backups <br> NginX <br><br> Backup to Indiana: etc, var
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
= New sysadmins =
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | BIGFE
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]] <br><br> Hosts BCCD related repositories and distributions.
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
These pages will be helpful for you if you're just starting in the group:
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | T-VOC
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]]
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:150px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px #0099cc; font-size:120%;" | ELWOOD
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc;" |  [[Sysadmin:Software Modules | Software Modules]] <br> <br> Used by BCCD to host www.bccd.net and www.littlefe.net. Will be deprecated when BCCD project offloads their sites onto cloud-based hosting platforms.
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:150px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | krasner
 
−
|-
 
−
| style="height:300px; width:150px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;" | [[Docker]] platform on an old lovelace machine upgraded to have 16GB of RAM.
 
−
|}
 
  
 +
* [[Sysadmin:New Sysadmins | Welcoming a new sysadmin ]]
 +
* [[Sysadmin:Troubleshooting|General troubleshooting tips for admins]]
 +
* [[Sandbox Notes|Sandbox Notes]]
 +
* [[Password managers]]
 +
* [[Server safety]]
 +
* [https://code.cs.earlham.edu/sysadmin/ticket-tracker Ticket tracking for current projects]
  
−
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
+
Note: you'll need to log in with wiki credentials to see most Sysadmin pages.
  
−
== Switches ==
+
= Additional information =
  
 +
These pages contain a lot of the most important information about our systems and how we operate.
  
 +
===Handy Tools===
 +
* [http://monitor.cluster.earlham.edu:8088/packages Porter's Package Explorer]
  
−
{| style="float:left; margin-right:2px;"
+
===Technical docs===
−
| style="height:55px; width:175px; text-align:center; background-color:#0099cc; border-left:solid 5px #0099cc; border-top:solid 5px #0099cc; border-bottom:solid 1px white; border-right:solid 5px      #0099cc; font-size:120%;" | SG538SF02J
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#0099cc; border-left:solid 5px #0099cc; border-bottom:solid 5px #0099cc; border-right:solid 5px #0099cc; font-size:80%;" |
 
−
*Model: HP Procurve 3400cl
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**88 Gbps
 
−
**64 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
**16 MB dual flash
 
−
**128 MB SDRAM
 
−
*Cut-through switching: No
 
−
*Unused as of May 12, 2017
 
−
|}
 
  
−
{| style="float:left; margin-right:2px;"
+
* [https://code.cs.earlham.edu/sysadmin/ticket-tracker Ticket tracking for current projects]
−
| style="height:55px; width:175px; text-align:center; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-top:solid 5px #ffdb4d; border-bottom:solid 1px white; border-right:solid 5px #ffdb4d; font-size:120%;" | CN63FP762S
+
* [[Server safety]]
−
|-
 
−
| style="height:200px; width:175px; background-color:#ffdb4d; border-left:solid 5px #ffdb4d; border-bottom:solid 5px #ffdb4d; border-right:solid 5px #ffdb4d;font-size:80%;" |
 
−
*Model: HP 2530-24G
 
−
*Ports: 24
 
−
*Switching Capacity:
 
−
**56 Gbps
 
−
**41.6 million pps
 
−
*Memory:
 
−
**1.5 MB packet buffer
 
−
**256 MB  flash
 
−
**128 MB DDR3 DIMM
 
−
*Cut-through switching: No
 
−
*Connected to Al-Salam as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-top:solid 5px #ff4d94; border-bottom:solid 1px white; border-right:solid 5px #ff4d94; font-size:120%;" | SG525SG025
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#ff4d94; border-left:solid 5px #ff4d94; border-bottom:solid 5px #ff4d94; border-right:solid 5px #ff4d94;font-size:80%;" |
 
−
*Model: HP Procurve 3400cl
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**88 Gbps
 
−
**64 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
**16 MB dual flash
 
−
**128 MB SDRAM
 
−
*Cut-through switching: No
 
−
*Connected to layout and whedon as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#39ad39; border-left:solid 5px #39ad39; border-top:solid 5px #39ad39; border-bottom:solid 1px white; border-right:solid 5px #39ad39; font-size:120%;" | Netgear JGS524
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#39ad39; border-left:solid 5px #39ad39; border-bottom:solid 5px #39ad39; border-right:solid 5px #39ad39;font-size:80%;" |
 
−
*Current cluster head-node
 
−
*Unmanaged (no console/configuration)
 
−
*Ports: 24
 
−
*Switching bandwidth:
 
−
**48 Gbps
 
−
**1.5 million pps
 
−
*Memory:
 
−
**2MB packet buffer
 
−
*Cut-through switching: No
 
−
*Connected to Al-Salam, Hopper, Pollock, Nagios, Dali, Kahlo, Bronte as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#E77471; border-left:solid 5px #E77471; border-top:solid 5px #E77471; border-bottom:solid 1px white; border-right:solid 5px #E77471; font-size:120%;" | cs-main
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#E77471; border-left:solid 5px #E77471; border-bottom:solid 5px #E77471; border-right:solid 5px #E77471;font-size:80%;" |
 
−
*Model: HP 5920AF-24XG
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**480 Gbps
 
−
**367 million pps
 
−
*Memory:
 
−
**3.6 GB packet buffer
 
−
**256 MB dual flash
 
−
**2 GB SDRAM
 
−
*Cut-through switching: Yes
 
−
*IP Address: 159.28.31.66
 
−
*Connected to layout, kahlo, and dali as of May 12, 2017
 
−
|}
 
−
 
 
−
{| style="float:left; margin-right:2px;"
 
−
| style="height:55px; width:175px; text-align:center; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-top:solid 5px #ADDFFF; border-bottom:solid 1px white; border-right:solid 5px #ADDFFF; font-size:120%;" | 5500denniscs-sw1
 
−
|-
 
−
| style="height:200px; width:175px; background-color:#ADDFFF; border-left:solid 5px #ADDFFF; border-bottom:solid 5px #ADDFFF; border-right:solid 5px #ADDFFF;font-size:80%;" |
 
−
*Model: HP 5500 JG542A
 
−
*Ports: 24
 
−
*Backplane bandwidth:
 
−
**224 Gbps
 
−
**166.6 million pps
 
−
*Memory:
 
−
**6 MB packet buffer
 
−
**512 MB dual flash
 
−
**1 GB SDRAM
 
−
*Cut-through switching: No
 
−
*IP Address: 159.28.31.67
 
−
*Connected to Babbage, Control, Nagios, and the cluster's netgear switch (via port 14) as of May 12, 2017
 
−
|}
 
−
 
 
−
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
 
−
 
 
−
= Systems Administration Documentation =
 
−
For old documentation, see: [[Sysadmin:Old | Old Wiki Information]]
 
−
 
 
−
{|
 
−
|- valign:"top"
 
−
|
 
−
<div style="border:10px solid #E0EAF8; padding:5px; width:230px; height:500px">
 
−
<div style="background-color:#CEDEF4; padding:5px;">
 
−
 
 
−
=== Admin Tasks ===
 
−
</div>
 
−
* [[Sysadmin:Shinken | Shinken Monitoring ]]
 
−
* [[Sysadmin:Upgrading SSL Certificate | Upgrading SSL Certificates ]]
 
−
* [[Sysadmin:User Management | User Management]]
 
−
* [[Modules | Installing software under modules ]]  
 
 
* [[Sysadmin:Backup|Backup]]
 
* [[Sysadmin:Backup|Backup]]
−
* [[Sysadmin:Contacting all users|Contacting all users]]
+
* [[Sysadmin:Monitoring | Monitoring ]]
−
* [[Sysadmin:New Sysadmins | Welcoming a new sysadmin to the fold]]
+
* [[Sysadmin:SSH|SSH info relevant to admins]]
−
* [[Sysadmin:AddComputer|Add a computer]]
+
* [[Sysadmin:User Management | User Management]] and [[Sysadmin:LDAP|LDAP]] generally
−
* [[Sysadmin:Setting up Lovelace Lab Machines | Setting up Lovelace Lab Machines]]
+
* [[Sysadmin:Jupyterhub Notebook Server|Jupyterhub]] and [[Nbgrader notes|NBGrader]]
−
* [[Reset password]]
+
* [[Sysadmin:MailStack|Email service]]
−
* [[Senior projects]]
+
* [[Sysadmin:XenDocs | Xen Server]]
−
* [[ShutdownProcedure| Shutdown and Boot up]]
+
* [[Sysadmin:NFS|Network File System (NFS)]]
−
* [[Sysadmin:ImportantInfo:SSLcerts| Generating SSL Certificates]]
+
* [[Sysadmin:Web Servers|Web Servers and Websites]]
−
* [[Password managers]]
 
−
 
 
−
 
 
−
<!-- This has to stay as part of the formatting -->
 
−
</div>
 
−
| style="float:left;" |
 
−
|
 
−
<div style="border:10px solid #FFDFFF; padding:5px; width:230px; height:500px;">
 
−
<div style="background-color:#FFCEFF; padding:5px;">
 
−
 
 
−
=== Services ===
 
−
</div>
 
−
* [[Sysadmin:Services:ClusterOverview|Cluster Overview]]
 
−
* [[Sysadmin:Services:Apache2|Apache2]]
 
 
* [[Sysadmin:Services:Databases|Databases]]
 
* [[Sysadmin:Services:Databases|Databases]]
 
* [[Sysadmin:DNS & DHCP|DNS and DHCP]]
 
* [[Sysadmin:DNS & DHCP|DNS and DHCP]]
−
* [[Sysadmin:Services:Virtualization | Virtualization]]
+
* [[Sysadmin:AWS|AWS]]
−
* [[Sysadmin:Services:XenServerSetup | Xen Server]]
+
* [[Bash_start_up_script|Bash startup scripts]]
 +
* [[Sysadmin:VirtualBox | VirtualBox]]
 +
* [[X Applications]]
 +
* [[Sysadmin:Services:ClusterOverview|Cluster Overview]] and [[Sysadmin:Ccg-admin|additional details]]
 +
* [[Sysadmin:Firewall|Firewall]] running on babbage.cs.e.e
 +
* [[Sysadmin:Setting_up_Lovelace_Lab_Machines|Setting up Lab Machines]]
  
−
|}
+
===Common tasks===
−
 
+
* [[Sysadmin:Recurring Tasks | Recurring tasks - e.g. software updates, hardware replacements]]
−
== Current Projects ==
+
* [[Sysadmin:Contacting all users|Contacting all users]]
−
This is the list we will work from in addition to service requests.
+
* [[Reset password]]
−
 
+
* [[Sysadmin:Software installation | Software installation]]
−
Some important procedural pages:
+
* [[Modules | Installing software under modules ]]  
−
* Use the [[Sysadmin/New_Task_Template|Sysadmin task template]] if you're starting a new project. Copy and paste the wiki source page and populate the basic fields.
+
* [[Sysadmin:AddComputer|Add a computer to CS or cluster domains]]
−
* This is a new way of [[Sysadmin/Task_Process|task processing]]. It's subject to change.
+
* [[Senior projects|Supporting senior projects]]
−
* You can see our [[:Category:Open Tasks|Open Tasks here]].
+
* [[ShutdownProcedure|How to do a planned shutdown and reboot of the system]]
−
* We will also start filling up the [[:Category:Closed Tasks|Closed Tasks category]].
+
** [[Sysadmin:TestingServices | Testing services]] (after a reboot, upgrade, change in the phase of the moon, etc.)
−
 
+
* [[Sysadmin:Upgrading SSL Certificate | Upgrading SSL Certificates ]]
−
Please update specific projects at their own page.
+
* [[Sysadmin:Launch at startup|Launch a process at startup]]
−
* [[Power down before outages]]
+
* [[Sysadmin:Psql-setup | setup psql for cs430 students]]
−
* [[Web logins]]
 
−
* [[Password management]]
 
−
* [[Docker and WebODM on Bronte]]
 
−
* [[Fix shinken server access]]
 
−
* [[Verify Lovelace DNS]] - check what machines we have vs. what's in the DNS file
 
−
* [[Layout Layout]]
 
−
* Backup in Lilly basement
 
−
* FIFO for requests rather than ad-hoc
 
−
* Accounting for hours logged
 
−
* key-based access to whedon, pollock, bronte (not just passwords)
 
−
* Power map additions and updates
 
−
* [[Backup on all machines]] - includes backup.cs.e.e (indiana?)
 
−
* [[Fix Lovelace machines]]
 
  
−
Post-shutdown, here are things that need fixed, updated, etc.:
+
===Group and institution information===
−
* Al-salam: PDU was powered off when we got to the basement - nodes 1-4 and 9-12 are connected to the PDU so they were powered down. Unsure when 9-12 were connected to the PDU; 1-4 were the only al-salam nodes connected this summer.
+
* [[Sysadmin:CS-ITS Interoperability|Working with ITS]]
−
* Sudo for whedon only pwd required
+
* [[Sysadmin:Recurring spending | Recurring spending ]]
−
* Hard to force shutdown on hopper
+
* [[Sysadmin:SlackAndGitLab | Slack and GitLab integration]]
−
* Babbage slow to shut down, had to reboot (even in the shell, shutdown -h now only rebooted it)
 
−
* Mounting FS in smiley, had to run: mount --source=/dev/vmdata/eccs-home-disk/ --target=/smiley-eccs-home-disk
 
−
* Pollock needed manual ifup
 
−
* ganglia monitoring comes back up on some nodes (definitely on head nodes) but needs to be started on compute nodes
 
−
* Are sysadmin accounts backing up to anywhere?
 
−
* How much power are we drawing at max from everything? (PDU, burnout, etc.)
 

Latest revision as of 09:32, 20 March 2024

This is the hub for the CS sysadmins on the wiki.

Overview

If you're visually inclined, we have a colorful and easy-to-edit map of our servers here!

Server room

Our servers are in Noyes, the science building that predates the CST. For general information about the server room and how to use it, check out this page.

Columns: machine name, IPs, type (virtual, metal), purpose, dies, cores, RAM

Compute Resources

Machines and VMs related information here!

Network

We have two network fabrics linking the machines together. There are three subdomains.

10 Gb

We have 10Gb fabric to mount files over NFS. Machines with 10Gb support have an IP address in the class C range 10.10.10.0/24 and we want to add DNS to these addresses.

1 Gb (cluster, cs)

We have two class C subnets on the 1Gb fabric: 159.28.22.0/24 (CS) and 159.28.23.0/24 (cluster). This means we have double the IP addresses on the 1Gb fabric that we have on the 10Gb fabric.

Any user accessing *.cluster.earlham.edu and *.cs.earlham.edu is making calls on a 1Gb network.

Intra-cluster fabrics

The layout cluster has an Infiniband infrastructure. Wachowski has only a 1Gb infrastructure.

Power

We have a backup power supply, with batteries last upgraded in 2019 (?). We’ve had a few outages since then and power has held up well.

HVAC

HVAC systems are static and are largely managed by Facilities.

See full topology diagrams here.

A word about what's happening between files and the drives they live on.

New sysadmins

These pages will be helpful for you if you're just starting in the group:

Note: you'll need to log in with wiki credentials to see most Sysadmin pages.

Additional information

These pages contain a lot of the most important information about our systems and how we operate.

Handy Tools

Technical docs

Common tasks

Group and institution information

Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Sysadmin&oldid=18143"