Indiana Storage Server
Indiana is our new (circa summer 2018) storage server.
The server has been configured for RAID-5 using their built-in software.
The server is at indiana.cluster.earlham.edu.
To use it, ssh into indiana as the user admin and then run sudo su -.
Through the browser-based tool, we've added the cluster gateway to the server, enabled NFS and SFTP, disabled some unnecessary file-sharing tools.
We've also edited /etc/fstab on hopper to take advantage of NFS mounting.
Hacking in to Indiana
Using the ACP Commander tool, downloaded as a JAR and then run via X11 from Whedon, we were able to gain access to run commands in a limited fashion.
We tried a few read only commands, such as whoami and hostname, to figure out what exactly existed and worked. Turns out, most common commands exist on indiana, contrary to what we had determined in a previous attempt at hacking into the server.
We then attempted to run vim, however this did not work due a limitation in how ACP Commander works. Then we tried to run the two complex functions built into ACP Commander: Enable SSH and change root password. The first of these worked, however the second did not seem to work, although this may have been a side effect of the fact that Indiana's sshd_config did not allow root logins.
The next step was to see if we could modify any of the config files, specifically sshd_config, to allow what we were trying to do. We determined that we could append to it, but we could not overwrite it. We also determined that we could copy files from /etc to /mnt/array1/indiana, which is the directory that we set up for our servers to mount. This way, we could view documents in their entirety, since ACP Commander has a limitation on the number of lines that can be transmitted to it.
This proved to not be useful for the sshd_config file, as we could not override the options we wanted, however we eventually realized that we could append a line to /etc/sudoers giving our admin user access to sudo privileges. We ran
echo "admin\tALL=(ALL:ALL) ALL" >> /etc/sudoers, and then tried again to ssh in with admin and run
sudo su -.
This time, it worked! So we now have access to all of Indiana's capabilities! We also now have a voided warranty :'(
A subset of the config files appear to get zorched on reboot. We have to fix this before Indiana goes into production.
Stuff you may need to rebuild if this goes wrong again:
- Get the acp_commander file from craig or laurence's directory
- Run using X forwarding -
ssh -Y cluster.earlham.edu- by running
java -jar acp_commander
- Connect to 220.127.116.11 (not the 10.10), port 22936 , admin password is the usual
whoami# command line, confirms you're root
- click "Enable SSH"
chown admin:admin ~admin/.ssh# command line, makes sure admin owns its own ssh directory
chown admin:admin ~admin/.bash*# command line, makes sure admin owns its own bash files
/etc/init.d/sshd.sh start# command line, starts the SSH service
- If you'd like you can append export
PATH=/sbin:/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbinto ~admin/bash_profile now. You'll have to do it later.
- Run other commands to debug. Note that you can use /mnt/array1/indiana as a place to run debugging
- A word on PAM
- A problem that arose was that su was not able to do anything and several commands could not be run as root or through sudo, such as the passwd command
- To fix this: look in /var/log/messages and grep for pam or su errors and see if it is having trouble opening a module; the one we ran into was pam_ldap.so which was in a directory that did not exist, /lib/security.
- If that is the problem, a quick fix is to go through all the pam configs in /etc/pam.d, look for all references to the specified module, and comment those lines out
- If you need that module, provide one in the path that PAM is looking at.
Other annoying things we fixed
Since we can now run as the root user on the shell, we've begun pruning the weeds on the server. Among other things:
- We edited something with usermod to handle rbash # specifics?
- We updated $PATH in /etc/bash.bashrc.
- We made the admin user own its own .ssh directory.
- We created a new sysadmin user with adduser, and its home directory is /home/sysadmin as usual.