Sysadmin:Upgrading SSL Certificate

An SSL Certificate is used so that user's connections to our HTTPS services on a browser are secure. We use letsencrypt and certbot to provision our certs for free.

Adding cerbot/letsencrypt to a machine

This part is pretty straightforward. We need to install cerbot: apt-get install certbot(Debian) or yum install certbot(RHEL).

Adding a new cert to a machine

If you have not set up cerbot on this machine, refer to section 1.

1. certbot certonly --manual --preferred-challenges=dns --email [EMAIL] --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d [DOMAIN NAME].
2. Copy the txt value and create an entry in DNS on the appropriate server, probably hopper (for cluster) or net (for CS). It should look like this: _acme-challenge.[DOMAIN NAME]. IN TXT [TXT VALUE]. You'll want to do this in a different terminal and leave the certbot installer running.
3. - Restart DNS server (systemctl restart bind9) and wait a minute to make sure DNS has propagated. You can use nslookup to test this if you keep running into problems.
4. Return to the cerbot terminal and hit enter. It should take a minute to complete its challenge, and then notify you of success or failure.
5. Browse to /etc/letsencrypt/live/ and check that your certs are all in place in the [DOMAIN NAME] folder.
6. Fill in the cert wherever needed.

Apache2

Use the following lines to add these SSL certs to an apache2 site. If you aren't familiar with apache, these lines belong in the config file for a website in /etc/apache2/sites-available/ (or /etc/httpd/conf.d/ on CentOS/RHEL).

SSLCertificateFile      /etc/letsencrypt/live/[DOMAIN NAME]/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/[DOMAIN NAME]/fullchain.pem      
SSLCertificateKeyFile   /etc/letsencrypt/live/[DOMAIN NAME]/privkey.pem

If you are adding SSL to the default site on apache2, you should make sure these lines go in default-ssl.conf rather than 000-default. If you can't find the default-ssl.conf file, you might need to enable ssl in apache with a2enmod ssl (then systemctl restart apache2). You can also add these lines to 000-default to redirect all HTTP traffic to the new HTTPS version:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI}

You might need to add another module for this to work: a2enmod rewrite and then systemctl restart apache2.

Jupyterhub

Use the following lines to add SSL certs to a jupyterhub instance. If you aren't familiar with jupyter, these lines belong in the config file in /etc/jupyterhub.

c.JupyterHub.ssl_key = '/etc/letsencrypt/live/[DOMAIN NAME]/privkey.pem'
c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/[DOMAIN NAME]/fullchain.pem'

Renewing SSL Certificates

Since we use letsencrypt and certbot to provision our certs for free, we need to renew them around every three months.

You can check the state of each cert with Nagios on Monitor in Service Groups -> SSL Status: nagios

The following are the steps for renewing a cert; you will need two terminals to complete the process:

1. SSH to the server you want to renew the SSL cert for and become root, as you will need the permissions to run the script.
2. Navigate to /etc/letsencrypt and look for the bash script that relates to the cert you need to renew (get-certs-bowie.sh for example), and execute it with the command bash get-certs-<server>.sh. DO NOT press enter before the next steps, otherwise the renewal will fail and you will need to start over.
3. The script will start the renewal process. When it pauses, it will give you an address and a string of text, something like _acme-challenge.cs.earlham.edu and kkC6e62GJ3ah_ezWQXN0rwqxcOTKUR9cMPshKG0A-aw.
4. In a NEW TERMINAL, SSH to the DNS server for the machine you are updating (Net if on CS, Hopper if on Cluster). Become root and navigate to /etc/bind and open either cluster.zone or cs.zone, depending again on the machine you are updating.
5. At the end of the file, add a new DNS record with the information from the script on your other terminal. The record should look something like this: _acme-challenge.cs.earlham.edu. IN TXT "kkC6e62GJ3ah_ezWQXN0rwqxcOTKUR9cMPshKG0A-aw". Don't forget the "." at the end of the address. There are probably some commented-out examples of this in the bind file, comment signifiers being a semicolon (;), that you can use as a reference.
6. Save and exit the file, and use systemctl restart bind9 to reload the DNS settings.
7. Jump back over to your first terminal (the one whose cert is being renewed) and hit enter to continue the certification process. It should finish on its own.
8. Check on Nagios to make sure the cert now has around 90 days remaining. You might have to wait up to 10 minutes for Nagios to update.

Archive (Pre 2022)

An SSL Certificate is used so that user's connections to our HTTPS services on a browser are secure. We buy our certificates from Namecheap.com usually, and it has to be renewed after so long. Below are instructions on how to upgrade the SSL certificate. This is assuming you've already purchased the cert and that you purchased it from Namecheap.com. Upgrading the SSL certificates

• Go to Namecheap and login with the credentials that were used to buy the cert.
• Under domains, find the domain you just purchased, click 'Activate.'
• It will ask you to enter the CSR (Certificate Signing Request). The CSR is just an encrypted text file with info about the organization and server using the certificate. For instructions on generating a CSR, see below.
• Open the .csr file you generated and copy it.
• Paste it into the CSR text box on Namecheap.
• It should load and then automatically fill in the domain name.
• Select server type (NginX, Apache, etc.).
• Hit next.
• Verify everything is correct on the next page.
• Select the DVC (domain validation control) method you want to use.
• Enter information for Earlham CS dept on next page and an email address where the cert will be sent.
• Hit next.
• The next page is a validation. If you chose email as the DVC method, make sure you look for an email to that address. You won't get the cert until you've followed the DVC confirmation method.
• Hit confirm.
• When you get the email with the DVC, follow the instructions. Shortly after that's done, you'll receive an email with the certs attached.

Installing the certs on the server

• Download the zip file from the email with the certs attached.
• Move them to the server you want to put them on.
• Unzip the zip file.
• There should be a bundle file and a crt file. These files are typically placed in /etc/ssl/certs, move them there.
• Move the key you generated when generating the CSR to /etc/ssl/private.
• Make sure whatever web server config files have the correct path and names of the bundle file, cert and key file.
• If none of the file names changed since the last upgrade, then you shouldn't have a problem.
• Restart the web server (apache/nginx, not the actual computer) just for good measure.
• Open a browser and go to a webpage on that machine and make sure the SSL works (https).
• You need to perform these steps for every machine in that domain.
• Note about mail: if we are running a mail stack when the certs are replaced next, both the mail cert and the cs cert should be replaced at the same time, by the same new cert.

Generating a CSR The certs we buy a wildcard SSL certs, so we can have multiple *.cs.earlham.edu or *.cluster.earlham.edu records without having to buy multiple certs. Namecheap has a good tutorial for generating a CSR and that's what this is based on.

• Log into hopper or home, depending on the domain you are generating a CSR for (cluster or cs).
• Become root.
• Run the following command. It doesn't matter the name of the key and csr, but it's good practice to make it the name of the domain
  • hopper# openssl req -new -newkey rsa:2048 -nodes -keyout *.cluster.earlham.edu.key -out *.cluster.earlham.edu.csr
• So text should pass saying it's generating a key and the name of the file (*.cluster.earlham.edu.key) where it's being written. Note: This is your private key.
• Now, it's gonna ask you for information about Earlham and the domain you are registering the cert for. It'll look something like this:
  • Country Name (2 letter code) [AU]:US
  • State or Province Name (full name) [Some-State]:Indiana
  • Locality Name (eg, city) []:Richmond
  • Organization Name (eg, company) [Internet Widgits Pty Ltd]:Earlham College
  • Organizational Unit Name (eg, section) []:Department of Computer Science
  • Common Name (e.g. server FQDN or YOUR name) []:*.cluster.earlham.edu
    • It's important that you put the * in front of the domain, because it's a wildcard certificate.
  • Email Address []:charliep@cs.earlham.edu

Upgrading to certificates generated with the existing CSR (this is the least good way to do this)

• Submit the existing CSR, go through the security mojo, get the new certs
• Unpack the zip file
• cp the-certificate-name.ca-bundle the-certificate-name.crt /etc/ssl/certs
• $ apachectl configtest && apachectl restart