Sysadmin:Backup
We use a simple series of rsync scripts (no deletion) to maintain the most recent version of each file saved on any of our servers. Key directories are listed here.
Backup scripts can be found here.
| Contents | Directory | Script | Backup host and location |
|---|---|---|---|
| ECCS Homes | 10.10.10.15:/earlhamcs (bowie) | /root/backup-scripts/earlhamcs-backup.sh | Rossum:/storage/earlhamcs |
| Bowie PSQL | 10.10.10.15:/postgres (bowie) | /root/backup-scripts/postgres-backup.sh | Rossum:/storage/postgres |
| Bowie Docker | 10.10.10.15:/docker (bowie) | /root/backup-scripts/docker-backup.sh | Rossum:/storage/docker |
| Cluster Directory | 10.10.10.1:/cluster (hopper) | /root/backup-scripts/cluster-backup.sh | Rossum:/storage/cluster |
| Server Files | $hostname:/etc,var,root,home | /root/backup-scripts/all-server-backup.sh | Rossum:/storage/servers/$hostname/etc,var,root,home |
| Lovelace Cold-Storage Directory | 10.10.10.35:/mounts/lovelace/cold-storage (lovelace) | /root/backup-scripts/lovelace-bio-backup.sh | Rossum:/storage/servers/lovelace/mounts/lovelace/cold-storage |
| Bowie Nbgrader Directory | 10.10.10.15:/home/nbgrader (bowie) | /root/backup-scripts/bowie_nbgrader_backup.sh | Rossum:/storage/bowie_nbgrader |
| Everything on Rossum | 10.10.10.24:/storage (rossum) | /root/backup-scripts/rossum-backup.sh | Dijkstra:/storage/ |
Restore by copying the file back to the place it needs to be saved.
Each of the backup servers has one or more root crontab entries that call an rsync script in the servers /root/backup-scripts directory. This copies (pulls) the listed directories to their local backup. Each of the backup servers has a /storage and/or /storage2 partition where the copies are made to.
Rossum is the primary backup server where we run individual scripts to backup files from each server every night. After that is done, each file in Rossum is again backed up on Dijkstra every morning.
Logs of each backup are stored in /root/backup-scripts/logs, which is the same information that is sent out in the crontab emails.
The Crontab entries that manage each backup are set with a MAILTO directive, so the output of each script is sent to the notify mailing list.
Archival
Until 8/6/2024
| Contents | Directory | Script | Backup host and location |
|---|---|---|---|
| ECCS Homes | 10.10.10.15:/earlhamcs (bowie) | /root/backup-scripts/earlhamcs-backup.sh | Miyamoto:/storage/earlhamcs |
| Bowie PSQL | 10.10.10.15:/postgres (bowie) | /root/backup-scripts/postgres-backup.sh | Miyamoto:/storage/postgres |
| Bowie Docker | 10.10.10.15:/docker (bowie) | /root/backup-scripts/docker-backup.sh | Miyamoto:/storage/docker |
| Cluster Directory | 10.10.10.1:/cluster (hopper) [EXCLUDE genomes, metagenomes, fieldscience] | /root/backup-scripts/cluster-backup.sh | Hopperprime:/storage/cluster |
| Fieldscience Directory | 10.10.10.1:/fieldscience (hopper) | /root/backup-scripts/fieldscience-backup.sh | Hopperprime:/storage2/fieldscience |
| Genomes Directory | 10.10.10.1:/cluster/genomes (hopper) | /root/backup-scripts/genomes-backup.sh | Sakurai:/storage/genomes |
| Metagenomes Directory | 10.10.10.1:/cluster/metagenomes (hopper) | /root/backup-scripts/metagenomes-backup.sh | Sakurai:/storage/metagenomes |
| Server Files | $hostname:/etc,var,root,home | /root/backup-scripts/all-server-backup.sh | Sakurai:/storage/$hostname/etc,var,root,home |
Restore by copying the file back to the place it needs to be saved.
Each of the backup servers has one or more root crontab entries that call an rsync script in the servers /root/backup-scripts directory. This copies (pulls) the listed directories to their local backup. Each of the backup servers has a /storage and/or /storage2 partition where the copies are made to.
Logs of each backup are stored in /root/backup-scripts/logs, which is the same information that is sent out in the crontab emails.
The Crontab entries that manage each backup are set with a MAILTO directive, so the output of each script is sent to the notify mailing list.
Current backups table as of 8/14/2018
| Machine | Where | What |
|---|---|---|
| hopper | dali | /cluster, /etc, /var, /home |
| as0 | dali | /etc, /var |
| w0 | N/A | No rsync scripts |
| dali | dali | /var/opt/gitlab/backups, /root/gitlab-backups (not with rsync), /etc/ |
| lo0 | dali | /etc, /var |
| bronte | dali | /etc, /var, /home/nbserver |
| pollock | dali | /etc, /var |
| kahlo | dali | /etc, /var |
| tools.cs | dali | /mnt/lovelace/software, /etc, /var, /home/sage/sage-6.8 |
| net.cs | dali | /etc, /var |
| smiley.cs | dali | /etc, /var |
| proto.cs | dali not mounted | /etc, /var |
| shinken | N/A | No rsync scripts |
Status update 8/14/2018
All cluster servers can now back up to indiana.
Hopper is currently the only one set to actually do so, the rest are still going to Dali.
CS machines are not yet able to mount indiana.
Adam has made some notes at http://cluster.earlham.edu/wiki/index.php/Ccg-admin#Bacula_Backup_Management
Each machine needs to have bacula-fd installed and configured to point to quartz (the bacula director, a jail which is running on fluorite). This is 'yum install bacula-client'. This will put the fd config file at /etc/bacula/bacula-fd.conf.
Feldspar is the actual storage daemon.
Each machine that you want to be backed up must have an entry in the bacula-dir.conf file on quartz at /usr/local/etc/bacula-dir.conf. There's a password that the fd and dir uses to authenticate with each other, make sure the password in the bacula-dir.conf file matches the password in bacula-fd.conf on the client machine.
Here's what I know about bacula-dir.conf:
Clients - each machine you want to back up
IP address of machine
Name = hostname
Password to connect to fd on client
Jobs - specifying a backup job
Name = 'backup-<name of client>'
Specify the name of a JobDef
JobDef - Information about the backup
Where it goes
What Schedule to use
What FileSet to use
What pool to put it in, etc.
FileSet - the actual set of files to backup
Include the actual set of files that you want to be backed up
Exclude any files you don't
/usr/local/sbin/bconsole is the bacula console. When you modify the bacula-dir.conf file you need to reload it in bconsole. If you add a job or want to remove a job you can disable and enable them in the bconsole as well.
You can test that the file daemon and director can talk to each other by using bconsole.
Type 'status'
3 (client)
Choose client
It will say if it's connected or not
To run a backup test, type 'run'
Then select the job to run
Everything is set up for hopper to begin backups with bacula. I cannot get hopper to connect to the storage daemon (feldspar.earlham.edu). I suspect it could be something with the firewall on feldspar because we don't have any firewalls running on hopper.
Barring Bacula stuff noted above, here is the current state of the union concerning backups, which are currently run with the rsync tool.
Backups are stored on dali in /media/r10_vol/backups. Note that r10_vol is shared over nfs and mounted as /media/dali.
The machines noted below mount dali and write their backups every day at 7 am, via a script in the /root directory called rsync-[hostname].sh:
- hopper
- dali
- pollock
- kahlo
- al-salam node 0
- al-salam nodes 1-12 mount dali but do not currently back up data
- layout node 0
- Likewise, layout nodes 2 and 3 mount dali but do not currently back up data
- bronte (note: the backup script on bronte is called rsync-fatboy.sh)
The following machines backup by specifying the address of dali in their rsync script without mounting its storage drive:
- net
- web
- control
- smiley
The following machines do not currently backup with rsync:
- bigfe
- krasner
- tools
- proto
The following machines are down/inaccessible to me at time of writing and their backup status is unknown:
- elwood
- t-voc