Sysadmin:Environment Modules

From Earlham CS Department
Jump to navigation Jump to search

Environment Modules

Unless Tcl is your middle name, you might be wondering where exactly module load comes from. To help get you started, this is the project homepage and here is the official github repository if sourceforge isn't your thing.

About Environmental Modules

The 'Environment Modules' software is a way to easily and quickly change the environment variables of a user and a way that can be done by the user themselves. Without the environmental modules package, changing environmental variables typically consists of having to export each variable individually. Changing environmental variables is not always enough if one user wants to use a different version than default for some program. For example, the default version of gcc is 4.4.7 and lives in /usr/bin/gcc. I log in and I want to use gcc version 5.0. I don't have root privileges and gcc 5 isn't installed. I can't. I could ask the System Administrators to install it, but who knows how long that could take. Using 'Environment Modules' can help with that, both on the user side and the admin side. It provides much flexibility. If the admins have installed both gcc 4.4.7 and gcc 5.0 as modules, having gcc 4.4.7 as the default, then the user can type one command and change which gcc they want to use. Here's the command: module swap gcc/4.4.7 gcc/5.0. With that command, the user's environment is now setup to do anything with gcc, using gcc 5.0. Poof.

Installing Env Modules

The environment modules is a piece a software like any other. Here's a little step-by-step guide that I hope proves to be helpful.

  1. Using your favorite package manager (yum, apt, etc.), install tcl and tcl-devel.
  2. Obtain a tarball source, this link will probably be useful.
  3. Put the tarball in the directory you want to install modules in.
    • On CS: /mnt/lovelace/software
    • On Cluster: /mounts/<machine-name>/software
  4. Unpack the tarball
  5. Inside the source dir, run the configure script making sure to set the prefix to where you want Modules installed (should be one directory above the one you're in now), and then build it.
    • ./configure --prefix="/mounts/<machine-name>/software/Modules"
    • make && make install

Those steps hopefully came out without errors, and in the directory that you set as a prefix, you should see a new directory called Modules. That's where all of the compiled source you just built was installed and where each modulefiles that is written for software modules will live.

Once the installation is done, put a symbolic link in the /mounts/<machine-name>/software/Modules directory named default pointing to the version 

"cd /mounts/<machine-name>/software/Modules; ln -s 3.1.6 default"

Next, we'll want to make module actually available as a command. 

cp /mounts/<machine-name>/software/Modules/default/init/bash /etc/profile.d/modules.sh

If we start up a new shell, we should be able to test if everything is working by running module -V.

Configuring Env Modules

We have installed Environment Modules, now we need to configure it so that users are able to use it.

We'd rather users not have to run module load modules at every login. So as root, change /etc/bashrc or /etc/bash.bashrc (whichever exists) to include the following: 

. /mnt/lovelace/software/Modules/3.2.10/init/bash # or equivalent for the cluster realm
module load modules

However, this is already done on any system with a working module command (i.e module avail).

Installing Software as Module (Recipes)

You can check out the Newmodules page, which is related.

Explanation of ModuleFile syntax

  • prereq denotes any pre-requisite modules that must be loaded before this module can be loaded
  • set allows you to set variables, used in this context to assist with copy and paste
  • preprend-path adds the successive line to the beginning of the spceified location, for example PATH for directories to executables or MANPATH for paths to man entries

General Notes and Tips

  • It is really important that the version in the set VERSION line is the same as the name of the folder that you installed cuda to, since the version is used to reference the software in that directory.
  • After creating a new modulefile, do a module avail (as a user, not as root) to check if it shows up.
  • Make sure to load and test it before confirming that you are done.

OpenMPI

  • cd /mounts/layout/software
  • mkdir openmpi; cd openmpi
  • mkdir 3.0.0
  • wget https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.gz
  • tar xzf openmpi-3.0.0.tar.gz && cd openmpi-3.0.0
  • ./configure --prefix=/mounts/layout/software/openmpi/3.0.0/
  • make all && make install
  • Finally we need a module file (in modulefiles) specifying a basic configuration. Add module file as /mounts/layout/software/Modules/3.2.10/modulefiles/openmpi/3.0.0 - if confused, compare to the corresponding file in the other domain (e.g. if you're working on cluster, check out /mnt/lovelace/software/Modules/3.2.10/modulefiles/anyprogram on a CS machine)

OpenMPI Modulefile: 

#%Module1.0###############################################################

prereq modules

set VERSION 3.1.1

set LDIR /mounts/pollock/software

prepend-path PATH $LDIR/openmpi/$VERSION/bin

prepend-path MANPATH $LDIR/openmpi/$VERSION/share/man

prepend-path LD_LIBRARY_PATH $LDIR/openmpi/$VERSION/lib

prepend-path C_INCLUDE_PATH $LDIR/openmpi/$VERSION/include

prepend-path -d " " CPPFLAGS -I$LDIR/openmpi/$VERSION/include

prepend-path -d " " LDFLAGS -L$LDIR/openmpi/$VERSION/lib

Python

Installing a python version as a module follows similar, but slightly different steps. This difference comes because we use the installer anaconda to install python. Replace <SERVER_NAME> with the name of the machine you are working on. Replace Python 3.11 (py311) with whatever version of python you are installing.

  1. First, we need to install a new python version to Anaconda.
    1. Run conda create -n py311 python=3.11 (history | grep conda will show example commands).
    2. If the command is not found, try loading the "conda" module with module load conda.
    3. Check if the python env was created, run conda env list and the name and location of your new env should be listed.
  2. Now we need create a module file.
    1. cd /mounts/<SERVER_NAME>/software/Modules/default/modulefiles/python and look at example files to create one for desired version.
      • If there is not a "default" alias on the server you are working from, module avail will tell you what directory is being used for modulefiles.
    2. Check if it is available in module avail, then load it with module load python/3.11, and run python --version. You should get the same version that you just installed.

EXAMPLE MODULEFILE FOR PYTHON: 

#%Module1.0#####################################################################
prereq modules

# Conflicts with other python versions
#conflict python/2.7
#conflict python/3.4
#conflict python/3.5
#conflict python/3.3

# Change numbers to match your current python version
set VERSION 3.11
set DIR /mounts/<SERVER_NAME>/software/anaconda/envs/py27/envs/py311
prepend-path PATH $DIR/bin
prepend-path PYTHONPATH $DIR/lib/python3.11/site-packages
prepend-path MANPATH $DIR/share/man
module-whatis "Anaconda Python 3.11 binaries and modules."

Software with specific Python Version

Sometimes software requires a lot of specific python dependencies that we don't want in our "general use" python environments. Conda + Modules is the solution here, similar to above.

Note: check what python version you need for the software. This is important, and can be very specific.

  1. conda create -n {software name} python={version}
    • to check if {software name} environment was created, go to /mounts/{machine}/software/anaconda/envs/ and there should be a folder for it.
    • you can also use conda env list.
  2. Now we can install the software.
    • activate the environment you just created: conda activate {software name}.
    • use pip install or conda install to install, depending on the software specific instructions.
  3. Now we need create a module file
    • cd /mounts/{machine}/software/Modules/default/modulefiles/ create or enter the folder matching the name of the software you are installing.
    • create a modulefile with the name of the version you are using (the version of the software, not the python version).
    • take a look at some examples of similar things. MultiQC and Qiime2 on lovelace are good examples.
  4. do a module avail to check if version is present
  5. test the commands to make sure the software is functional.

CUDA

  • Make a directory for it in (something like /mounts/{machine}/software/cuda-{version})
  • wget [url] the .run file from the nvidia site into your new directory
  • sh cuda_{version}_otherstuff_linux.run --toolkitpath="/mounts/layout/software/cuda-{version}" (a menu will open)
  • Uncheck everything but the toolkit and the documentation, thats all we need:
????????????????????????????????????????????????????????????????????????????????
? CUDA Installer                                                               ?
? - [ ] Driver                                                                 ?
?      [ ] 450.51.06                                                           ?
? + [X] CUDA Toolkit 11.0                                                      ?
?   [ ] CUDA Samples 11.0                                                      ?
?   [ ] CUDA Demo Suite 11.0                                                   ?
?   [X] CUDA Documentation 11.0                                                ?
?   Options                                                                    ?
?   Install                                                                    ?


  • Go to options -> Toolkit options, and unselect "create symlink" and "Create desktop menu shortcuts" and "Install manpage", we wont need those since its a module.

(hit done and return to the previous screen) 

????????????????????????????????????????????????????????????????????????????????
? CUDA Toolkit                                                                 ?
?   Change Toolkit Install Path                                                ?
?   [ ] Create symbolic link from /usr/local/cuda                              ?
? - [ ] Create desktop menu shortcuts                                          ?
?      [ ] All users                                                           ?
?      [ ] Yes                                                                 ?
?      [ ] No                                                                  ?
?   [ ] Install manpage documents to /usr/share/man                            ?
?   Done                                                                       ?


  • Select library install path and change it to /mounts/{machine}/software/cuda-{version}/lib

(hit done and return to the previous screen)

select install

go to the modulefiles directory (probably /mounts/{machine}/software/Modules/default/modulefiles/)

if there is no cuda folder, make one. cd into cuda/ and make a new modulefile with just the version as its name (like "11.0"). edit it with something like this: 

#%Module1.0#####################################################################
prereq modules
set VERSION 12.1
set DIR /mounts/layout/software/cuda-$VERSION

setenv CUDA_HOME $DIR
prepend-path PATH $DIR/bin
prepend-path LD_LIBRARY_PATH $DIR/lib64
prepend-path C_INCLUDE_PATH $DIR/include
prepend-path C_INCLUDE_PATH $DIR/include/CL

module-whatis "cuda 12.1 runtime environment"
Retrieved from "https://wiki.cs.earlham.edu/index.php?title=Sysadmin:Environment_Modules&oldid=18127"