Deep Learning Workflow
Where to store data?
For your project to be successful, it is critical that you have the ability to iterate over the training process in a timely manner. One thing you should make sure of is that reading the data is not costly.
Your directory is a very bad place to store your data for a simple reason: user directories are not stored on the local disk of every machine, they are mounted via NFS over the network. Transferring data over the network is much, much slower than reading it from the disk (keep in mind that you also have to first read data from the disk, in order to send it over the network).
In order to avoid this pitfall, simply put the data on the local disk. /mounts is a standard directory we’ve been putting data. If you don’t have access to this directory, just shoot an email to sysadmins and we will help you out.
Deep learning has been around for almost half a century, but only recently it has become a prevalent machine learning method. The popularity of deep learning has risen dramatically since the 2000s and it was due to two main factors:
- Mass digitization provided computer scientists with a ton of data that is necessary to train models that generalize well.
- The computation power has become cheaper and more accessible. Especially with the arrival of GPUs, training the models has become faster than ever. You too have to leverage these two factors to make sure your project is successful.
GPU vs CPU
The majority of the work done by neural networks is just matrix multiplication. This operation is highly parallelizable and GPUs are designed for parallel processing of the data. Let's conduct an experiment to demonstrate how much faster GPUs are when it comes to training neural networks. I will train a simple CNN on digit classification.
from tensorflow.keras.datasets import mnist import os os.environ["CUDA_VISIBLE_DEVICES"] = "-1" #if commented, runs on GPU, otherwise CPU import tensorflow as tf from tensorflow.keras import Sequential, datasets, layers, models train_X, train_y), (test_X, test_y) = mnist.load_data() height = train_X.shape width = train_X.shape num_classes = 10 model = Sequential([ layers.experimental.preprocessing.Rescaling(1./255, input_shape=(height, width,1)), layers.Conv2D(32, 3, padding='same', activation='relu'), layers.MaxPooling2D(), layers.Conv2D(64, 3, padding='same', activation='relu'), layers.MaxPooling2D(), layers.Conv2D(128, 3, padding='same', activation='relu'), layers.MaxPooling2D(), layers.Flatten(), layers.Dense(256, activation='relu'), layers.Dense(num_classes) ]) model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy']) epochs=10 import time start = time.time() history = model.fit( train_X, train_y, epochs=epochs, batch_size=64 ) end=time.time() print("Time taken:",end-start)
CPU with 16 cores: 188.98 s
GPU: 54.0 s
As you can see I achieved 3.5x acceleration using GPU, even though I was using all 16 CPU cores.
Setting up environment for GPU
Layout is a good place to do machine learning projects. We will be adding GPUs to other machines soon so this info might change. One way to check the specs of gpu is to run the command :
$ lshw -C display
On layout this outputs:
- description: VGA compatible controller
- product: GK110 [GeForce GTX 780]
- vendor: NVIDIA Corporation
- physical id: 0
- bus info: pci@0000:03:00.0
- version: a1
- width: 64 bits
- clock: 33MHz
- capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
- configuration: driver=nvidia latency=0
- resources: irq:181 memory:de000000-deffffff memory:d0000000-d7ffffff memory:d8000000-d9ffffff
- ioport:8000(size=128) memory:df000000-df07ffff
Depending on the model and the make of GPU other commands might also be available. For example, for nvidia gpus you can display the info about the current state of GPU using the command:
(Note: you can use this command to see whether or not the resources are busy).
The way we manage different software versions and environments is through Modules. You can display the available modules via command: module avail If you run this command on layout you will see different versions of python, conda and cuda modules. Different python versions have different tensorflow versions installed.
You can see tensorflow’s compatibility chart with python, cuda and cudnn here: https://www.tensorflow.org/install/source#gpu
The latest version of tensorflow (2.3.1) is available on python/3.7 and it’s compatible with cuda/10.1.
If you are planning to use this version of tensorflow, then simply run these two commands:
$module load python/3.7 $module load cuda/10.1
It’s very convenient to run DL/ML projects on python notebooks for multiple reasons. It’s easier to visualize the data, you can make changes on the fly, it helps you to make sure everything is set up correctly before you start running lengthy experiments, etc.
We have set up a jupyterhub instance on layout that is designed to support DL projects. (Link: https://lo0.cluster.earlham.edu/jupyterhub) It comes prepackaged with cuda10.1, tf2.3.1, py3.7 -> it automatically runs the tensorflow projects on the available GPUs. Just to make sure everything’s set up correctly, use this python script: