Running Multi-threaded Jobs
Multi-threading is a type of execution model that allows multiple threads to exist within the context of a process. Simply speaking, a Slurm multi-threading job is a single process, multi-core job. Many application can belong to this category, e.g.
- OpenMP programs,
- Matlab programs with (Parallel Computing Toolbox) enabled,
- Custom propgrams that use a threading library
A sample Slurm script for a multi-threaded job
#!/bin/bash
#SBATCH --job-name=MyJob
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
./myprog.exe
This script tells Slurm this is a multi-threading job. It has only 1 Unix process, but that process needs 8 CPU cores to execute. You will need to tell your program how many cores you have. This depends on the program - they may do this different ways.
- a command line option e.g.
myprog.exe --cpus=8 - a shell environment variable e.g.
export OMP_NUM_THREADS=8 - the program may try and deduce it itself
A sample openMP program
This is an example on now to run an OpenMP job using Slurm. OpenMP is a parallel technology built into most compilers. It uses multiple threads in a single Linux process, i.e. an OpenMP program can not span multiple computers, and must be run on a single server.
To compile your code with OpenMP:
| Compiler | Option To Use When Compiling |
|---|---|
| gcc | -fopenmp |
| icc | -openmp |
To submit to Slurm
#!/bin/env bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --ntasks-per-socket=1
export OMP_NUM_THREADS=8
./openmp_program.exe
Please note:
- Set OMP_NUM_THREADS to the same value as --cpus-per-task
- Your script can become more robust if you use a Slurm environment variables. Note that this variable is only set when the job is running in a Slurm environment
export OMP_NUM_THREADS=$SLURM_JOB_CPUS_PER_NODE
./openmp_program.exe
Mapping to hardware (NUMA)
The Slurm commands given above do not specify how the threads are mapped to the hardware.
Here is one example on how the threads might be mapped. Note that the one Unix Process (red line) has 8 cores on one socket
However this can also be mapped a different way. Note that the cores are now mapped across two sockets.
Users can give Slurm more information to Slurm on how to map threads with the --cores-per=socket flag e.g.
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --cores-per-socket=4
Hybrid OpenMP/MPI Jobs
It is possible to run MPI tasks which are in turn multi-threaded, e.g. a program that uses both MPI and OpenMP technologies. Here is en example to assist you.
Two nodes with 1 MPI process per node and 16 OpenMP threads each
#!/bin/env bash
#SBATCH --job-name=MPIOpenMP
#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --time=00:05:00
export OMP_NUM_THREADS=16
srun ./HybridMpiOpenMPI.exe