Selecting Particular Hardware

A number of Slurm mechanisms are available to select different hardware:

• partitions
• QOS
• gres
• constraint

Not all of these mechanisms need to be specfied when submitting, and are listed here only for completeness. Users have a default partition and QOS, so do not need to specify them when submitting jobs. However doing so will do no harm, and may remind them about their values.

Name	Default Value
partition	comp
QOS	normal

The values of the defaults is likely to change over time, as we add new hardware and optimize the system. Current values can be found with some of these commands. To view more information on the outputs, please use the manual pages.

• man scontrol
• man sacctmgr

Command	Values
scontrol show partitions	Lists All Our Partitions, currently short, comp and gpu
scontrol show partition comp	Detailed information on the comp partition including Maximum Wall Time (7 days) and Default Memory per CPU (4096M)
sacctmgr show qos normal format="Name,MaxWall,MaxCPUSPerUser,MaxTresPerUser%20"	Name MaxWall MaxCPUsPU MaxTRESPU normal 7-00:00:00 65 cpu=65,gres/gpu=3

Please note that MonARCH uses the QOS to control how much of the cluster one user can use. For the QOS normal a user has:

• a maximum of 65 CPUs (cores)
• a maxmimum of 3 GPU cards
• a maximum wall time of 7 days

Partitions Available

MonARCH hardware is split into several partitions.

The default partition for all submitted jobs is:

• comp for compute nodes

Other partitions include:

• short for jobs with a walltime < 1 day. This will run only on previously MonV1 hardware.
• gpu for the GPU nodes
• highmem a restricted partition for access to the 1.5TB server

Example: To use the short partition for jobs < 1 day, put this in your Slurm submission script.

#SBATCH --partition=short

Selecting a particular CPU Type

The hardware available consists of several sort of nodes: All nodes have hyper-threading turned off. In the Slurm configuration file, certain nodes are tagged with Features (a text string) that can be used to identify them.

• mi* nodes are 36 core Xeon-Gold-6150 @ 2.70GHz servers wtih 158893MB usable memory
• gp* nodes are 28 core Xeon-E5-2680-v4 @ 2.40GHz servers with 241660MB usable memory. Each gp server has two P100 GPU cards.
• mk* nodes are 48 core Xeon-Platinum-8260 @ 2.4GHz servers with 342000M usable memory.
• md* nodes are 48 core Xeon-Gold-5220R @ 2.20GHz servers with 735000M usable memory. Each server has two processors with 28 cores each.
• hm00. This single node is 36 core Xeon-Gold-6150 @ 2.7GHz server with 1.4TB usable memory.

Sometimes users may want to constrain themselves to use a particular CPU type, e.g. for timing reasons. In this case, they need to specify this with a constraint flag in the Slurm submissions script. As a parameter, this flag specifies the CPU type needed. The constraint flag in the job submission script must match the string in the feature field in the Slurm configuration file.

The CPU type of a particular node can be viewed by runinng this command:

scontrol show node <nodename>

command and then looking for the Feature field.

Examples:

# this command requests only mi* nodes that have Xeon-Gold processors
#SBATCH --constraint=Xeon-Gold-6150

This feature should only be used if you must have a particular processor. Jobs will schedule faster if you do not use it.

Selecting a particular server

Users can specify to use only a particular server if they wish. This could be useful if doing timing experiments that require multiple jobs to run on exactly the same server. Please note that if you use this feature, your job will take longer to schedule, as you will be ignoring similar nodes that might be idle.

Example: Only run jobs on server ge00

#SBATCH --nodelist=ge00

Selecting a GPU Node

To request one or more GPU cards, you need to specify:

• the gpu partition (i.e. or in general, a partition that has a GPU card in it)
• the name and type of GPU in a gres statement. Your running program will only be allowed access to the number of cards that you specify. i.e. a command like nvidia-smi ran in the context of the Slurm job, will only show the GPU allocated to you.

You should not use the constraint feature described above.

# this command requests one P100 card on a node. Run *show_cluster*  for a  list of the available GPUs
#SBATCH --partition=gpu
#SBATCH --gres=gpu:P100:1