Skip to main content

Effective Usage

Since the cluster is a shared resource, It is important to consider the efficiency of your utilisation. Not only does this make you a good citizen on the cluster, it may even give you a faster time to solution.

When a job is submitted, it waits in queue before being executed by the compute nodes. Many things go into calculating the order of this queue, but you can control some of them - specifically the more resources you request, the longer it may take for those resources to become available. You can decrease your overall time to solution by specifying the minimum requirements for your job, reducing the unnecessary queue time.

Some key resources to consider are:

  • Job wall time
  • Number of CPU cores
  • CPU memory (RAM)
  • GPU memory (VRAM) and GPU selection
  • Number of GPUs
  • Number of nodes

Job wall time​

The wall time is the time limit for your job. After the wall time is reached the job will be killed if it has not completed. To choose an effective wall time, try to consider an accurate wall time, but leave a buffer of extra time for safety. Other considerations include check pointing your work so that in the case that your job DOES end early (due to crash or termination) the work can be resumed from the last checkpoint.

Number of CPU cores​

Ensure that you are executing parallel code before requesting multiple CPU cores. Running your code with more cores will not in of itself increase the speed of the execution if the code has not been written to take advantage of the extra resources.

CPU memory (RAM)​

Trying to use more RAM than allocated will cause an 'out of memory error'. One way to check the memory usage of your workload is to use our jobstats command. Another way is to run your job on your laptop or workstation and checking the activity monitor. If your job is too big to run on your workstation, consider using an interactive terminal job on our cluster and running htop -u $USER, looking in the RES column.

As with CPU cores, requesting more RAM will not in of itself accelerate your code.

GPU memory and GPU selection​

Often times, the best GPU for your workload is the one that is available right now. High end GPUs are a highly requested resource. Using a slower GPU may take longer to execute, but as they are lower in demand you may be allocated faster, reducing your overall time to solution.

That being said, like with CPU memory, trying to use more than available will cause a 'CUDA out of memory error'. You need to consider the minimum VRAM that your process can run on when choosing a GPU. You may also need to tune parameters (like batch size for machine learning users) to make sure that your process can fit on your selected GPU.

Like with other resources, requesting more GPUs than your code is designed to handle will not in of itself make your code run faster. Splitting your work across mutliple GPUs can accelerate your code, but consider the overhead of the complexity of your code and the extra work that the process needs to do to communicate between the GPUs. Consider completely exhausting the compute capabilities of a single GPU and ensure that the GPU is the bottleneck in your process before requesting multiple GPUs.

For more information about available GPUs on M3 visit this page.

Number of nodes​

You should always specify the number of nodes when requesting resources since requesting more than one task without specifying the number of nodes can result in your job being split across nodes, leading to inefficiencies or in the worst case your job not finishing within the wall time.

Consider using all of the CPU cores on a node before requesting an additional node and make sure that your code is written to support multiple nodes before requesting the resources.

For more information on the nodes available in which partitions, see this page. The most up-to-date way of seeing all partitions on M3 is to use the show_cluster command.

How do I check my usage?​

  • If you are running a job on a node you can ssh into it. You can determine the job name with squeue -u $USER. Once inside the node you can:
    • top
    • htop
    • nvidia-smi (for GPU usage)
    • attach debuggers to the running code
  • The tool jobstats shows you your average usage in real time.

Common problems you may encounter​

  • You ask for more cores/tasks in Slurm than your program actually uses.
    • Check the program parameters so that it uses the correct number of cores. Using Slurm environment variables may make your scripts more robust. See man sbatch for a list of Slurm environment variable set inside a job, i.e. SLURM_NTASKS, SLURM_CPUS_PER_TASK
    • Do not be confused when running Array jobs. Each Array Job is a unique process using the parameters that you request in the job. See here for more information on requesting Array Jobs.
  • Your job does not scale well. Doubling the number of cores may not halve the run time.
  • MPI programs not being called properly. See here for more information on MPI programs.
  • You did not specify #SBATCH --gres= when wanting a GPU. See here for more information on requesting GPUs.
  • You requested all the cores/memory on a server, when all you were using was 1 core and a GPU.
  • You requested more than 1 GPU but your software can only use 1 GPU.