Skip to main content

Using Python in HPC Environments

Python is a commonly used programming language used in scientific research. This page describes some of the best-practice methods of using Python on HPC Clusters.

A number of Python executables will be available to you on any HPC system. There will be a system level one (say in [/usr/local/bin]) and others will be available via Environment modules, i.e. [module avail python]. There will be other versions of Python available, say Anaconda [module avail anaconda], which is Python with the conda package manager, and the Intel Version [module avail intel-python])

One feature off all these Pythons is that they are static, i.e once installed we do not modify them to add or update libraries. This is to ensure reproducibility of users' code, i.e. updating one library may cause many other packages to be updated.

You can't modify Python distributions

Python distributions are in read-only folders, and you can not modify them.

If you need to update or modify a Python distribution, the solution is to use virtual environments so you can modify and maintain your own library system.

What is a Virtual Environment?​

A Python virtual environment is a folder structure that gives you everything you need to run a lightweight yet isolated Python environment.

You can create, modify and delete virtual environments. To use one, you activate it, and when finished, you can also deactivate it

If these folders are in your HOME directory, then it will fill up quickly. We recommend users place the virtual environments in the project or scratch folders.

Vanilla Python​

You no longer need to load any Python version via Software Modules

The default OS Python (/usr/bin/python3) should be sufficient for you to make most software enviroments.

#create a virtenv using OS Python
python3 -m venv /path/to/new/virtual/environment

#activate a virtual environment
source  /path/to/new/virtual//bin/activate

#check it is activated
which pip3

#install our software
python3 -m pip install tensorflow
#test. E.g. on a node with a GPU you  can type
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))

#deactivate a virtual environment
deactivate

Conda​

For a guide on using conda on MonARCH, please see our Conda page.