Skip to main content

Conda

conda is a cross-platform and language-independent package and environment manager. Users often rely on conda to make isolated and reproduceable environments, particularly for Python, though conda repositories also offer many packages for R, Perl, and other languages.

To use conda on the cluster, use the miniforge module. You can then create your own conda environments and activate environments using conda activate. This module includes mamba, which is a faster drop-in replacement for conda.

If this is your first time using this module, you should run some configuration commands once, but you will never need to think about these again afterwards.

Configuration (do this only once)​

To check if you have already done this before, run the following command.

[lexg@monarch-login4 ~]$ cat ~/.condarc 
pkgs_dirs:
- /home/lexg/scratch/conda/pkgs
envs_dirs:
- /home/lexg/scratch/conda/envs

If you see something similar to the above, where pkgs_dirs and envs_dirs are both set to scratch directories, then you can skip this configuration section! Otherwise, run the below commands.

$ module load miniforge3
$ CONDA_HOME=~/scratch/conda
$ conda config --add pkgs_dirs $CONDA_HOME/pkgs
$ conda config --add envs_dirs $CONDA_HOME/envs
$ echo "export PS1" >> ~/.bashrc

That's it, you should never need to read this section again now! Read on if you want more detail though...

This configuration ensures that your conda environments are installed in your scratch directory. This is our recommended location for conda environments because the default location (your home directory) has too little storage quota for conda environments, and we prefer that conda environments are not backed up (as in /mnt/lustre/projects/) since our backup system is not suited for the many little files that conda produces. The conda config commands achieve this.

The export PS1 line ensures that you will see the name of your active conda environment appear in your shell prompt. For example, when I activate my environment called my-env, I see my prompt is prefixed with (my-env) as below.

[lexg@monarch-login4 ~]$ module load miniforge3/24.3.0-0 
(base) [lexg@monarch-login4 ~]$ conda activate my-env
(my-env) [lexg@monarch-login4 ~]$

Creating and activating a conda environment​

As an example, let's create and activate an environment called test-env with Python 3.11 installed.

[lexg@monarch-login4 ~]$ module load miniforge3
(base) [lexg@monarch-login4 ~]$ mamba create -y --name test-env python=3.11
(base) [lexg@monarch-login4 ~]$ mamba activate test-env
(test-env) [lexg@monarch-login4 ~]$

Note we use mamba here since it's a faster drop-in replacement for conda, but the conda command is still available to you.

Let's now verify that this environment has Python 3.11.

(test-env) [lexg@monarch-login4 ~]$ python --version
Python 3.11.9
(test-env) [lexg@monarch-login4 ~]$ which python
~/scratch/conda/envs/test-env/bin/python

For more details on using conda in general, please see the conda docs.

Using conda in scripts​

Using conda inside shell scripts (e.g. when preparing SLURM scripts <Quick Start> ) the same as using it outside of scripts! For example, I can activate my test-env environment in a script like so.

[lexg@monarch-login4 ~]$ cat test-conda.sh 
#!/bin/bash
module load miniforge3
conda activate test-env
python --version
which python
[lexg@monarch-login4 ~]$ ./test-conda.sh 
Python 3.11.9
~/scratch/conda/envs/test-env/bin/python

Note on conda init and your bashrc​

You should not need to run conda init. This is not a huge issue on MonARCH, but to avoid unexpected behaviour it is probably a good idea to undo this command if you have run it before.

To check if you have run this before, run cat ~/.bashrc and search for something like the below snippet.

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/apps/miniforge3/24.3.0-0/miniforge3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/apps/miniforge3/24.3.0-0/miniforge3/etc/profile.d/conda.sh" ]; then
        . "/apps/miniforge3/24.3.0-0/miniforge3/etc/profile.d/conda.sh"
    else
        export PATH="/apps/miniforge3/24.3.0-0/miniforge3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

If you see a similar snippet, consider deleting it (e.g. with nano or vim, or whichever text editor you prefer!).