Skip to content

Using the cluster

Author: R. Bordas bordasraph@gmail.com

As a general recommendation, do not create nor use Python environments from the /neurospin/... storage server. Everything described below should be performed on the homedir of alambic.

Be careful that while all unicog workstations use the same homedir, alambic has a different one.

First step: 

ssh alambic # if no access, ask GIPSI
Second step: Choose and setup a Python environment. See details below.

Third step: Submit jobs using sbatch, e.g., if define_job.sh contains 

#!/bin/bash

#SBATCH -p nspin_8
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task 8
#SBATCH --mem=8g

source envs/mne/bin/activate
python /path/to/my-script.py
then just do bash define_jobs.sh. See docs here for details.

uv: the fast way

Installation

As fast as 

curl -LsSf https://astral.sh/uv/install.sh | sh

See here for details.

Usage

Environment creation and usage

Option 1: default name and location 

uv venv
# it creates the env in a .venv folder in the current working directory
# then uv always load this environment if you stay in that directory (e.g., your project)

Option 2: full customization (similar, but faster than pyenv imo)

# let's say I want to install several environments accessible outside my projects,
# such as in /home/rb266841/envs. Let's create a mne environment inside this folder
# it is also useful to specify the python version you wish to use
uv venv envs/mne --python 3.11

source .venv/bin/activate

In any case, once you've created an environment and are using it, just prepend uv to any usage of pip you are use to.

MNE installation

uv pip install "mne[full]"
# you've never installed mne so fast :)

Installing specific version

uv pip install "mne=1.5.1"

Projects

A better alternative to requirements.txtseems to be the uv projects. Just create a pyproject.toml file: 

[project]
name = "my-project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "mne>=1.11",
    "fooof==1.1.0",
    ...
]

Apptainer: the proper but geeky way

Define an apptainer

Bootstrap: docker
From: python:3.11-slim

%labels
    Author your_name
    Version 1.0
    Description Python environment with NumPy

%post
    # Update package list and install system dependencies
    apt-get update && apt-get install -y \
        gcc \
        g++ \
        && rm -rf /var/lib/apt/lists/*

    # Upgrade pip and install numpy
    pip install --no-cache-dir --upgrade pip
    pip install --no-cache-dir "mne[full]"

%environment
    export LANG=C.UTF-8
    export LC_ALL=C.UTF-8

%runscript
    exec python "$@"

Here a version when there are custom packages to load (editable directories added to the environnent)

Bootstrap: docker
From: python:3.11-slim

%arguments
    SCRIPTS_PATH=/neurospin/meg/meg_tmp/2024_alpha_TW_Bordas/scripts_tw

%files
    /neurospin/meg/meg_tmp/2024_alpha_TW_Bordas/scripts_tw/pyproject.toml /opt/scripts_tw/pyproject.toml

%post
    apt-get update && apt-get install -y --no-install-recommends \
        git \
        curl \
        && apt-get clean && rm -rf /var/lib/apt/lists/*

    curl -Ls https://astral.sh/uv/install.sh | sh
    export PATH="/root/.local/bin:$PATH"

    uv pip install --system --upgrade pip
    uv pip install --system mne libigl statsmodels openpyxl pycircstat2 pyvista pyarrow odfpy joblib numba scikit-learn

%environment
    export PATH="/root/.local/bin:/usr/local/bin:$PATH"
    export PYTHONUNBUFFERED=1

%runscript
    SCRIPTS_PATH="{{ SCRIPTS_PATH }}"
    uv pip install --system -e "$SCRIPTS_PATH"

Create the apptainer

apptainer build traveling_alpha.sif traveling_alpha.def

Use srun (submit a job) if you need more resources than allowed by default by the cluster manager.

export APPTAINER_TMPDIR=/home/rb266841/Documents/traveling_alpha/tmp/
mkdir -p $APPTAINER_TMPDIR
srun --mem=16g --cpus-per-task=4 apptainer build traveling_alpha.sif traveling_alpha.def --force

Bash script to use

The simplest way is to create a bash script and populate it with the required paths: 

#!/bin/bash

CONTAINER=$HOME/Documents/traveling_alpha/traveling_alphs.sif
SCRIPTS=/neurospin/meg/meg_tmp/2024_alpha_TW_Bordas/scripts_tw
PERM_SCRIPTS=analysis/bursts_vs_tw/regressions/permutations

sbatch --parsable --mem=32G --time=00:30:00 --partition=nspin --wrap=" \
    apptainer exec \
        --bind /neurospin:/neurospin,$HOME$HOME,$SCRIPTS:/opt/scripts_tw \
        --pwd $SCRIPTS \
        --env PYTHONPATH=/opt/scripts_tw/src \
        $CONTAINER \
        python $PERM_SCRIPTS/prepare_permutations.py

A useful feature is slurm arrays. For example, you can submit a job per participant. Thus, instead or parallelizing your computation participant-wise, you submit one job / participant and parallelize computations inside each participants. Because some queues allow the execution of up to 32 jobs (depending on available resources), this is the fastest and most efficient way!

Below is the example of the usage of slurm for the same computation, but with different hyperparameters. Replace EXTRA_ARGS by your sub_id. 

 #!/bin/bash

#SBATCH -p nspin_8
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task 8
#SBATCH --mem=8g
#SBATCH --output=tw_ddg.%j.out
#SBATCH --job-name=tw_ddg
#SBATCH --array=0-3

# export MNE_SKIP_MEMORY_CHECK=1

case ${SLURM_ARRAY_TASK_ID} in
    0)
        EXTRA_ARGS=""
        ;;
    1)
        EXTRA_ARGS="--pop-basis"
        ;;
    2)
        EXTRA_ARGS="--pop-basis --use-3d"
        ;;
    3)
        EXTRA_ARGS="--use-3d"
        ;;
esac

apptainer exec \
    --bind /neurospin:/neurospin,$HOME:$HOME,/neurospin/meg/meg_tmp/2024_alpha_TW_Bordas/scripts_tw:/opt/scripts_tw \
    --pwd /neurospin/meg/meg_tmp/2024_alpha_TW_Bordas/scripts_tw \
    --env PYTHONPATH=/opt/scripts_tw/src \
    $HOME/Documents/traveling_alpha/traveling_alpha.sif \
    python analysis/discrete_calculus/compute_calculus_operators.py \
        --config-name=params_rsMEG \
        --condition=rsMEG \
        --compute characteristics \
        --overwrite \
        ${EXTRA_ARGS}

Run it

bash compute_gradients.sh

The pyenv way

See also documentation on the wiki

What is it?

pyenv is a python package manager. It allows creating separate environments with different packages, in the same way that anaconda does. Christophe Pallier installed a common UNICOG pyenv environment on the neurospin server, which allows using it through any computer connected to it. Importantly, this includes narval.

How to use it?

pyenv is already installed in neurospin/unicog/resources/pyenv.

If the pyenv is not found, you need to add to following lines at the end of your $HOME/.bashrc file: 

export PYENV_ROOT=/neurospin/unicog/resources/pyenv

command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"

eval "$(pyenv init -)"

# eval "$(pyenv virtualenv-init -)"

Warning: always create a copy before modifying you .bashrc! To do so, simply use: cp ~/.bashrc ~/.bashrc.bak

Editing a file in command line: use vi (press "i" to insert text, then press ":x" to save and quit) or nano (use ctrl+x to exit; you'll be asked if you want to save, press Y or N; then press enter to leave)

A virtual environment named unicog_3.11.2 based on Python 3.11.2 and comprising the most relevant packages for unicog users exists (this includes mne). To activate it, just run:

pyenv activate unicog_3.11.6

You can also check the full list of environments with 

pyenv versions

If when you try to create your virtual environment you get this error: pyenv: no installed versions match the prefix-q', try this command:PYENV_DEBUG=1 pyenv virtualenv 3.11.6 env_name`

A few useful commands

You can check which version of python is installed using pyenv python.

You can find which version of mne is installed with python -c 'import mne; print(mne.__version__);'

You can list all packages, with their version, using pip freeze

To manipulate environments:

# activate an environment
pyenv activate {env_name}

# deactivate the current environment
pyenv deactivate

# create a new environment
pyenv virtualenv {env_name}

# create a new environment with a specific python version (which must have been installed previously)
pyenv virtualenv {python_version} {env_name}

# list all available environments
pyenv virtualenvs

# associates the current directory to {env_name}
pyenv local {env_name}

Install pyenv from scratch

PYENV_ROOT=/opt/pyenv # for example

mkdir -p "$PYENV_ROOT"

git clone https://github.com/pyenv/pyenv.git "$PYENV_ROOT"

git clone https://github.com/pyenv/pyenv-virtualenv.git
"$(PYENV_ROOT)/plugins/pyenv-virtualenv"

Then add this to your bash:

export PYENV_ROOT="/opt/pyenv"

command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"

eval "$(pyenv init -)"

eval "$(pyenv virtualenv-init -)"