Apptainer

See also: HPC and HTC tutorial | Run MPI On Grid'5000 | GPUs on Grid5000 | GNU Parallel | Modules | Conda | Guix | Apptainer | Deep Learning Frameworks

	Note
	This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Apptainer is a popular container solution for HPC systems. It natively supports GPU and high performance networks in containers and is compatible with Docker images. Grid'5000 supports the Apptainer containers. It is available on the base system and does not require root privileges. More info at: https://apptainer.org/

Apptainer is a successor-fork of Singularity and should be a drop-in replacement for the latter.

Basic usage

Apptainer 1.4.5 is available as part of the of the standard Debian 13 install on Grid'5000. And is accessible via the apptainer command. For compatibility with previous workloads the singularity command also executes Apptainer.

The Apptainer user documentation is available at https://apptainer.org/docs/user/1.5/. It describes the various ways to run programs inside a container and how to build your own container image.

Building a Apptainer image

Recent versions of Apptainer allow building images without root access:

node:

apptainer build mpi.sif mpi.def

For more information about building Apptainer containers, see https://apptainer.org/docs/user/1.5/build_a_container.html

Using docker containers with Apptainer

Apptainer can also be used to start docker containers. For instance:

node:

apptainer run docker://debian

Running Apptainer containers in OAR submission

Apptainer containers can also be run in an OAR submission (non-interactive batch job). For instance:

frontend:

oarsub -l core=1 "apptainer run docker://ghcr.io/apptainer/lolcow"

Running MPI programs in Apptainer containers

MPI programs may be run in Apptainer containers, by leveraging both the MPI implementation available in the host, i.e. a Grid'5000 physical node (which has direct access to the high-performance network hardware if present), and the MPI library that must be installed inside the container. MPI programs in the Apptainer container can then be started using the the mpirun command on the host.

See https://apptainer.org/docs/user/1.5/mpi.html for more information.

For instance, to submit such an MPI job under OAR, assuming an Apptainer image named my_mpi_image.sif in your home directory, use:

frontend:

oarsub -l nodes=2 "mpirun -hostfile \$OAR_NODE_FILE --mca orte_rsh_agent oarsh -- `which apptainer` exec my_mpi_image.sif /opt/mpitest"

Using GPUs in Apptainer containers

GPUs available in the host can be made available inside the container by using the --nv option (for Nvidia GPUs only).

For instance, to start an interactive tensorflow environment with one GPU, first submit the job reserving 1 GPU:

frontend:

oarsub -I -l gpu=1

	Note
	You may need to add "-q production" or "-t exotic" depending which GPU cluster you want to use

Then on that node:

node:

apptainer run --nv docker://tensorflow/tensorflow:latest-gpu

More info at: https://apptainer.org/docs/user/1.5/gpu.html

Example: Using Apptainer to port a software environment between HPC infrastructures

Using Apptainer is a good way to port software environments between HPC infrastructures, for example, between Grid'5000 and IDRIS' Jean Zay.

The following example describes how Apptainer could be used together with Conda to share your software environment on two different HPC infrastructures (it is inspired by this stackoverflow question).

Step 1 - On Grid'5000, create a Docker container with your Conda environment

(based on the micromamba Quick Start guide)

Create a env.yaml file to describe your Conda environment:

name: base
channels:
  - conda-forge
dependencies:
  - tensorflow-gpu

Create a Dockerfile:

FROM mambaorg/micromamba:latest
COPY --chown=$MAMBA_USER:$MAMBA_USER env.yaml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml && \
    micromamba clean --all --yes

Create a Docker image using this environment: (see Docker)

node:

g5k-setup-docker

node:

docker build --tag my_app .

Check your docker image:

node:

docker run -it --rm my_app python3 -c "import tensorflow as tf; print('Num GPUs Available:', len(tf.config.list_physical_devices('GPU')))"

Step 2 - On Grid'5000, convert your Docker image to a Apptainer/Singularity image

Export your Docker image:

node:

docker save -o my_app.tar my_app

Convert it to a Singularity image:

node:

apptainer build my_app.sif docker-archive://my_app.tar

Test your Apptainer image:

$ apptainer shell --nv my_app.sif 
Apptainer> eval "$(micromamba shell hook --shell bash)"
Apptainer> micromamba activate
(base) Apptainer> python -c "import tensorflow as tf; print('Num GPUs Available:', len(tf.config.list_physical_devices('GPU')))"
Num GPUs Available: 2
(base) Apptainer>

Step 3 - Copy your Apptainer image to Jean Zay (using scp/rsync) and run it there

Note that unless you added Grid'5000's external addresses to your IDRIS account as described in the FAQ, you must copy your image locally and then copy it to Jean Zay.

Specific information about running Singularity containers on Jean Zay is available at http://www.idris.fr/eng/jean-zay/cpu/jean-zay-utilisation-singularity-eng.html.

$ module load singularity
$ idrcontmgr cp my_app.sif 
1 file copied.
$ singularity shell --nv $SINGULARITY_ALLOWED_DIR/my_app.sif 
Singularity> eval "$(micromamba shell hook --shell bash)"
Singularity> micromamba activate
(base) Singularity> python -c "import tensorflow as tf; print('Num GPUs Available:', len(tf.config.list_physical_devices('GPU')))"
Num GPUs Available: 1
(base) Singularity>