Ubuntu-ARM64-L4T environment

From Grid5000
Jump to navigation Jump to search

This tutorial describes how to build a kadeploy environment for the Nvidia AGX Xavier SoC nodes of the estats cluster in Toulouse. The operating system installed in this environment is Nvidia's Ubuntu 20.04 with the addition of Nvidia's Linux for Tegra layer.

Deploying Nvidia's supported Ubuntu 20.04 with L4T allows one to easily install Nvidia's tools such as the Tegra-specific version of Cuda.

Note.png Note

The Tegra-specific version of cuda is not provided in the environment deployed by default on the estats nodes, which is a Debian that however supports the Grid'5000 stack with the oar-node service, g5k-check, sudo-g5k, etc. Please also note that some classical GPU tools (such as nvidia-smi) do not support the Tegra GPU.

More info on Nvidia's website: https://developer.nvidia.com/embedded/jetson-linux-r3531

An Ubuntu ARM64 system including Nvidia's L4T

Pre-built environment

Grid'5000 provides the ubuntu2004-nfsl4t environment (see the kaenv output in the toulouse site), which supports the Grid'5000 accounts (ldap & nfs) additionally to the L4T specific services, Nvidia Jetpack (Cuda & more) and Jetson stats (jtop). This environment complies with the build process of all other Grid'5000 supported environments.

Note.png Note

Among the Grid'5000 supported environments, only the ubuntu2004-nfsl4t environment is compatible with the estats cluster. Other ARM64 environments do not support the estats hardware (the OS will not boot). Likewise, the ubuntu2004-nfsl4t environment does not boot on the pyxis cluster in Lyon, even though it's ARM64 as well.

The kameleon recipe used to build the ubuntu2004-nfsl4t environment follows closely what Nvidia documents (although not using Nvidia's sdkmanager, which is cumbersome and optional), with only minimal required tricks.

  • The build installs Ubuntu 20.04 in a VM
  • Install Nivdia's Linux for Tegra inside + the CTI Board Support Package (required by the estats hardware: Connecttech UAGX).
  • Run the Linux for Tegra tool (apply_binaries.sh) that adds to the rootfs everything that is required for the SoC to function.
  • Installs the nvidia-jetpack package set and the jetson-stats python tool.
  • Export the rootfs as a new kadeploy environment (ARM64), ready to boot estats nodes.

More details in the recipe itself: https://gitlab.inria.fr/grid5000/environments-recipes/-/blob/master/ubuntul4t200435-arm64-big.yaml

Custom environment

Furthermore, Grid'5000 provides the from_grid5000_environment/ubuntul4t200435-arm64-custom.yaml kameleon recipe template, which allows one to build a custom Ubuntu 20.04 environment that can be deployed on the estats cluster. This recipe is easier to read and quicker to build (because its build uses docker and runs on any X86 machine). Thanks to it, a custom kadeploy environment can be built with for instance:

  • more tools installed
  • a different on-boot NVP model
  • ...

Deploying the pre-built ubuntul4t200435-arm64-big environment

Example of a kadeploy command to deploy the pre-built environment on a set of reserved estate nodes: 

pneyron@ftoulouse:~$ oarsub -q testing -t exotic -p estats -t deploy -l nodes=3 -I
...
pneyron@ftoulouse:~$ kadeploy3 ubuntul4t200435-arm64-big -f $OAR_NODE_FILE
...

You can then connect to your node and begin your experimentation.

You are done reading this document if you do not wish to build your custom environment.

Build a customized environment

This section details how to use the from_grid5000_environment/ubuntul4t200435-arm64-custom recipe template, in order to build a kadeploy environment customized for a specific need. It uses the kameleon tool.

For more information about Kameleon, see:

Prepare the kameleon tool

First, get a job on a node for the kameleon build

The build can run either on an aarch64 or an x86_64 node, even though the target system is an aarch64 node.

If building on a x86_64 node, the qemu aarch64 emulation will be needed).

If building on estats (target system) or pyxis (other aarch64 cluster in Lyon), the build will be native, which should speed up any compilation (e.g. gcc) for instance. 

pneyron@ftoulouse:~$ oarsub -q testing -p montcalm -I
# Filtering out exotic resources (estats).
# Set walltime to default (3600 s).
OAR_JOB_ID=439704
# Interactive mode: waiting...
# Starting...
pneyron@montcalm-7:~$
Add the kameleon templates repository for the Grid'5000 environment recipes

If not already added, do as follows: 

pneyron@montcalm-7:~$ kameleon repo add grid5000 https://gitlab.inria.fr/grid5000/environments-recipes
Cloning into '/home/pneyron/.kameleon.d/repos/grid5000'...
warning: redirecting to https://gitlab.inria.fr/grid5000/environments-recipes.git/
remote: Enumerating objects: 23035, done.
remote: Counting objects: 100% (2467/2467), done.
remote: Compressing objects: 100% (434/434), done.
remote: Total 23035 (delta 2004), reused 2431 (delta 1987), pack-reused 20568
Receiving objects: 100% (23035/23035), 2.83 MiB | 21.79 MiB/s, done.
Resolving deltas: 100% (14477/14477), done.

Else, you may just update it: 

pneyron@montcalm-7:~$ kameleon repo update grid5000
...

Create the environment recipe

Create the my-ubuntu-for-estats recipe, which extends the from_grid5000_environment/ubuntul4t200435-arm64-bigcustom template
pneyron@montcalm-7:~$ mkdir -p my_kameleon_recipes && cd my_kameleon_recipes
pneyron@montcalm-7:~/my_kameleon_recipes$ kameleon new my-ubuntu-for-estats grid5000/from_grid5000_environment/ubuntul4t200435-arm64-bigcustom
      create  grid5000/from_grid5000_environment/ubuntul4t200435-arm64-bigcustom.yaml
      create  grid5000/from_grid5000_environment/base.yaml
      create  grid5000/steps/backend/docker.yaml
      create  grid5000/steps/backend/chroot.yaml
      create  grid5000/steps/aliases/defaults.yaml
      create  grid5000/steps/checkpoints/docker.yaml
      create  grid5000/steps/bootstrap/prepare_docker.yaml
      create  grid5000/steps/bootstrap/start_docker.yaml
      create  grid5000/steps/bootstrap/download_upstream_tarball.yaml
      create  grid5000/steps/bootstrap/extract_upstream_tarball_in_rootfs.yaml
      create  grid5000/steps/disable_checkpoint.yaml
      create  grid5000/steps/export/export_docker_rootfs.yaml
      create  grid5000/steps/export/export_docker_image.yaml
      create  grid5000/steps/export/export_modified_g5k_env.yaml
      create  grid5000/steps/data/helpers/kaenv-customize.py
      create  grid5000/steps/env/bashrc
      create  grid5000/steps/env/functions.sh
      create  my-ubuntu-for-estats.yaml
Adapt the recipe to your needs

Edit the generated recipe file: my-ubuntu-for-estats.yaml. Required changes are highlighted. 

#==============================================================================
#
# DESCRIPTION: Recipe to build a customized ARM64 ubuntu 20.04 L4T system
#
#==============================================================================
# This recipe extends another. To look at the step involved, run:
#   kameleon build -d my-ubuntu-for-estats.yaml
# To see the variables that you can override, use the following command:
#   kameleon info my-ubuntu-for-estats.yaml
---
extend: grid5000/from_grid5000_environment/ubuntul4t200435-arm64-bigcustom.yaml

global:
  grid5000_environment_export_name: "my-ubuntu-for-estats"
  grid5000_environment_export_author: "pneyron"
  # To install additional Ubuntu packages, just uncomment and modify the following line.
  #extra_deb_packages: "nvidia-jetpack"
  # To install some Python packages from pipy, just uncomment and modify the following line.
  #extra_pip3_packages: "jetson-stats"
  # To set the default L4T nvp model, uncomment and modify the following line.
  #l4t_nvp_model: 3

bootstrap:
  # This section should not be modified.
  - "@base"

setup:
  # This section does not need to be modified if you just want to install additional Ubuntu or pip packages or set the nvp model.
  # Just keep "@base" and set the variables above.
  - "@base"
  # If other customizations are needed, additional recipe steps can be added below this line.

export:
  # This section should not be modified
  - "@base"

Uncommenting lines 17 and 19 will install Nvidia's Jetpack (all tools including Cuda) and jetson-stats which provides the jtop command. Line 21 allows changing the on-boot power settings of the SoC. This is just an example.

It is of course advised to version the my_kameleon_recipes directory in git for instance.

Build the environment

The build requires docker. It requires also qemu-user-static if not building on the native architecture (estats or pyxis). Thus we first install them: 

pneyron@montcalm-7:~/my_kameleon_recipes$ g5k-setup-docker -t
...
pneyron@montcalm-7:~/my_kameleon_recipes$ sudo-g5k apt-get install qemu-user-static
...

We can now launch the build: 

pneyron@montcalm-7:~/my_kameleon_recipes$ kameleon build my-ubuntu-for-estats.yaml
Creating kameleon build directory : /home/pneyron/my_kameleon_recipes/build/my-ubuntu-for-estats
Starting build recipe 'my-ubuntu-for-estats.yaml'
Step 1 : bootstrap/_init_bootstrap/_init_0_pull_and_tag_base_image
--> Running the step...
Starting command: "bash"
[local] The local_context has been initialized
[local] docker is /usr/bin/docker
Step 2 : bootstrap/_init_bootstrap/_init_1_start_docker_container
--> Running the step...
[local] docker is /usr/bin/docker
Step 3 : bootstrap/_init_bootstrap/_init_2_start_docker_container
--> Running the step...
[local] qemu-aarch64-static is /usr/bin/qemu-aarch64-static
Step 4 : bootstrap/_init_bootstrap/_init_3_disable_checkpoint
--> Running the step...
Step _init_bootstrap took: 0 secs
Step 5 : bootstrap/prepare_docker/pull_and_tag_base_image
--> Running the step...
[local] Importing docker base image: ubuntu:20.04, which the recipe builds from.
[local] 20.04: Pulling from library/ubuntu
[local] 96d54c3075c9: Pulling fs layer
[local] 96d54c3075c9: Verifying Checksum
[local] 96d54c3075c9: Download complete
[local] 96d54c3075c9: Pull complete
[local] Digest: sha256:ed4a42283d9943135ed87d4ee34e542f7f5ad9ecf2f244870e23122f703f91c2
[local] Status: Downloaded newer image for ubuntu:20.04
[local] docker.io/library/ubuntu:20.04
Step prepare_docker took: 6 secs
Step 6 : bootstrap/start_docker/start_docker_container
--> Running the step...
[local] 79fa4922f99f17567420d6abcc845602ccf22402115731150f1aad2b5d76e330
Step start_docker took: 0 secs
Step 7 : bootstrap/download_upstream_tarball/download
--> Running the step...
[local] Retrieve image from Grid5000 environment 'ubuntu2004-nfsl4t'
[local] Image is server side, try and fetch it from local file /grid5000/l4t/ubuntul4t200435-arm64-big-snapshot-pneyron.tar.zst
[local] '/grid5000/l4t/ubuntul4t200435-arm64-big-snapshot-pneyron.tar.zst' -> '/home/pneyron/my_kameleon_recipes/build/my-ubuntu-for-estats/upstream_build/upstream.tar'
Step download_upstream_tarball took: 12 secs
Step 8 : bootstrap/extract_upstream_tarball_in_rootfs/clean_rootfs
--> Running the step...
Starting command: "test -s MAIN_CONTAINER_ID && LC_ALL=POSIX docker exec -uroot -i $(< MAIN_CONTAINER_ID) /bin/bash"
[out] The out_context has been initialized
Step 9 : bootstrap/extract_upstream_tarball_in_rootfs/extract_image_in_rootfs
--> Running the step...
[local] Upstream tarball: /home/pneyron/my_kameleon_recipes/build/my-ubuntu-for-estats/upstream_build/upstream.tar
Starting command: "test -s MAIN_CONTAINER_ID && LC_ALL=POSIX docker exec -uroot -i $(< MAIN_CONTAINER_ID) /bin/bash"
Step extract_upstream_tarball_in_rootfs took: 67 secs

...

Step 26 : export/export_modified_g5k_env/export_files
--> Running the step...
[local] Copying grid5000 environment files to /home/pneyron/public/
[local] '/home/pneyron/my_kameleon_recipes/build/my-ubuntu-for-estats/my-ubuntu-for-estats.tar.zst' -> '/home/pneyron/public/my-ubuntu-for-estats.tar.zst'
[local] 'my-ubuntu-for-estats.dsc' -> '/home/pneyron/public/my-ubuntu-for-estats.dsc'
Step export_modified_g5k_env took: 10 secs
Step 27 : export/_clean_export/_clean_0_clean_containers
--> Running the step...
Step _clean_export took: 3 secs

Successfully built 'my-ubuntu-for-estats.yaml'
Total duration : 225 secs

The environment is now ready to be deployed on estats nodes!

  • environment description file: $HOME/public/my-ubuntu-for-estats.dsc
  • environment image: $HOME/public/my-ubuntu-for-estats.tar.zst

If we are done with the environment creation. We can quit the current job and create a new one of type deploy for the deployment (bare-metal) of the environment on some estats nodes.

Deploy the environment

Reserve an estats node for 2h in a deploy job

(The estats cluster is currently in the testing queue, and is flagged as "exotic".) 

pneyron@ftoulouse:~$ oarsub -I -t deploy -t exotic -q testing -p estats -l nodes=1,walltime=2
# Include exotic resources in the set of reservable resources (this does NOT exclude non-exotic resources).
OAR_JOB_ID=1125125
# Interactive mode: waiting...
# Starting...
[OAR] OAR_JOB_ID=1125125
[OAR] Your nodes are:
      estats-4.toulouse.grid5000.fr*64

pneyron@ftoulouse:~$
Open the console of the node

This is optional but allows to follow what's going on. 

pneyron@ftoulouse:~$ kaconsole3 estats-4

No action is required in the console (any action during the boot/deploy process may jeopardize the deployment).

Deploy the environment on the node
pneyron@ftoulouse:~$ kadeploy3 -a ~/public/my-ubuntu-for-estats.dsc -m estats-4
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 started
Grab the key file ...
Grab the tarball file ...
Grab the postinstall file ...
Launching a deployment on estats-4.toulouse.grid5000.fr
...
...
...
End of step Deploy[BootNewEnvHardReboot] after 196s
End of deployment for estats-4.toulouse.grid5000.fr after 526s
End of deployment on cluster estats after 526s
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 done

The deployment is successful on nodes
estats-4.toulouse.grid5000.fr

We can now ssh to the estats-4 node.

Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Ubuntu-ARM64-L4T_environment&oldid=106502"