PMEM-environment

Generating an environment with the PMEM tools

We will build a PMEM environment thanks to the kameleon tool.

Kameleon is one of the tools to build Grid'5000 environments, along with tgz-g5k. Contrary to tgz-g5k, it allows to script the environment build process, and is lighter (build in a VM).

Kameleon is provided as a ruby gem. One can install and use it without requiring root privileges.

Kameleon's documentation:

• http://kameleon.imag.fr
• https://www.grid5000.fr/w/Environments_creation_using_Kameleon_and_Puppet

Prepare the kameleon environment

Install kameleon

pneyron@fgrenoble:~$ gem install --user kameleon-builder
Fetching: progressbar-1.10.1.gem (100%)
Successfully installed progressbar-1.10.1
Fetching: kameleon-builder-2.10.4.gem (100%)
Successfully installed kameleon-builder-2.10.4
Parsing documentation for progressbar-1.10.1
Installing ri documentation for progressbar-1.10.1
Parsing documentation for kameleon-builder-2.10.4
Installing ri documentation for kameleon-builder-2.10.4
Done installing documentation for progressbar, kameleon-builder after 1 seconds
2 gems installed

Make sure to have the latest version of kameleon-builder: 2.10.4 as of writing this.

Make sure to have the kameleon command in your path

pneyron@fgrenoble:~$ which kameleon
/home/pneyron/.gem/ruby/2.5.0/bin/kameleon

Add the Grid'5000 environment recipes to your kameleon repositories

pneyron@fgrenoble:~$ kameleon repo add grid5000 https://gitlab.inria.fr/grid5000/environments-recipes
Cloning into '/home/pneyron/.kameleon.d/repos/grid5000'...
warning: redirecting to https://gitlab.inria.fr/grid5000/environments-recipes.git/
remote: Enumerating objects: 144, done.
remote: Counting objects: 100% (144/144), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 15154 (delta 67), reused 110 (delta 44), pack-reused 15010
Receiving objects: 100% (15154/15154), 1.90 MiB | 14.11 MiB/s, done.
Resolving deltas: 100% (9391/9391), done.

Create a customized environment recipe

We create a debiantesting-x64-pmem recipe, which will build on top of the debiantesting-x64-min environment, which is maintained by the Grid'5000 tech team.

Using Debian testing and not Debian 10 is important, since the support of PMEM in Debian 10 lacks the latest functionalities provided by more recent Linux kernels (e.g. kmem).

Instead of extending the debiantesting-x64-min recipe and thus building our environment from scratch (which means running the debian installer, etc), we choose to use the from_grid5000_environment/debian10-x64-custom recipe, which will let kameleon extract the already builded debiantesting-x64-min environment tarball and add our stuff on top of it. As a reminder, kameleon does all that in a VM.

Create the debiantesting-x64-pmem recipe, which extends from_grid5000_environment/debian10-x64-custom

pneyron@fgrenoble:~$ mkdir debiantesting-x64-pmem
pneyron@fgrenoble:~$ cd debiantesting-x64-pmem
pneyron@fgrenoble:~/debiantesting-x64-pmem$ kameleon new debiantesting-x64-pmem grid5000/from_grid5000_environment/debian10-x64-custom
      create  grid5000/from_grid5000_environment/debian10-x64-custom.yaml
      create  grid5000/steps/backend/qemu.yaml
      create  grid5000/steps/backend/VM.yaml
      create  grid5000/steps/aliases/defaults.yaml
      create  grid5000/steps/checkpoints/qemu.yaml
      create  grid5000/steps/bootstrap/prepare_ssh_to_out_context.yaml
      create  grid5000/steps/bootstrap/download_upstream_tarball.yaml
      create  grid5000/steps/bootstrap/create_appliance.yaml
      create  grid5000/steps/bootstrap/prepare_appliance.yaml
      create  grid5000/steps/bootstrap/start_qemu.yaml
      create  grid5000/steps/disable_checkpoint.yaml
      create  grid5000/steps/export/save_appliance_VM.yaml
      create  grid5000/steps/export/create_kadeploy_environment.yaml
      create  grid5000/steps/data/helpers/create_appliance.py
      create  grid5000/steps/data/qemu-sendkeys.rb
      create  grid5000/steps/data/helpers/export_appliance.py
      create  grid5000/steps/data/helpers/kaenv-customize.py
      create  grid5000/steps/env/bashrc
      create  grid5000/steps/env/functions.sh
      create  debiantesting-x64-pmem.yaml

Adapt the debiantesting-x64-pmem recipe to add our stuff

Edit the generated recipe file: debiantesting-x64-pmem.yaml, to make Grid'5000 LDAP and home directories service available, add the PMEM tools and setup an additional postinstall that will switch the PMEM to AppDirect during the deployment (PMEM is in Memory mode by default).

Required changes are high-lighted.

#==============================================================================
# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
#==============================================================================
#
# DESCRIPTION: debiantesting-x64-pmem
#
#==============================================================================
# This recipe extends another. To look at the step involed, run:
#   kameleon dryrun debiantesting-x64-pmem.yaml
# To see the variables that you can override, use the following command:
#   kameleon info debiantesting-x64-pmem.yaml
---
extend: grid5000/from_grid5000_environment/debian10-x64-custom.yaml

global:
  ### Grid'5000 environment information
  ## (Uncomment and change any line if needed)

  ## Frontend to run kaenv3 on
  #grid5000_frontend: "frontend"

  ## Site used in the build
  #grid5000_site: "grenoble"

  ## Environment to build from
  grid5000_environment_import_name: debiantesting-x64-min
  #grid5000_environment_import_user: "deploy"
  #grid5000_environment_import_version: ""

  ## New environment description
  #grid5000_environment_export_name: "$${kameleon_recipe_name}"
  #grid5000_environment_export_format: "tar.gz"
  grid5000_environment_export_description: "debian testing with pmem tools"

  ## Set where to store the environment and the assiated kadeploy URL base
  #grid5000_environment_export_dir: "$HOME/public/"
  #grid5000_environment_export_baseurl: "local://$HOME/public/"

  ## Optionaly, the environment postinstall script can be changed, e.g. to
  ## enable NFS homes, LDAP accounts, if not enabled in the imported env.
  grid5000_environment_export_postinstall_script: "g5k-postinstall --net debian --fstab nfs --restrict-user current"

  ## Optionaly, an additional postinstall can be given, e.g. to do some custom
  ## operations. Use the following variables to set the archive name and script.
  grid5000_environment_export_additional_postinstall_archive: "$${kameleon_recipe_name}-postinstall.tar.gz"
  grid5000_environment_export_additional_postinstall_script: "pmem-setup.sh"
  ## The recipe will have to prepare the additional postinstall content in a
  ## directory to create in the local context and name "additional_postinstall"
  ## by default (next line to change it). The archive is created by the export.
  #grid5000_environment_export_additional_postinstall_dir: "additional_postinstall"

  ### You can add below any other global variable definition
  ## See the variables which can be overloaded, by running:
  ##   kameleon info debiantesting-x64-pmem.yaml
  ## Or define any new variable you would need. e.g.:
  #my_variable: my_value

bootstrap:
  ### The bootstrap section takes in charge the import of the Grid'5000
  ## environment to customize. No modification should be needed here.
  - "@base"

setup:
  # We first need to setup Grid'5000 LDAP and NFS, since this is not done in
  # the debiantesting-x64-min environment, nor by g5k-postinstall (See
  # https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=11839).
  - prepare_user_access:
    - update-system:
      - exec_in: apt-get -y update && apt-get -y upgrade
    - install_nfs_service:
      - exec_in: apt-get install -y nfs-common autofs
    - install_ldap_service:
      - exec_in: apt-get install -y nslcd
    - setup_ldap:
      - exec_in: mkdir -p /etc/ldap/certificates/
      - local2in:
        - /etc/ldap/certificates/ca2019.grid5000.fr.cert
        - /etc/ldap/certificates/ca2019.grid5000.fr.cert
      - local2in:
        - /etc/ldap/ldap.conf 
        - /etc/ldap/ldap.conf 
      - local2in:
        - /etc/nsswitch.conf
        - /etc/nsswitch.conf
      - local2in:
        - /etc/nslcd.conf
        - /etc/nslcd.conf
  # We install the PMEM tools and some other utilities in the environement
  - install_tools:
    - install_pmem_tools:
      - exec_in: apt-get install -y ipmctl ndctl daxctl
    - install_common_tools:
      - exec_in: apt-get install -y sudo vim-nox hwloc-nox htop dstat
  # Last, we add a postinstall which will configure the PMEM is AppDirect Mode
  # during the deployment, so that the deployed nodes are directly in AppDirect
  # mode, avoiding the need to do the switch and reboot again after deploying.
  - prepare_additional_postinstall:
    - create_pmem_setup_script:
      - exec_local: |
          rm -rf $${grid5000_environment_export_additional_postinstall_dir}
          mkdir $${grid5000_environment_export_additional_postinstall_dir}
          cat <<'EOF' > $${grid5000_environment_export_additional_postinstall_dir}/$${grid5000_environment_export_additional_postinstall_script}
          #!/bin/bash
          # Make stdout&err display in the console as well as in kadeploy logs.
          exec 1> >(tee -a /dev/console)
          exec 2> >(tee -a /dev/console 1>&2)
          set -x -e
          # Get the current memory size, if > 400MB, that surely means that PMEM
          # is in Memory Mode (troll nodes have 384GB RAM).
          declare -a MEMINFO
          MEMINFO=($(head -n1 /proc/meminfo))
          if [ ${MEMINFO[1]} -gt 400000000 ]; then
            echo "Switch PMEM to AppDirect Mode..."
            # The deploy kernel runs Debian buster but ipmctl is only available
            # in buster-backports, thus we add the buster-backports source.
            echo "deb http://deb.debian.org/debian/ buster-backports main contrib non-free" > /etc/apt/sources.list.d/buster-backports.list
            apt-get update
            apt-get install -y ipmctl
            # Switch PMEM to AppDirect, after this the node need to reboot for 
            # the switch to actually happen. For that purpose, the kadeploy3
            # command must be called with the --no-kexec option.
            ipmctl create -force -goal MemoryMode=0
          else
            echo "PMEM already in AppDirect Mode"
          fi
          EOF
          chmod 755 $${grid5000_environment_export_additional_postinstall_dir}/$${grid5000_environment_export_additional_postinstall_script}
export:
  ### The export section takes in charge the export of your customized Grid'5000
  ## environment. No modification should be needed here.
  - "@base"

One can easily add more in the environment, for instance installing more software in the install_tools step, or adding any other steps in the setup section.

Build the environment

To build the environment, we use a Grid'5000 node

pneyron@fgrenoble:~/debiantesting-x64-pmem$ oarsub -I
[ADMISSION RULE] Set default walltime to 1 hour.
[ADMISSION RULE] Modify resource description with type constraints
[ADMISSION_RULE] Resources properties : \{'property' => 'type = \'default\'','resources' => [{'resource' => 'host','value' => '1'}]}
[ADMISSION RULE] Job properties : maintenance = 'NO'
Import job key from file: /home/pneyron/.ssh/id_rsa
OAR_JOB_ID=1931031
Interactive mode: waiting...
Starting...
Connect to OAR job 1931031 via the node dahu-20.grenoble.grid5000.fr
[OAR] OAR_JOB_ID=1931031
[OAR] Your nodes are:
      dahu-20.grenoble.grid5000.fr*32

[pneyron@dahu-20 debiantesting-x64-pmem](1931031-->59mn)$

In the job shell, we run the build

[pneyron@dahu-20 debiantesting-x64-pmem](1931031-->58mn)$ kameleon build debiantesting-x64-pmem.yaml
Creating kameleon build directory : /home/pneyron/debiantesting-x64-pmem/build/debiantesting-x64-pmem
Starting build recipe 'debiantesting-x64-pmem.yaml'
Step 1 : bootstrap/_init_bootstrap/_init_0_create_appliance
...
...
...
Step 29 : export/create_kadeploy_environment/create_kaenv_file
--> Running the step...
Step 30 : export/create_kadeploy_environment/copy_files
--> Running the step...
[local] Copying grid5000 environment files to /home/pneyron/public/
[local] '/home/pneyron/debiantesting-x64-pmem/build/debiantesting-x64-pmem/debiantesting-x64-pmem.tar.gz' -> '/home/pneyron/public/debiantesting-x64-pmem.tar.gz'
[local] 'debiantesting-x64-pmem.yaml' -> '/home/pneyron/public/debiantesting-x64-pmem.yaml'
Step create_kadeploy_environment took: 2 secs
Step 31 : export/_clean_export/_clean_1_start_vm
--> Running the step...
Step 32 : export/_clean_export/_clean_0_delete_initial_image_at_the_end
--> Running the step...
Step _clean_export took: 0 secs

Successfully built 'debiantesting-x64-pmem.yaml'
Total duration : 156 secs

Our environment is now ready to be deployed on a troll node !

We quit this job, we will create a new one on troll nodes and of type deploy.

Deploy the environment

Reserve a troll node for 2h in a deploy job

pneyron@fgrenoble:~/debiantesting-x64-pmem$ oarsub -p "cluster='troll'" -t deploy -l nodes=1,walltime=2 -I
[ADMISSION RULE] Modify resource description with type constraints
[ADMISSION_RULE] Resources properties : \{'property' => 'type = \'default\'','resources' => [{'resource' => 'host','value' => '1'}]}
[ADMISSION RULE] Job properties : ((cluster='dahu') AND deploy = 'YES') AND maintenance = 'NO'
Import job key from file: /home/pneyron/.ssh/id_rsa
OAR_JOB_ID=1931038
Interactive mode: waiting...
Starting...
Connect to OAR job 1931038 via the node frontend
[OAR] OAR_JOB_ID=1931038
[OAR] Your nodes are:
      troll-2.grenoble.grid5000.fr*32

[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->119mn)$

Open the console of the node

Since the deployment is pretty long on the troll nodes, we open the console help our patience...

pneyron@fgrenoble:~$ kaconsole3 -m troll-2

Deploy the environment on the node

Run the deployment with the --no-kexec option, so that the real reboot makes the PMEM config take effect during the deployment.

[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->119mn)$ kadeploy3 --no-kexec -a ~/public/debiantesting-x64-pmem.yaml -f $OAR_NODE_FILE -k
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 started
Grab the key file /home/pneyron/.ssh/authorized_keys
Grab the tarball file /home/pneyron/public/debiantesting-x64-pmem.tar.gz
Grab the postinstall file server:///grid5000/postinstalls/g5k-postinstall.tgz
Launching a deployment on troll-2.grenoble.grid5000.fr
...
...
...
End of step Deploy[BootNewEnvHardReboot] after 196s
End of deployment for troll-2.grenoble.grid5000.fr after 526s
End of deployment on cluster troll after 526s
Deployment #D-ea5e7cb0-da6a-4b46-8726-f9f41ee2c115 done

The deployment is successful on nodes
troll-2.grenoble.grid5000.fr
[pneyron@fgrenoble debiantesting-x64-pmem](1931038-->109mn)$

Well done, we can now ssh to the machine with our user account or root. The PMEM is in AppDirect mode, we can use the ipmctl, ndctl or daxctl tools to configure it and run any experiments.