Deploy environment-OAR1

From Grid5000
Jump to: navigation, search
Warning.png Warning

This practice was designed for the old OAR1 infrastructure of Grid'5000. Now Grid'5000 uses OAR2, please refer to the corresponding practice.

Contents

Connect to Grid'5000

  • you should connect to the frontend node of the cluster you were granted access to:
Terminal.png outside:
ssh login@acces.site.grid5000.fr
  • if needed, you may then internally jump onto the frontend of another cluster following your deployment purposes:
Terminal.png frontend:
ssh oar.another_site.grid5000.fr

Find and deploy existing images

Where you will learn to locate environments, find deployable partitions, make a reservation on a deployable node and use kadeploy...

Locate a suitable image

Each site maintains an environment library in the /grid5000/images directory of the oar node. Image descriptions should be made on pages like:

You should also be able to browse through all other available environments:

To deploy an environment, you must know its name as registered in the Kadeploy database. It is the first information on the environment description page. This tutorial uses the debian4all environment.

Being able to reproduce the experiments that are done is a desirable feature. Therefore, you should always try to control as much as possible the environment the experiment is done in. Therefore, we will attempt to check that the environment that was chosen in the environement directory is the one available on a given cluster. On the cluster you would like to deploy, type:

Terminal.png frontend:
kaenvironments --environment debian4all

Then, check that the filebase file is the expected one by checking its name and its checksum:

Terminal.png frontend:
md5sum /grid5000/images/image_debian4all-sid-amd64.tgz

In theory, you should also check the post-install script. A post-install script adapts an environment to the site it is deployed on. In the same way as for environments, you should be able to find a description of the post-install script on pages such as this one. Post-install scripts is an evolving matter, so don't be too worried if you don't find things as described here. If everything seems ok, please proceed to the next step.

Make a reservation on a deployable node

For this part of the tutorial, reservation made will be interactive (-I), using deploy queue (-q deploy), on only one machine (-l nodes=1) to do environment customization (we will give ourselves 4 hours with -l walltime=4), which gives us the following command, that will open a new shell session on the frontend node:

Terminal.png frontend:
oarsub -I -q deploy -l nodes=1 -l walltime=4

Indeed, when you reserve using deploy queue, a new shell is opened on the frontend node and not on the first machine of the reservation as for standard reservations. When you leave this shell, the reservation will end. The shell is populated with OAR_* environment variables. You should look at the list of available variables to get an idea of the information you can use to script deployment later. As usual, if the reservation is successfull, you will get the reserved machine name with:

Terminal.png frontend:
cat $OAR_FILE_NODES

Deploy an environment

You will first need to select the partition you will deploy on. To this end, each site describes the partition scheme it uses in its "message of the day", that is displayed upon succesful login. If you forgot to read that message of the day, you can show it again with:

Terminal.png frontend:
cat /etc/motd

You should find there the list of free partitions you can use. You should select one, in general hda3 or sda3, according to the most recent universal partition scheme that has been adopted on Grid'5000. Then, to deploy your environment, you must discover the nodes you were allocated by OAR. The simplest way of doing this is to look at the contents of the file whose name is stored in $OAR_FILE_NODES or the messages displayed when the reservation was made. Deployment happens when you run the following command:

Terminal.png frontend:
kadeploy -e debian4all -m node.site.grid5000.fr -p partition

You can automate this discovery like this:

Terminal.png frontend:
kadeploy -e debian4all -m `head -1 $OAR_FILE_NODES` -p partition

Once kadeploy has run successfully, the allocated node is deployed under "debian4all" environment. It will then be possible to tune this environment according to our needs.

Tune an environment to build another one: customize authentification parameters

Where you learn to connect to a deployed environment, make a simple change and create a derived environment (karecordenv)... Here, we could have to use kaconsole to connect to the deployed machine.

Connect to the deployed environment and customize it

Connection, the usual way

You can now connect using the user/password given in the environment description page. In general, you can try root/grid5000 before looking that information up.

On most environments, there is at least one configured account for login through ssh (kadeploy checks that sshd is running before declaring success of a deployment). We will therefore connect using standard tools:

Terminal.png frontend:
ssh root@node.site.grid5000.fr
Note.png Note

default root password for user environments is often "grid5000". Please check the environment description...

In case this doesn't work, please take a look at:

Connection, the brutal way

When playing with environments, you could loose services such as sshd or rshd and then become unable to get a shell on the deployed machine. It is therefore possible to connect to the machine's console (for the OS, it is seen as physical access, thanks to remote console capabilities). It should be possible using the kaconsole command, provided nobody else has already opened a write access to the remote console and that the post-install script has remote console properly configured:

Terminal.png frontend:
kaconsole -m node.site.grid5000.fr

You should be shown the login screen as if you were locally connected onto the machine. Remote console should be kept working all allong the boot process, which is useful to debug an environment (see also expert-level practices). To get out of this remote console without having to kill your shell or terminal window, you should type in the proper escape sequence:

Customization

For automated deployment of experiments, the easiest is to enable password-less connections to the machines. Therefore, we will simply create an environment where our ssh key is in the authorized_keys file of the root account. Beware: post-install scripts sometimes change this file. In this case, you can use authorized_keys2. Therefore:

  1. connect as root to the machine where the environment is deployed;
  2. edit /root/.ssh/authorized_keys2, and add to it a public sshkey with no passphrase;
    • because oar submission does not transfer ssh authentication through agent-forwarding, you need a key with no passphrase!
    • you can therefore generate a key for this tutorial and pass it as an additional parameter to the scripts we will be building further along...
  3. check that connection is now possible from your user account on the frontend node, with no user interaction.

Using the root account for all your experiments is possible, but you will probably be better off creating a user account. You could even create user accounts for all the users that would be using your environment. However, if the number of users is greater than 2 or 3, you should better configure an LDAP client by tuning the post-install script or using a fat version of this script (beyond the scope of this tutorial). Two ways of doing things.

The simplest is to create a dedicated account (e.g. the generic user g5k) and move in all experiment data at the beginning and back at the end of an experiment, using scp or rsync.

A more elaborate approach is to locally recreate our usual Grid'5000 account with the same uid/gid on our deployed environment. This second approach could simplify file rights management if you need to store temporary data on shared volumes:

To create your local unix group on your environment, first find your primary group on the frontend node with:

Terminal.png frontend:
id

The output of this command is for instance of the form:

uid=19002(dmargery) gid=19000(rennes) groups=9998(CT),9999(grid5000),19000(rennes)

Then, as root, on the deployed machine:

Terminal.png node:
addgroup --gid groupId groupName

Now, to enable access to your local user account on your environment, as root, on the deployed machine:

Terminal.png node:
adduser --uid userId --ingroup groupName userLogin

Finally, as root, become the newly created user and place your ssh key:

Terminal.png node:
su - userLogin
mkdir ~/.ssh
cp /root/.ssh/authorized_keys2 ~/.ssh/

Create a new environment from a customized environment

We now need to save this customized environment, where you have a user account, to be able to use this account again each time you deploy it.

The first step to create an environment is to create an archive of the node you just customized. Because of the various implementations of the /dev filesystem tree, this can be a more or less complex operation. The state-of-the-art command for archive creation can be found on the wiki:

If /dev is mounted using "udev" (it's probably the case if you have a /dev/.udevdb directory), you should follow directions from:

Warning.png Warning

Kadeploy insists archives be named with a .tgz extension and not .tar.gz. Moreover, the system user deploy should have "read access" to your archive file (this hence also means "execute access" on each subdirectories from your homedir to that file) if you want to deploy an environment based on this archive.

Record this environment for future deployments

This is done using the karecordenv command. It allows to create a pointer in the Kadeploy database to your archive and associate to it a name, that can be recalled by the kadeploy command, as well as a post-install archive to finely adjust some system files and make the system bootable on each different deployed node. An environment thus consists of this association between an image archive, a name and a post-install archive, along with some other minor parameters.

To find all the parameters of this command, one method is to read the output of karecordenv -h.

Usage : karecordenv
        -n  | --name             environment name
        -v  | --version          version              (default is 1)
        -d  | --description      description
        -a  | --author           author's email
        -fb | --filebase         environment image path
        -ft | --filesite         post-installation file path
        -s  | --size             size (Mo)
        -i  | --initrdpath       initrd path
        -k  | --kernelpath       kernel path
        -p  | --param            kernel parameters
        -fd | --fdisktype        fdisk type           (default is 83)
        -fs | --filesystem       file system          (default is ext2)
        -o  | --optsupport       optimisation support (default is 0)
        -fe | --file_environment
        -h                       get this help message

A typical command line could hence be:

Terminal.png frontend:
karecordenv -n "debian4all-myLoginName" \
-v 1 \
-d "customized debian4all by myFullName" \
-a myEmailAddress \
-fb file:///home/mySite/myLoginName/myPath/myImage.tgz \
-ft file://usr/local/kadeploy/Scripts/postinstall/postinstall4all_gen.tgz \
-i /boot/initrd.img-2.6.13.5 \
-k /boot/vmlinuz-2.6.13.5 \
-fd 83 \
-fs ext3

Here, the -i and -k parameters are pointers to system files within your environment needed to build the correct boot loader (Kadeploy uses "grub"). The files in the preceding example correspond to the default kernel installed in the debian4all environment, which are still valid in our customized environment (unless we had recompiled another kernel...).

When customizing an already existing environment, it could be easier to restart from the original record, since karecordenv also accepts a file as a single parameter, following the output format of kaenvironments -e. For instance:

Terminal.png frontend:
kaenvironments -e debian4all > myImage.env

You may then edit the content of the myImage.env file:

#debian4all v2

name = debian4all
id = 153
version = 2
description = Environnement minimal debian
author = xavier.delaruelle@loria.fr
filebase = file://grid5000/images/image_debian4all-sid-amd64.tgz
filesite = file://usr/local/kadeploy/Scripts/postinstall/postinstall4all_gen.tgz
size = 1000
initrdpath = /boot/initrd.img-2.6.13.5
kernelpath = /boot/vmlinuz-2.6.13.5
kernelparam = 
fdisktype = 83
filesystem = ext3
siteid = 1
optsupport = 0
user = deploy

Just:

  1. update the name, description, author and filebase lines
  2. delete the id, size, version and user lines
  3. save
  4. and then record your environment with:
Terminal.png frontend:
karecordenv -fe myImage.env


Adding software to an environment

Where you learn to install software using the package repository of your distribution on Grid'5000 (using proxys)...

In theory, Grid'5000 machines can not access the outside public internet. But for practical reasons, some holes are poked to be able for instance to update an environment with the latest packages from the distributor or to add usefull software (in .deb or .rpm forms). One of these holes is the apt-cacher available on most the sites at apt.site.grid5000.fr (Nancy, Orsay or Rennes for example). Therefore, on the debian4all environment, you can find the following lines in the /etc/apt/sources.list file:

deb http://apt.orsay.grid5000.fr/apt-cacher/ftp.de.debian.org/debian/ sid main

These lines enable the apt-get commands to work as expected. You can therefore update your environment (to add missing libraries that you need, or remove packages that you don't so that sizes down the image and speeds up the deployment process, etc.) using:

Terminal.png node:
apt-get update
apt-get upgrade
apt-get install list of desired packages and libraries
apt-get --purge remove list of unwanted packages
apt-get clean

Scripting a deployment

To plan an experiment, you need to script most of the work to avoid too many interactive steps. The main points to note are the following:

  • a non-interactive reservation on queue deploy will see the given script running on the node from which deployment are run (usually the oar or frontend node);
  • kadeploy accepts the -f parameter to read the list of machines to deploy on from a file;
  • in non-interactive mode, scripts started by OAR write their output in file whose names are OAR.scriptName.jobId.sdterr and OAR.scriptName.jobId.sdtout.

Using this additionnal information, we will now write a script to be run by OAR to deploy machines before using them for an experiment. You can use the following script as a base. The latest version of this script is available in the subversion repository of the Grid'5000 on InriaGforge. To get a copy of this script, as well as other shared material:

Terminal.png outside:
#!/bin/sh
# written by David Margery for the 2006 Grid'5000 spring school
# feel free to copy and adapt 

echo -n "Script running on : "
hostname 

#$OAR_FILE_NODES file may contain some nodes more than once
#We will create a second file with each node only once 

OAR_FILE_NODES_UNIQUE=${OAR_FILE_NODES}-unique
cat $OAR_FILE_NODES | sort -u > $OAR_FILE_NODES_UNIQUE 

#deploy our test environnement
kadeploy -e ubuntu4all-`whoami` -f $OAR_FILE_NODES_UNIQUE -p sda2

#do stuff with this deployed environment
for i in `cat $OAR_FILE_NODES | sort -u`
do
 echo attempt to get information from $i
 ssh root@$i -o StrictHostKeyChecking=no cat /etc/hostname
 ssh root@$i -o StrictHostKeyChecking=no uname -a
done
  1. save your version of this script in your home directory (/home/mySite/myLogin/myPath/auto_deploy.sh);
  2. check the name of the environment it will attempt to deploy;
  3. and submit a job to OAR:
Terminal.png frontend:
oarsub -l nodes=2 -q deploy /home/mySite/myLogin/myPath/auto_deploy.sh

To follow progress of your job:

Terminal.png frontend:
oarstat -j jobId

or

Terminal.png frontend:
tail -f OAR.auto_deploy.sh.jobId.stdout

To see the associated error log:

Terminal.png frontend:
less OAR.auto_deploy.sh.jobId.stderr
Personal tools
Namespaces

Variants
Actions
Public Portal
Users Portal
Admin portal
Wiki special pages
Toolbox