Deploying the Kerrighed SSI Operating System on the Grid'5000 platform
Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs.
The goals of Kerrighed are high performance of applications, high availability of the cluster, efficient resources management, high customizability of the operating system and ease of use.
Kerrighed is implemented as an extension to Linux operating system (a set of Linux modules and a small patch to the kernel).
kadeploy issues with Kerrighed
When booting a Kerrighed kernel, you must provide at boot time some parameters which identify the node and the cluster. These are:
- node_id : a unique id on the Kerrighed cluster
- session_id : a id shared by nodes owned by the same Kerrighed cluster (refer to session_id allocation table further along)
- nb_nodes : the number of nodes in the Kerrighed cluster
A modified version of kadeploy, available in Rennes, allows to provide these parameters to the commands kadeploy and kareboot:
- as a string with the option --bootparam
- as a file containing <machine>=<bootparams> lines with the option --bootparams-file
This modified version of kadeploy is available in: /opt/kadeploy-2.1.2-krg/usr/bin on parasol-dev
Deploying an existing Kerrighed environment
To use Kerrighed on Grid'5000, you should deploy a Kerrighed environment. You have to reserve some nodes with oarsub or any other reservation mechanism as long as you get deploiement rights (-q deploy with oar). Here you can find a page about OAR. The nodes must be on the same cluster.
ubuntu4all-5.10.i386.krg-636, built from revision 636 of the repository. This is a public version of Kerrighed.
kerrighed-i386 This is an old version of a Kerrighed environment.
To build your own environment for a specific version of Kerrighed, please refer to the next session.
The script /site/data0/kerrighed/bin/krg_make_cluster.sh takes as parameters:
- -p <partition> : the partition where to deploy the environment, as for kadeploy
- -s <int> : the number of the Kerrighed session, shared only by the nodes of the same Kerrighed cluster. This enables two different people to work on a subset of nodes without interfering with one another
- -e environment_name : the name of the environment with Kerrighed
- -l <username> : optional : the user that should be used by kadeploy to find environment_name
- -n <node1>,<node2>,etc... : optional : comma-separated list of nodes where to deploy the cluster. All the nodes of your reservation will be used if not specified
- -q : do not ask for confirmation
The last script is just a wrapper around kadeploy. Then, you can setup the cluster with kadeploy. Don't forget to use the modified version of kadeploy in /opt/kadeploy-2.1.2-krg/usr/bin.
- Write a file where each line contains the boot parameters for each node. For example (don't forget to put the FQDN for each node, ie node.rennes.grid5000.fr, not node):
node1.site.grid5000.fr = node_id=0 session_id=1 nb_nodes=3
node2.site.grid5000.fr = node_id=2 session_id=1 nb_nodes=3
node3.site.grid5000.fr = node_id=3 session_id=1 nb_nodes=3
- Write a file where each line contains the FQDN of the nodes of the cluster, for example:
- Launch the command kadeploy with the right option, for example:
kadeploy -f nodes_file -e kerrighed-i386 -r params_file -p sda3
Updating a Kerrighed environment to the last (svn) version
(For Kerrighed building, please have a look at 
- Deploy the environment kerrighed-i386-update on one node. It contains the same filesystem as kerrighed-i386 but boots on an a non-kerrighed kernel (for stability reasons), with SMP support (Kerrighed is not yet SMP-aware and you are going to compile a new kernel).
- Once the environment is deployed, log on the node with user root, password is the usual grid5000 root password.
- Get the sources (replace développeur with your Kerrighed account name):
svn checkout svn+ssh://développeur@scm.gforge.inria.fr/svn/kerrighed/trunk kerrighed
- Set some variables containing the path to the parts of Kerrighed:
Updated Instructions (David Margery)
The general idea here is to build Kerrighed and the kernel on a node using a system as close as possible to the system Kerrighed will be running. Therefore, we will use an image derived from ubuntu4all-5.10.i386, ubuntu4all-5.10.i386-4krg. This image has the following properties
/root/.configis a correct config file for the Kerrighed on the parasol nodes. You would need to check it on other architectures
/etc/mkinitrd/modulescontains most of the modules needed to build an initrd that will work on most Grid'5000 nodes (I don't keep it up-to-date, so you might have surprises)
/tmp/imagefor quick image creation.
/site/data0are mounted because declared in
/etc/fstab. This only applies if working on machines located in Rennes.
/etc/nsswitch.confhas been modified to use ldap. (Here again, ldap configuration will probably only correct for Rennes).
/tmp/image/devcontains the two special files needed by
- mknod -m 660 console c 5 1
- mknod -m 660 null c 1 3
- automake1.9 pour pouvoir compiler Kerrighed
- As a bonus, root's history file contains the commands to make an image (.tgz) of the current node
Therefore, to build an image with the latest version of Kerrighed, the suggested approach is
- from the head node, download the latest version of kerrighed to your home dir or to scratch.
- kadeploy ubuntu4all-5.10.i386-4krg on a node
- cd /site/data0/$USER/
- svn checkout svn+ssh://$USER@scm.gforge.inria.fr/svn/kerrighed/trunk kerrighed
- from the deployed node, as an user
- cp /root/.config /site/data0/$USER/kerrighed/kernel
- cd /site/data0/$USER/kerrighed/kernel
- make -j4
- from the deployed node, as root
- cd /site/data0/<login>/kerrighed/kernel
- make -j2 install modules_install
- mkinitrd -o /boot/initrd.img-2.6.11-krg 2.6.11-krg #(you'll see warnings, but hey should have no impact)
- from the deployed node, as a user
- cd /site/data0/$USER/kerrighed/
- ./configure --with-kernel=/site/data0/$USER/kerrighed/kernel/ --enable-tests
- from the deployed node, as root (using su ?)
- make install
- vi /etc/default/kerrighed #if you want to have Kerrighed boot automatically
- cd /tmp/image && tar --numeric-owner -zcf ../image.tgz *
You now have a Kerrighed image ready for recording, and then deployement.
Used session ids
- David Margery : 12
- Thomas Ropars : 14, 15