From Grid5000

This practice shows how to create multi-site Kadeploy environments. It is split in 2 parts:

• compilation of a Linux kernel working on all sites.
• configuration of a GNU/Linux system environment working on all sites.

The first part will quickly recall Linux kernel modular compilation and the initrd boot mechanism. A recent Debian kernel is taken as an example (mkinitd + 2.6.15 Debian + udev). The work is based on the debian4all kadeploy environment, which a miscellanious kernel will be recomplied for, using Grid'5000 hardware information (found on the wiki). Then a new initrd will be generated against this kernel and finally a new kadeploy environment will be recorded.

The second part details the main options needed to get a working network configuration on all sites for instance. Then it shows how to customize an environment postinstall (things to do or not to do in a postinstall).

Contents 1 compiling multisite kernels 1.1 Compilation environment setup 1.1.1 derivating the compilation environment from a base environment 1.2 Kernel options 1.2.1 Necessary options 1.2.1.1 initrd support 1.2.1.2 remote console capabilities 1.2.1.3 filesystems support 1.2.2 Hardware support for different sites 1.3 Compilation and installation of the kernel 1.3.1 Copying the kernel and building the initrd 1.3.1.1 initrd 1.4 Testing the kernel 1.4.1 Creating, registering and deploying a new environment 1.4.2 Troubleshooting 2 minimal multisite configuration

compiling multisite kernels

Compilation environment setup

where the compilation environment is setup with the latest available kernel sources

derivating the compilation environment from a base environment

The first step is to reserve 2 nodes, for a sufficient amount of time (2 hours should be alright), and to deploy the first one with the debian4all environment.

oarsub -I -q deploy -l nodes=2,walltime=2
kadeploy -e debian4all -p hda6 -m first_node

Then log on the first node as root (passwd: grid5000) and install compilation tools

apt-get update
apt-get install gcc make libncurses-dev linux-source-2.6.15

Now all the base tools required for the kernel compilation are available on the node.

cd /usr/src/
tar -jxf linux-source-2.6.15.tar.bz2
cd linux-source-2.6.15

This way, we have a complete environment, as well as the kernel sources, without having to retrive any file from the outside of the confined grid5000 network. The main advantage from retriving kernel sources from a classical linux distribution lies in the default kernel source configuration already containing many configured usefull options. I will describe those in the next section.

Kernel options

Classical:

make menuconfig

Necessary options

Those options ensure the resulting compiled kernel compatibility with the specific grid5000 boot process and hardware flexibility: the kernel should be modular and compatible with the monitoring hardware installed on every site (mainly for remote console through kaconsole).

The task consists only to browsee available options an verify if the main ones are selected:

initrd support

Loadable module support
  <*> Enable loadable module support
Device Drivers
  Block devices
    <*> RAM disk support
    (16)  Default number of RAM disks
    (65536) Default RAM disk size (kbytes)
    [*]   Initial RAM disk (initrd) support

remote console capabilities

Device Drivers
  Character devices
    Serial drivers
      <*> 8250/16550 and compatible serial support                                        
      [*]   Console on 8250/16550 and compatible serial port

filesystems support

Now we have to check if the main filesystems are supported in the File systems section, Ext2, Ext3 must be supported and even more if required (nfs....). Those can be compiled as modules.
The initrd filesystem should be built within the kernel (built-in: depending on the tools used to generate it or the linux distribution used (and its version since things evolves), its format differs (debian: cramfs, then cpio, fedora: gzipped ext2 then cpio).

Hardware support for different sites

All the modules required by every grid5000 sites should be added to the kernel configuration. They are referenced on the hardware page.
Depending on the kernel version you want to compile, the location of some modules can differ, their name can also differ. One should verify manually all the kernel options, when upgrading, instead than just using the previous kernel configuration file.

Compilation and installation of the kernel

Very standard (for 2.6 kernels):

make -j2 && make -j2 modules_install

Now, the kernel is compiled, and all the selected modules are installed

Copying the kernel and building the initrd

To install the kernel, the standard way:

cp arch/x86_64/boot/bzImage  /boot/vmlinuz-2.6.15

initrd

For my example, I will use mkinitrd.
First, you have to edit the file /etc/mkinitrd/modules to verify that it contains all the modules required by all the disk controller of all the grid5000 target sites for your environment (cf hardware page).
Then, to generate the initrd:

mkinitrd -o /boot/initrd.img-2.6.15 2.6.15

Testing the kernel

Creating, registering and deploying a new environment

You must create and register the new environment with this freshly built kernel: the process is the same than the one involved in the Deploy an environment practice. You should refer to it, as well as the page dedicated to kadeploy. You just have to register the new environement with karecordenv specifying the compiled kernel /boot/vmlinuz-2.6.15 and initrd /boot/initrd.img-2.6.15.

kaenvironments -e debian4all > image_tpexpert.dsc

Then, edit image_tpexpert.dsc so that it looks almost like this be carefull, you should only take the information related to the latest version of the environment:

name = tpexpert
description = image en noyau 2.6.15 compilé à la main qui marche partout
author = prenom.nom@monlabo.fr
filebase = file://home/<site>/<login>/image_tpexpert.tgz
filesite = file://grid5000/postinstalls/post4all.tgz
size = 1000
initrdpath = /boot/initrd.img-2.6.15
kernelpath = /boot/vmlinuz-2.6.15

Then register it with:

karecordenv -fe image_tpexpert.dsc

Now, we will use the second node to test the deployment procedure, and watch the boot process with kaconsole to debug boot problems. If everything is alright:

ssh root@second_node uname -sorm
 Linux 2.6.15 x86_64 GNU/Linux

Troubleshooting

minimal multisite configuration

I will detail all tha main options to have a working network configuration, as well as a guide to customize an environment's postinstall (as well as what to do or not in the postinstall).