Multi-site environment creation
This practice was designed for the old OAR1 infrastructure of Grid'5000 and is now deprecated.
This practice shows how to create multi-site Kadeploy environments. It is split in 2 parts:
- compilation of a Linux kernel working on all sites.
- configuration of a GNU/Linux system environment working on all sites.
The first part will quickly recall Linux kernel modular compilation and the initrd boot mechanism. A recent Debian kernel is taken as an example (mkinitd + 2.6.15 Debian + udev). The work is based on the debian4all kadeploy environment, which a miscellanious kernel will be recomplied for, using Grid'5000 hardware information (found on the wiki). Then a new initrd will be generated against this kernel and finally a new kadeploy environment will be recorded.
The second part details the main options needed to get a working network configuration on all sites for instance. Then it shows how to customize an environment postinstall (things to do or not to do in a postinstall).
compiling multisite kernels
Compilation environment setup
where the compilation environment is setup with the latest available kernel sources
derivating the compilation environment from a base environment
The first step is to reserve 2 nodes, for a sufficient amount of time (2 hours should be alright), and to deploy the first one with the
oarsub -I -q deploy -l nodes=2,walltime=2 kadeploy -e debian4all -p hda6 -m first_node
Then log on the first node as
grid5000) and install compilation tools
apt-get update apt-get install gcc make libncurses-dev linux-source-2.6.15
Now all the base tools required for the kernel compilation are available on the node.
cd /usr/src/ tar -jxf linux-source-2.6.15.tar.bz2 cd linux-source-2.6.15
This way, we have a complete environment, as well as the kernel sources, without having to retrive any file from the outside of the confined grid5000 network. The main advantage from retriving kernel sources from a classical linux distribution lies in the default kernel source configuration already containing many configured usefull options. I will describe those in the next section.
Those options ensure the resulting compiled kernel compatibility with the specific grid5000 boot process and hardware flexibility: the kernel should be modular and compatible with the monitoring hardware installed on every site (mainly for remote console through
The task consists only to browsee available options an verify if the main ones are selected:
Loadable module support <*> Enable loadable module support Device Drivers Block devices <*> RAM disk support (16) Default number of RAM disks (65536) Default RAM disk size (kbytes) [*] Initial RAM disk (initrd) support
remote console capabilities
Device Drivers Character devices Serial drivers <*> 8250/16550 and compatible serial support [*] Console on 8250/16550 and compatible serial port
Now we have to check if the main filesystems are supported in the
File systems section,
Ext3 must be supported and even more if required (
nfs....). Those can be compiled as modules.
The initrd filesystem should be built within the kernel (
built-in: depending on the tools used to generate it or the linux distribution used (and its version since things evolves), its format differs (debian:
cpio, fedora: gzipped
Hardware support for different sites
All the modules required by every grid5000 sites should be added to the kernel configuration. They are referenced on the hardware page.
Depending on the kernel version you want to compile, the location of some modules can differ, their name can also differ. One should verify manually all the kernel options, when upgrading, instead than just using the previous kernel configuration file.
Compilation and installation of the kernel
Very standard (for 2.6 kernels):
make -j2 && make -j2 modules_install
Now, the kernel is compiled, and all the selected modules are installed
Copying the kernel and building the
To install the kernel, the standard way:
cp arch/x86_64/boot/bzImage /boot/vmlinuz-2.6.15
For my example, I will use
First, you have to edit the file
/etc/mkinitrd/modules to verify that it contains all the modules required by all the disk controller of all the grid5000 target sites for your environment (cf hardware page).
Then, to generate the
mkinitrd -o /boot/initrd.img-2.6.15 2.6.15
Testing the kernel
Creating, registering and deploying a new environment
You must create and register the new environment with this freshly built kernel: the process is the same than the one involved in the Deploy an environment practice. You should refer to it, as well as the page dedicated to kadeploy.
You just have to register the new environement with
karecordenv specifying the compiled kernel
/boot/vmlinuz-2.6.15 and initrd
kaenvironments -e debian4all > image_tpexpert.dsc
image_tpexpert.dsc so that it looks almost like this be carefull, you should only take the information related to the latest version of the environment:
name = tpexpert description = image en noyau 2.6.15 compilé à la main qui marche partout author = email@example.com filebase = file://home/<site>/<login>/image_tpexpert.tgz filesite = file://grid5000/postinstalls/post4all.tgz size = 1000 initrdpath = /boot/initrd.img-2.6.15 kernelpath = /boot/vmlinuz-2.6.15
Then register it with:
karecordenv -fe image_tpexpert.dsc
Now, we will use the second node to test the deployment procedure, and watch the boot process with
kaconsole to debug boot problems.
If everything is alright:
ssh root@second_node uname -sorm Linux 2.6.15 x86_64 GNU/Linux
minimal multisite configuration
I will detail all tha main options to have a working network configuration, as well as a guide to customize an environment's postinstall (as well as what to do or not in the postinstall).