From Grid5000
Jump to: navigation, search

This page describes the 1.0 version of the minimal environment based on Etch version of Debian distribution for AMD64/EM64T machines. It intends to explain how this environment was built and how to use it with Kadeploy. This page is inspired from Sid-x64-base-1.0 and Sid-x64-base-1.1


Identification sheet


Kernel version 2.6.18-6-amd64 from Debian for amd64/em64t


  • Remote console: enabled on ttyS0 at 34800 bps
  • Services: ldap:no, nfs:no
  • Accounts: root:grid5000, g5k:grid5000




Here are explanations on how the system was installed and tuned.


Start-it with a debian deployed environment

The environment is initially installed from debootstrap. On the node, the debootstrap package was installed.

Create a directory where you want to install ("where" doesn't have many importance):

mkdir /mnt/hda3

To know what you can install, type the following command :

ls /usr/share/debootstrap/scripts/  

We want to initialise the installation with an etch :

debootstrap  etch  /mnt/hda3  

There is some task to do before chroot in our new etch :

cp /etc/fstab /mnt/hda3/etc/fstab
cp /etc/network/interfaces /mnt/hda3/etc/network/interfaces 
cp /etc/hostname >  /mnt/hda3/etc/hostname
mount -o bind /dev /mnt/hda3/dev  
mount -o bind /proc /mnt/hda3/proc 

You can now chroot in your new environment

chroot /mnt/hda3  /bin/bash

The following procedure was done

apt-get update
apt-get install locales
apt-get install linux-image-2.6.18-6-amd64

At this step, only base system is installed and minor things are configured:

  • root account is created with grid5000 as password
  • Default locale is set to en_US.UTF-8. en_GB.UTF-8 and fr_FR.UTF-8 are also configured

Finish-it with APT

APT must use etch's repositories in /etc/apt/sources.list:

deb etch main contrib non-free

Moreover, APT must know the HTTP proxy to use to access the repository:

export http_proxy=""

After that, packages are upgraded to their last version:

apt-get update
apt-get dist-upgrade

Post installation


Packages authentication requires a valid repository's keys. These set of keys is hard to maintain on a grid environment. To avoid problems, APT is told to not authenticate packages in /etc/apt/apt.conf:

APT::Get::AllowUnauthenticated "true";

Add packages

Default Debian install does not contain all the packages we need:

  • Text editors
apt-get install vim less xemacs21 gawk jed joe nano
  • Cleaning tools
apt-get install cruft debfoster deborphan localepurge
  • Script interpreters
apt-get install ruby python perl
  • Alternate shell
apt-get install tcsh
  • ia32/i386 shared libraries
apt-get install ia32-libs
  • Network tools
apt-get install tcpdump netcat nmap 
  • Misc
apt-get install acpi bzip2 rsync strace tree x11-common rcs ethtool


SSH client and server are installed to allow remote connections:

apt-get install ssh

Minor changes are done in configuration file /etc/ssh/sshd_config:

PrintMotd no
PasswordAuthentication no
ChallengeResponseAuthentication yes
PermitEmptyPasswords no
IgnoreUserKnownHosts yes
X11Forwarding yes


The ifup and ifdown programs work with so-called "physical" interface names. These names are assigned to hardware by the kernel. Unfortunately it can happen that the kernel assigns different physical interface names to the same hardware at different times; for example, what was called "eth0" last time you booted is now called "eth1" and vice versa. This creates a problem if you want to configure the interfaces appropriately. A way to deal with this problem is to use mapping scripts that choose logical interface names according to the properties of the interface hardware. --quoted from interfaces man page

To avoid this assignment problem, ifrename helps to freeze an interface name on a given MAC address through the /etc/iftab:

apt-get install ifrename

/etc/iftab will be automatically filled by postinstall mechanisms.

DHCP client

Headline text

Request time-out is reduced in /etc/dhcp3/dhclient.conf, to avoid waiting on an interface with no media associated:

timeout 25;

NTP server address must be retrieved by DHCP client, so ntp-servers request is added to request list in /etc/dhcp3/dhclient.conf:

request subnet-mask, broadcast-address, time-offset, routers,
        domain-name, domain-name-servers, host-name, ntp-servers,
        netbios-name-servers, netbios-scope, interface-mtu;

Some scripts are needed to adequately configure the host after receiving its network properties:

  • /etc/dhcp3/dhclient-exit-hooks.d/update-host-name, which updates hostname
  • /etc/dhcp3/dhclient-exit-hooks.d/update-ntp-server, which updates the NTP server to use


To sync time, the NTP service is installed:

apt-get install ntp

Then server statements are commented out from /etc/ntp.conf to stay site-neutral:

#server iburst
#server iburst
#server iburst
#server iburst

At last, sync at boot time is done by a third part tool:

apt-get install ntpdate


This environment intends to provide authentication to only the local accounts. This way LDAP or NFS distrubances are avoided during experiments. Thus PAM's common configuration, describes by the /etc/pam.d/common-* files, only refer to the pam_unix module:

account  required
auth     required likeauth nullok_secure 
password required nullok md5 obscure min=6 max=8
session  required 

Boot kernel


Recent Debian kernel image include support for Kexec:

$ zgrep KEXEC /proc/config.gz

Kexec tools are needed to have complete support of this feature:

apt-get install kexec-tools

Site walkthrough

Please read the page dedicated to the setup of the site walkthrough to learn what was done on this environment to configure the site walkthrough.

Myri10g driver


cd /usr/local/src/
tar xvfz mx_1.2.4.tar.gz
cd mx-1.2.4
./configure --prefix=/usr/local --enable-64b --enable-10g --enable-2g
make install

Initialization script

ln -s /usr/local/sbin/mx_start_stop /etc/init.d/mx
update-rc.d mx defaults


ifconfig -a

Infiniband driver

Install Infiniband utilities :

apt-get install ibverbs-utils libibverbs1 libibverbs-dev libmthca1

Force the loading of Infiniband drivers : /etc/modules


Test if Infiniband is working :

ifconfig -a

Start a ping-pong test :

node-1# ibv_ud_pingpong &
node-1# ibv_ud_pingpong

e1000 driver

Last drivers are needed to support the new ethernet gigabit interface of the chinqchint cluster nodes.


cd /usr/local/src/
tar xvfz e1000-
cd e1000-
make install


udev, which provides a dynamic /dev directory, creates at boot time device node files and tries to set their ownership. These ownership settings generate NSS requests. So it generates LDAP requests when users or groups are locally unknown and a LDAP server is used to map users and groups.

Some groups, whose node device's permissions are managed by udev, are locally unknown (eg. they are not defined into /etc/group). When udev creates their device node file, if it tries a LDAP NSS request it will wait during minutes before falling, because network is down when all of this happens and so udev cannot reach the LDAP server.

To avoid this behaviour, it is a good solution to disable udev's locally unknown group permission rules in /etc/udev/permissions.rules:

#SUBSYSTEMS=="scsi", ATTRS{type}=="3", ATTRS{vendor}=="HP",     GROUP="scanner"
#SUBSYSTEMS=="scsi", ATTRS{type}=="3", ATTRS{vendor}=="Epson",  GROUP="scanner"
#SUBSYSTEMS=="scsi", ATTRS{type}=="6",                          GROUP="scanner"
#KERNEL=="nvram",                               GROUP="nvram"
#KERNEL=="tpm*",                        MODE="0600",    OWNER="tss", GROUP="tss"
#KERNEL=="fuse",                                        GROUP="fuse"
#KERNEL=="uverbs*",                             GROUP="rdma"
#KERNEL=="ucm*",                                        GROUP="rdma"


Here is various hints that make the system compliant to minimal environment conventions.

Web proxy

Please read the page dedicated to the setup of the Web proxy client configuration to learn what was done on this environment to configure the use of this service.

Max open file descriptors

To make some experiments possible, max open file descriptors limit must be lifted. This is done modifying /etc/security/limits.conf, as it is described on the related tuning page.

TCP bandwidth

On a grid, network kernel settings must be tuned to maximize inter-site connections bandwidth. This is done by editing /etc/sysctl.conf, as it is described on the related tuning page.

g5k user

You have to setup the root passwd

$ passwd

A default user is added to make non-root connections possible:

$ groupadd --gid 1000 g5k
$ useradd --gid g5k --home-dir /home/g5k --create-home --uid 1000 g5k
$ passwd g5k

Remote console

Serial console use getty under Debian. Serial console login is configured in /etc/inittab:

T0:23:respawn:/sbin/getty -L ttyS2 38400 vt100

root is allowed to login via remote console by adding into /etc/securetty:


Default editor

We need to configure a apolitical text editor to avoid a war between Emacs and Vim users:

update-alternatives --set editor /bin/nano

Message Of The Day

Pieces of the image's identification sheet are put inside the Message Of The Day file to help users understanding what they can do with this image. Debian generates the Message Of The Day file at boot time and bases it on the /etc/motd.tail file. This base file was modified to provide the identification sheet information.


During CD setup, unwanted packages were installed and have to be removed. There are

  • Workstation tools
apt-get --purge remove laptop-detect
  • Misc
apt-get --purge remove base-config dselect
apt-get --purge remove mawk nvi 

At the end, unused libraries are also removed:

apt-get --purge remove $(deborphan)

If ia32-libs were removed by deborphan, you can reinstall ia32-libs

apt-get install ia32-libs 


Creating image's archive

Now, a valid minimal environment has been build. Its corresponding image file has to be generated. At this point, the environment contains configuration files that depends of the site which installs it. These files must be excluded from image archive to stay as neutral as possible.

TGZ-G5K allow to create Grid'5000 neutral system archive, so it is installed on this environment:

dpkg -i tgz-g5k_1.0.2-2_all.deb

System archive creation and retrieving is now an easy task:

tgz-g5k cconstantin@frontale:images/etch-x64-base-1.0.tgz

Creating postinstall's archive

Postinstall archive is associated to image archive because the latter is site neutral and as any image, it needs to become site-specific on deployed system to perform well. etch-x64-base-1.0-post, etch-x64-base-1.0's postinstall, takes advantage from prepost mechanisms so it is site-independent: the same postinstall's archive is used over all the Grid'5000 sites.

Postinstall's script

Postinstall's script manages to transform the site-neutral deployed system into a site-specific one. As etch-x64-base-1.0-post relies on prepost mechanisms, prepostinst plays the role of the postinstall's script.

Postinstall's template files

Postinstall's configuration files of etch-x64-base-1.0 are templates as for any prepost-aware postinstall. Thanks to these templates, postinstall is site-independant. Following template files are put inside etch-x64-base-1.0-post:

Note.png Note

Some configuration files are not required in postinstall's archive, because they are automatically generated or modified on deployed system at boot time:

  • /etc/ntp.conf and /etc/resolv.conf are generated by dhclient and its related exit-scripts
  • /etc/default/ntpdate automatically guesses itself which NTP server to use

Recording environment

Recording environment can be done from a description file. So we create etch-x64-base-1.0.dsc:

name = etch-x64-base-1.0
description =
author =
filebase = file:///grid5000/images/etch-x64-base-1.0.tgz
filesite = file:///grid5000/postinstalls/etch-x64-base-1.0-post.tgz
size = 1000
initrdpath = /boot/initrd.img-2.6.18-6-amd64
kernelpath = /boot/vmlinuz-2.6.18-6-amd64
fdisktype = 83
filesystem = ext2

With karecordenv, the new environment can be known by Kadeploy:

karecordenv -fe etch-x64-base-1.0.dsc
Personal tools

Public Portal
Users Portal
Admin portal
Wiki special pages