From Grid5000

This page describes the minimal environment based on Fedora distribution. It intends to explain how this environment was built and how to use it with Kadeploy.

Contents 1 Identification sheet 2 Build 2.1 Installation 2.2 Post installation 2.2.1 APT 2.2.1.1 Installation 2.2.1.2 Configuration 2.2.2 Upgrade packages 2.2.3 Add packages 2.2.4 NTP 2.2.5 Cleaning 2.2.5.1 Unwanted packages 2.2.5.2 Unwanted services 2.2.6 Boot kernels 2.2.7 Misc 2.2.7.1 Max open file descriptors 2.2.7.2 TCP bandwidth 2.2.7.3 g5k user 2.2.7.4 Remote console 2.3 Create image file 2.3.1 Cleaning 2.3.2 Generating image 3 Usage 3.1 Kadeploy's postinstall 3.1.1 /etc/fstab 3.1.2 /etc/apt/sources.list.d/ 3.1.3 dhclient generated files 3.2 Recording environment

Identification sheet

Build

Here are explanations on how the system was installed, how it was tuned and finally how the image was generated.

Installation

The environment was installed from a Fedora Core 4 Release DVD for x86_64. The following Kickstart file summaries the installation properties:

install
cdrom

lang en_US.UTF-8
langsupport --default=en_US.UTF-8 en_US.UTF-8 en_US en en_US.UTF-8 en_US en en_US.UTF-8 en_US en
keyboard fr-latin9
xconfig --card "NVIDIA GeForce 2 MX (generic)" --videoram 8192 --hsync 31.5-37.9 --vsync 50-70 \
  --resolution 800x600 --depth 16
network --device eth0 --bootproto dhcp
network --device eth1 --bootproto dhcp
rootpw --iscrypted $1$sVkoCVvf$diaPafbKILytFouxr8bAL1
firewall --disabled
selinux --disabled
authconfig --enableshadow --enablemd5
timezone --utc Europe/Paris
bootloader --location=none

# The following is the partition information you requested
part / --fstype ext3 --onpart sda5
part swap --noformat --onpart sda13

%packages
@ language-support
kernel-smp
e2fsprogs

Post installation

APT

In order to provide an easy way for user to install packages, like it is required for minimal environments, APT have to be installed. APT is preferred to YUM, which is the default package manager under Fedora, because many Grid'5000 sites already have an APT proxy configured. With this choice only one proxy will be used for the two main distributions.

Installation

APT for x86_64 is not available in official Fedora package repository, but we can find it on third-party repository:

$ wget http://dag.wieers.com/packages/apt/apt-0.5.15cnc6-4.rf.src.rpm
$ rpmbuild --rebuild apt-0.5.15cnc6-4.rf.src.rpm
$ rpm -Uvh apt-0.5.15cnc6-4.rf.x86_64.rpm

Configuration

APT non-availability for x86_64 in official Fedora package repository comes from x86_64 possibility to mix packages for x86_64 and i386. APT does not know how to manage two installed packaged with identical name and version, which only differ from their processor architecture.

So, in /etc/apt/apt.conf, we must told APT to let these packages coexist:

RPM {
    ...
    Allow-Duplicated-Warning "false";
};

Some repositories must be specified inside /etc/apt/sources.list.d definition files, according to local apt-cacher. Here is Nancy's configuration:

rpm http://apt.nancy.grid5000.fr/apt-cacher/ayo.freshrpms.net fedora/linux/4/x86_64 core updates extras
rpm http://apt.nancy.grid5000.fr/apt-cacher/ayo.freshrpms.net fedora/linux/4/x86_64 freshrpms

Upgrade packages

Fedora Core 4 was released on may 2005. Since this time, lot of bugs have been upgraded and lot of security holes have been fixed. Packages are freshen this way:

$ apt-get update
$ apt-get upgrade

For kernel, you have to specify the exact version, say the last one. Moreover, we do not want to bother with multiple kernel version: previous one will be removed.

$ apt-get install kernel-smp#2.6.14-1.1656_FC4
$ rpm -e kernel-2.6.11-1.1369_FC4
$ rpm -e kernel-smp-2.6.11-1.1369_FC4

Add packages

We need some supplementary packages:

• Text editors

$ apt-get install emacs vim-enhanced jed

• Compiler

$ apt-get install gcc

• Ruby interpreter

$ apt-get install ruby

NTP

NTP service have to be installed on the environement to synchronize time with site's NTP server.

$ apt-get install ntp

By default, the NTP server is not started at boot time, so:

$ chkconfig --levels 2345 ntpd on

Cleaning

Fedora system installation is a big black box. So a lot of packages are installed and a lot of services are enabled, without administrator consent. Creating minimal environment makes cleaning the installed system necessary.

Unwanted packages

Many of the installed packages are intended to a workstation use, so they are useless for a cluster compute node. Removing these packages improves system autonomy.

Workstation use packages:

$ rpm -e autofs
$ rpm -e NetworkManager
$ rpm -e cpuspeed
$ rpm -e eject
$ rpm -e bluez-hcidump bluez-libs bluez-utils
$ rpm -e pcmcia-cs
$ rpm -e irda-utils
$ rpm -e isdn4k-utils
$ rpm -e ppp wvdial rp-pppoe
$ rpm -e kudzu system-config-mouse system-config-network-tui

Some package management:

$ rpm -e up2date rhnlib
$ rm -f /etc/sysconfig/rhn/up2date-uuid.rpmsave

Printing subsystem:

$ rpm -e redhat-lsb cups cups-libs
$ rm -rf /var/log/cups

Mail subsystem and its dependencies:

$ rpm -e procmail sendmail hesiod mailcap
$ rm -f /var/log/mail/statistics.rpmsave

RAID subsystem:

$ rpm -e mdadm dmraid

Name server subsystem:

$ rpm -e bind bind-libs bind-utils caching-nameserver
$ rm -f /etc/rndc.key.rpmsave

Local Boot loader:

$ rpm -e grub
$ rm -rf /boot/grub

Misc:

$ rpm -e ksh

Unwanted services

Many services start at boot time and few of them are useless for a cluster compute node. They have to be disabled:

$ chkconfig auditd off
$ chkconfig iptables off
$ chkconfig netfs off
$ chkconfig gpm off

Boot kernels

For the environment to walk throught Grid'5000, its kernels must have all the disk drivers used in their ramdisk image at boot time. To include all these drivers into initrd files, /etc/modprobe.conf has to contain their name. So this file looks like:

alias eth0 tg3
alias eth1 tg3
# Sun V20Z
alias scsi_hostadapter mptspi
# HP DL145
alias scsi_hostadapter sata_nv
# IBM e326
alias scsi_hostadapter sata_sil
# IBM e326m
alias scsi_hostadapter sata_svw

This way, each mkinitrd initrd build will automatically include modules referenced as scsi_hostadapter. So kernel upgrade on this image will use a well formed initrd from the start. We only have to update already installed kernel's initrd.

$ mkinitrd -f /boot/initrd-2.6.14-1.1656_FC4smp.img 2.6.14-1.1656_FC4smp

Misc

Here is various hints that make the system compliant to minimal environment conventions.

Max open file descriptors

To make some experiments possible, max open file descriptors limit must be lifted. This is done modifying /etc/security/limits.conf, as it is described on the related tuning page.

TCP bandwidth

On a grid, network kernel settings must be tuned to maximize inter-site connections bandwidth. This is done by editing /etc/sysctl.conf, as it is described on the related tuning page.

g5k user

A default user is added to make non-root connections possible:

$ groupadd --gid 1000 g5k
$ useradd --gid g5k --home-dir /home/g5k --create-home --uid 1000 g5k
$ echo "grid5000" | passwd --stdin g5k

Remote console

Serial console use mgetty under Fedora. Using serial console implies defining its device properties inside /etc/mgetty+sendfax/mgetty.config.

For example, if your site serial console is bound to /dev/ttyS0, its properties can be defined this way:

port ttyS0
   speed 38400
   direct yes
   data-only yes
   toggle-dtr yes
   need-dsr yes
   port-owner root
   port-group root
   port-mode 600
   login-prompt @ \P login:\040
   login-time 60
   term vt100

After that, login must be enabled by adding one line to /etc/inittab:

s0:2345:respawn:/sbin/mgetty ttyS0

Finally, root is allowed to login via remote console by adding into /etc/securetty:

ttyS0

Create image file

Now, a valid minimal environment has been build. Its corresponding image file has to be generated.

Cleaning

At this point, the environment contains few configuration that depends of the site which installs it. Various files have to be removed or updated to create an image file as neutral as possible.

This is done by the insip4all toolkit:

$ /root/insip4all/insipid.sh

As explained on its dedicated page, this toolkit:

• Replaces specific configuration files by their neutral version or remove them
• Runs scripts to clean some databases, like APT's one

Generating image

As described in environment conventions, making image of a system using udev, as Fedora does, requires a special hint:

$ mount -o bind -t ext3 / /mnt/
$ cd /mnt
$ tar --numeric-owner -zcf /fedora4all-core4-x86_64.tgz --exclude fedora4all-core4-x86_64.tgz *

Usage

The image archive file is now created. A postinstall archive must be associated to it and the overall environment must be recorded into Kadeploy's database, in order to deploy the Fedora based minimal environment.

Kadeploy's postinstall

Postinstall archive is not unique for a system image, because postinstall depends of each site. Only requirements, for the image to run, can be expressed here.

/etc/fstab

Filesystems static informations have to be well defined for a Fedora to correctly boot up. In opposition of Debian based system, the devpts virtual filesystem properties will not be guessed by Fedora at boot time. So, if it is not defined into /etc/fstab, /dev/pts will not be mounted and SSH will be able to create valid connections.

Here is a Fedora well defined postinstall /etc/fstab:

proc            /proc           proc   defaults   0       0
sysfs           /sys            sysfs  defaults   0       0
devpts          /dev/pts        devpts gid=5,mode=620 0   0
tmpfs           /dev/shm        tmpfs  defaults   0       0

/etc/apt/sources.list.d/

In this directory, files describe repositories to use with APT to download packages. So for APT to work, some file named repository.list, where repository must be replaced by repository name, must be placed into this directory.

For example, to use base distribution repository hosted at ayo.freshrpms.net and cached by apt.nancy.grid5000.fr, you have to put inside a os.list file:

rpm http://apt.nancy.grid5000.fr/apt-cacher/ayo.freshrpms.net fedora/linux/4/x86_64 core updates extras

dhclient generated files

Some configuration files are automatically generated or modified by dhclient according to informations offered by DHCP server. So it not necessary to include these files into Kadeploy postinstall archive. These files are:

/etc/ntp.conf
/etc/ntp/step-tickers
/etc/resolv.conf

Recording environment

Recording environment can be done from a description file. So we create fedora4all-core4-x86_64.dsc:

name = fedora4all-x86_64
description = fedora based minimal environment
author = xavier.delaruelle@loria.fr
filebase = file://IMAGE_PATH/fedora4all-core4-x86_64.tgz
filesite = file://ASSOCIATED_POSTINSTALL_FILE
size = 1000
initrdpath = /boot/initrd-2.6.14-1.1656_FC4smp.img
kernelpath = /boot/vmlinuz-2.6.14-1.1656_FC4smp
fdisktype = 83
filesystem = ext3

With karecordenv, the new environment can be known by Kadeploy:

karecordenv -fe fedora4all-core4-x86_64.dsc