GRUDU

From Grid5000
Jump to: navigation, search


Contents

GRUDU

Version note

This version of the page corresponds to GRUDU v1.1.2

GRUDU web page

GRUDU has now a dedicated web page on the INRIA GForge : http://grudu.gforge.inria.fr. On the site you can access different information :

  • The latest news concerning the project
  • The important dates for GRUDU
  • The presentation of the features of GRUDU
  • The links to the different projects elements such as the bug tracking systems, the forums, the mailing lists, etc.
  • The different versions with the elements to download.
  • The wiki where you have all the information about the use of GRUDU.
  • A page where ou can contact the GRUDU team if you have any question, comments, etc.

Downloading GRUDU

To download GRUDU, you can visit the following page http://grudu.gforge.inria.fr/downloads.html where you will find the latest release.

Introduction

If you encounter installation difficulties don’t hesitate to send an email to: mailto:grudu-users@lists.gforge.inria.fr. If you find a bug in GRUDU, please don’t hesitate to submit a bug report on Bug tracking system. If you have multiple bugs to report, please make multiple submissions, rather than submitting multiple bugs in a single report.

If you want to provide feedback, ask for a feature that the software does not provide do not hesitate to send features request to the repecting tracking system at Feature Requests Tracking System.

A mailing list concerning the development of GRUDU is also available at the following address: mailto:grudu-devel@lists.gforge.inria.fr.

Requirements and installation

Requirements

Since GRUDU is dedicated to GRID’5000 grid computing infrastructure, the first thing you need to use, is a GRID’5000 account and an access to, at least one cluster of those composing the platform. For more information about how to access GRID’5000 grid, please refer to the web pages of GRID’5000: SSH, First_steps or OAR.

GRUDU is written in Java and thus it can be executed on any platform that offers a recent version of the Java runtime (At least, 1.5.0 version or higher). Currently, GRUDU support only Bourne shell, so if your GRID’5000 account uses another shell type, you need to change it. This requirement is only for your GRID’5000 account and you still can use your preferred shell on your machine.

To allow GRUDU to access to all the platform, you need to configure your account to authorize a direct access to the different sites. To do that make sure you all these conditions are fulfilled:

  • you have your ssh key in every site: each GRID’5000 site has its own NFS filesystem, so you need to copy your ssh key (at least the public one) in your .ssh directory.
  • you have you public ssh key in the file $HOME/.ssh/authorized_keys. You can do this by a command like this:
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
  • the following option should be present in your $HOME/.ssh/config file :
Host *
StrictHostKeyChecking no

For more information about ssh access to GRID’5000, key management or file exchange please refer to the SSH page.

Installation

Automatic installation

GRUDU is provided through a single installation jar file containing the GRUDU software, the required libraries, the source files and the documentation (User Manual and JavaDoc). This installation file has been created with IzPack (IzPack is an installer’s generator for the Java platform).

To launch the installer you can either double-click on the installer jar file (works on operating systems where the jar mime-type is managed by java), or launch the jar file from a shell terminal with the following command:

java -jar GRUDU_installer.jar

The installation is separated into two parts: the installation of the software itself (the jar file, the libraries and the resources files), and then its configuration (locally and remotely).

Installation of the software

The first one corresponds to the selection of the different packages you want to install. Four packages are available:

  • The base package contains the software and the mandatory libraries (It is required).
  • The JFTP module for GRUDU. This module corresponds to a File transfert Protocol module. This module allows you to transfert data between GRID’5000 and your local machine, but also between the frontales of GRID’5000.
  • The Ganglia module for GRUDU. This module corresponds to a plugin retrieving data from Ganglia to display low-level information about all the nodes of a site or the history of these metrics for the nodes of your jobs.
  • The documentation package corresponds to the User’sManual and the JavaDoc of GRUDU.
  • The source code of GRUDU.

Installation packages selection

If you have an Unix-like operating system (Linux or BSD variants) or Windows, the seventh panel will allow you to put shortcuts on your desktop and also in the program group if you want to.

Shortcuts configuration

Configuring GRUDU

After having installed GRUDU, you should configure it. The configuration panel is separated into two parts, the first concerns the access to GRID’5000. In this tab you have to define :

  • a preferred access point (the external frontal that will be used to enter in the GRID’5000 network). The ComboBox contains the different sites of GRID’5000.
  • your user name (your GRID’5000 login)
  • your ssh public key, rsa or dsa (for more information about ssh access with public/private keys, please refer to the SSH page.

GRID’5000 access configuration

The second tab of the panel consists in selecting the clusters you want to enable in GRUDU (e.g. the clusters that will be considered when launching oarstat or oarnodes commands, or when reserving machines). In this tab you will also be able to define the partitions used by KaDeploy for the deployment of an image (for more informations about the partitions you can specify please refer to the pages of the sites on the or to the messages of the day displayed when you get connected to a cluster).

GRID’5000 sites configuration

When all these information will be filled out, you will be able to write the configuration by clicking the “Write configuration” button. This action will create the local hierarchy of files mandatory for GRUDU (a directory called .diet containing all the files for GRUDU will be created in your home directory), and the remote hierarchy of files mandatory for GRUDU (approximately the same on the clusters of GRID’5000).

Manual installation

If you want to compile GRUDU sources, you need to install the Java build tool Ant. Then simply execute the following command: ant GRUDU for the compilation. If the compilation succeeds, you will get a new jar file GRUDU.jar representing the program. Launching GRUDU can be done like a typical jar file:

 java -jar GRUDU.jar. 

It is preferable that your first step with GRUDU is to configure it. The configuration window contains three tabs, the first corresponding to the user parameters, the second tab is relative to GRID’5000 sites and the last one allows you to define specific Kadeploy partitions for the sites’ clusters.

The following figure represents the first tab of configuration windows, where we can found the following parameters:

  • Preferred access point: this is the GRID’5000 site GRUDU will use to access to all the platform. Your account in this site must contain your private ssh key file.
  • Username: your GRID’5000 username.
  • Private SSH key file: your private ssh key file.

Configuration window: First tab

The second configuration tab shown in the following figure allows the user to configure the following parameters:

  • Enabling/Disabling a site: by default all sites are disabled.
  • Kadeploy partition for each site (the same partition will be used for every clusters of that site).
  • OAR batch Scheduler main version (you can choose between oar1 and oar2)

Configuration window: Second tab

The third tar shown in the following figure allows the user to configure the Kadeploy partition for every clusters independently.

Configuration window: Third tab

Interface presentation

GRUDU is composed of one principal frame shown in the following figure. From this frame the user will be able to:

  • Log in GRID’5000
  • Monitor GRID’5000 and his/her reservations
  • Have a terminal on the different sites main nodes of his/her reservations on GRID’5000
  • Deploy images through Kadeploy on the appropriate nodes
  • Exchange files between the locale machine and GRID’5000 but also synchronize files between GRID’5000 frontends.
  • Log out
  • Display the Help of GRUDU

Main interface of GRUDU

Legend of the figure:

  • A Options toolbar (left-hand side).
    • 1 Button used to log in GRID’5000 (When connected to GRID’5000 you will have a button to log out).
    • 2 Button used to display the reservation frame.
    • 3 Button used to update the GRID’5000 tree of sites and jobs.
    • 4 Button used to display the configuration frame.
    • 5 Button used to display a summary of the information about GRID’5000 and your reservations.
    • 6 Button used to display a terminal on the preferred access point you have defined.
    • 7 Button used to display the Kadeploy frame for the deployment if images with user defined environments.
    • 8 Button used to display the JFTP module for GRUDU. This module allows the user to transfert data between your locale machine and GRID’5000. You can also transfert data between the GRID’5000 frontales.
  • B Options toolbar (right-hand side)
    • 9 JavaHelp dedicated to the Help of GRUDU.
    • 10 Application settings of GRUDU.
  • C Legend of the colors used for the sites, clusters and jobs.
  • D Main panel where information are displayed. Information about GRID’5000, the different sites and the jobs are displayed here.
  • E GRID’5000 sites and jobs.
    • 11 Root node of the GRID’5000 sites and jobs tree. this node allows you to display information about the grid. When right-clicking on this node, you can either update the GRID’5000 view, open a shell on your preferred access point or delete all your reservations on GRID’5000.
    • 12 Node corresponding to a site. Information about the site, i.e. the occupation of the nodes and the existing reservations on this site. When right-clicking on a site node, you can either delete the reservations you have on the site or open a shell on the cluster frontale.
    • 13 Node corresponding to a job. Information on the job are displayed in the information panel. When right-clicking on that node, you will able to delete the corresponding reservation, update the site view or open a shell on the main node of the corresponding job.

A tip of the day frame is shown (is you want so) at GRUDU startup and presents you some tips for the use of GRUDU. You can enable/disable the frame in the application configuration frame (see 9).

Tip of the day for GRUDU

Resources allocation

Using OAR interface

The most used operation is probably resources allocation. In GRID’5000, this operation can be done by the OAR system. GRUDU provides an easy way to manipulate OAR (either the OAR1 or OAR2 versions). The resources allocation window on the following figure shows a map of France with GRID’5000 sites and jobs characteristics (time, queue, oargridsub behaviour, the script to launch). These information are presented on the first tab of the window. The second tab provides the definition of the properties for the different sites. Since some sites include more than one cluster, you have to click first on the site, and then select the number of desired nodes for each cluster or you can specify that you do not care where they are located). When selecting resources numbers, the map displays the total number of requested resources for each site. Jobs characteristics are:

  • Time parameters: date and reservation walltime. The starting time can be specified manually, or through the use of a calendar for the day and through boxes where you can specify the hour, the minute and the second. For the walltime you have to define the number of hours and the minutes of your reservation.
  • Queue: default, deploy (for Kadeploy) or allow_classic_ssh (specific for OAR2 but corresponds to default for OAR1).
  • OARGridSub behaviour: the user can specify if the reservation should be done with the OARGridSub behaviour, i.e. when the user chooses to realize several reservations on different sites, if one fails, all the previous successful reservations are deleted.
  • A script to launch: The user can specify a script that will be launched in order to be executed on the reserved nodes. The reservation will be stopped when the script ends.

Resources allocation frame – Main panel

Concerning the second tab of the window presented on the following figure, it allows the user the ability to define the OAR properties that will be used for the reservation.

Resources allocation frame – OAR properties for the reservation

After the reservation, a status frame summarizes the information about the success (or not) of your jobs submission.

Resources allocation frame – Reservation status

Monitoring GRID’5000

To monitor GRID’5000 three views can be displayed:

  • The first view in the following Figure corresponds to the GRID’5000 view. You can see the occupancy of the grid in term of free/occupied/dead/absent/suspected/possessed by you for each site and for the entire grid. Added to the states of the nodes, you can also see which nodes you have reserved. You can also see a table summarizing these information. Finally you have a table of your reservation(s) on the grid. Thanks to two buttons you can save your reservation(s) in a directory for a future use (for example in the DIET Mapping Tool or with the XMLGoDIETGenerator)

Grid’5000 view

  • The second view in the following figure corresponds to a site view. A graph represents the different numbers of nodes for each node state ant the ones corresponding to your possible reservation(s), a table presents these information in a different way. Another table presents the reservation(s) realized on the site. You can also display a Gantt chart of the different reservations of the cluster to know when you are able to reserve.

If you selected the Ganglia plugin at the installation step, you also have a button bar on the right hand side of the frame that will be populated with a Ganglia information button allowing you to get low-level information on every machines of the site (data are instantaneous).

Site view

Site view

  • The third view in the following figure corresponds to the job view. Here you can see the different information of the job such as the nodes of the reservation, the job state, the walltime, etc . . . If you selected the Ganglia plugin at the installation step, you also have a button bar on the right hand side of the frame that will be populated with a Ganglia history information button allowing you yo get history on the low-level information of the nodes of your reservation.

Job view

Job view

The Ganglia module for GRUDU

Ganglia short introduction

Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.

Ganglia is an open-source project that grew out of the University of California, Berkeley Millennium Project which was initially funded in large part by the National Partnership for Advanced Computational Infrastructure (NPACI) and National Science Foundation RI Award EIA-9802069. NPACI is funded by the National Science Foundation and strives to advance science by creating a ubiquitous, continuous, and pervasive national computational infrastructure: the Grid. Current support comes from Planet Lab: an open platform for developing, deploying, and accessing planetary-scale services.

You can find more information on the Ganglia Website at http://ganglia.sourceforge.net .

Ganglia plugin

As Ganglia is installed on GRID’5000, the GRUDU users can have acces to the information provided by the software inside GRUDU. If you selected the ganglia plugin during the installation step of GRUDU, there is two ways to use it:

  • From the site information panels, where you can have instantaneous low-level information on every nodes of the site (computation nodes but also frontends).

Ganglia plugin for the site view

  • From the job information panels, where you can get the history of the low-level information brought to you by Ganglia. Concerning the generation of the history you have first to configure the history generation, which means : defining the period of data refreshing, the range of the chart, and the path to the java home on the main node of your reservation for the launch of the remote jar creating the history.

Configuration of the Ganglia plugin for the job view

Ganglia plugin for the job view

JFTP Module for GRUDU

Presentation

[1] is an acronym for Java File Transfert Protocol. [2] is a graphical Java network and file transfer client. At the origin [3] is developed as a project under GNU GPL license. You can find more information about the initial project at http://j-ftp.sourceforge.net/ .

[4] has been modify to corresponds to the needs of the GRUDU users. Thanks to the modified [5] you can transfer data between you local machine and GRID’5000, but also between GRID’5000 frontales.

Interface

The [6] module presents three internal frames, one for the local machine, one for GRID’5000 with one tab per site, and the last frame for the log of the module.

[Image:GRUDU_GRUDU_jftp1.png|JFTP main interface]]

For the configuration of the options for the Rsync transfert between GRID’5000 frontales, you can click on the option menu and you will find the following frame where you can edit the Rsync options:

JFTP rsync options

Deploy your Kadeploy images

Kadeploy is a fast and scalable deployment system for cluster and grid computing. Kadeploy is the reconfiguration system used in GRID’5000, allowing the users to deploy their own OS on their reserved nodes.

For more information about how to create and manage Kadeploy images, please refer to the Kadeploy page of the wiki.

The following figure shows you the frame allowing you to deploy images on which you have rights. The left hand side of the frame corresponds to clusters and nodes available for deployment (i.e. reserved on the deploy queue). You can click on the checkboxes to select/deselect the nodes. If you want to select/deselect all nodes, you can click on the corresponding button on the right hand side of the frame. Then you can select the image you wan to deploy from the lists on the right hand side of the frame.

When you are done with the configuration, you can click on the deploy button. A new frame will be displayed corresponding to the log of the deployment (with both standard output and error).

Kadeploy frame

Application settings

By clicking the application settings button the following frame appears :

Application settings

For the instance you can only define if you want the tip of the day frame to be displayed on startup.

Troubleshooting

Some problems can appear during the installation or during the use of GRUDU. Here are some solutions :

  • Question : I am under Gentoo, and when I am launching the installer/GRUDU, I get the following error :
java: xcb_xlib.c:50: xcb_xlib_unlock: Assertion ‘c->xlib.lock’ failed.
Aborted

Answer : This problem come from the Java runtime you are using. Simply run the following peace of code in a console and you will be able to run you java application :

CFLAGS="-DNDEBUG" emerge -av1 libxcb
  • Question : I have two screens and when I am launching the installer/GRUDU, java complains and the following exception is raised :
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:164)
at java.awt.Toolkit$2.run(Toolkit.java:821)
at java.security.AccessController.doPrivileged(Native Method)
at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:804)
at javax.swing.UIManager.initialize(UIManager.java:1262)
at javax.swing.UIManager.maybeInitialize(UIManager.java:1245)
at javax.swing.UIManager.getUI(UIManager.java:851)
at javax.swing.JPanel.updateUI(JPanel.java:104)
at javax.swing.JPanel.<init>(JPanel.java:64)
at javax.swing.JPanel.<init>(JPanel.java:87)
at javax.swing.JPanel.<init>(JPanel.java:95)
at diet.application.settings.SettingsPanelImplementation.
<init>(SettingsPanelImplementation.java:13)
at diet.application.settings.DIETInstallationSettingsPanel.
<init>(DIETInstallationSettingsPanel.java:32)
at diet.application.DDBApplicationConfiguration.
initializeSpecificSettingsPanelList(DDBApplicationConfiguration.java:49)
at diet.application.ApplicationConfiguration.
loadConfiguration(ApplicationConfiguration.java:165)
at diet.application.ApplicationConfiguration.
setApplicationContext(ApplicationConfiguration.java:118)
at diet.DietOffice.main(DietOffice.java:834)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at sun.awt.X11GraphicsEnvironment.
getDefaultScreenDevice(X11GraphicsEnvironment.java:178)
at sun.awt.X11.XToolkit.<clinit>(XToolkit.java:98)
... 18 more

Answer : To solve this juste launch the installer or GRUDU on the other screen (it should be the first display, under Linux something like :

> echo $DISPLAY
:0:0

Configuration files

Files used by GRUDU

GRUDU uses several configuration files saved in .diet directory which is located at the root of your home directory:

  • GRUDUApplicationProperties.xml : main configuration file containing the high level information.
  • g5K.xml : this file contains the main information for the GRID’5000 connection management.
  • g5k_cfg.xml : this file corresponds to the grid description.

For each file the main content for the GRUDU usage will be described.

GRID’5000 configuration file: GRUDUApplicationProperties.xml

<application>
    <properties name="tipOfTheDayShowOnStartup" value="false" />
    <properties name="tipOfTheDayFileOfTips" value="languages/totd/defaultGRUDUFileOfTips_eng.xml" />
    <properties name="version" value="1.1.0" />
</application>

This files defines the application wide configuration information. For the moment every elements are a properties with a name and a value. For the moment there is three properties declared in that file :

  • version : the purpose of that property is obvious, it corresponds to the version of GRUDU.
  • tipOfTheDayShowOnStartup : This property declares whether the tip of the day should (or not) be displayed on startup.
  • tipOfTheDayFileOfTips : This property defines the file from which the tips of the day should be taken.


GRID’5000 configuration file: g5k.xml

<?xml version="1.0" standalone="yes"?>
<g5k>
    <preferredAccesPoint host="acces.lyon.grid5000.fr.fr" />
    <username id="myG5KLogin" />
    <sshkey file="thePathToMySSHKeyFile" />

    <!-- G5K Sites -->

. . .

</g5k>

This file defines the global information used to log in GRID’5000. The elements that are no used by GRUDU have been removed from the description. Here are the parameters that should be supplied:

  • preferredAccesPoint : the node has an attribute named host. This attribute have to be the name of one of the access frontale of GRID’5000.
  • username : the node has an attribute names id that is the login of the user.
  • sshkey : the node has one attribute, file which is the file storing the ssh private key file.


GRID’5000 configuration file: g5k_cfg.xml

<g5k>
    <site id="Lyon" enable="false" batch_schedulers="OAR1" >
        <cluster name="Lyon--Capricorne" xda="" />
        <cluster name="Lyon--Sagittaire" xda="" />
    </site>

. . .

</g5k>

This file describes the platform of GRID’5000, the properties of the sites (id, batch_scheduler, etc) but also the clusters of the sites with their deployment partitions. Here are the parameters that should be supplied for each site:

  • id : Name of the site
  • enable : (true/false) defines whether the site is considered in the interrogation parts of GRUDU
  • batch_schedulers : Name of the batch scheduler to use.
  • For each cluster :
    • name : the name of the cluster
    • xda : the partition used be Kadeploy
Personal tools
Namespaces

Variants
Actions
Public Portal
Users Portal
Admin portal
Wiki special pages
Toolbox