Grid5000:School2009 TPs: Difference between revisions
(24 intermediate revisions by 8 users not shown) | |||
Line 1: | Line 1: | ||
Practical session will give participants a hands on experience of tools available for Grid'5000. Some sessions will be based on the online tutorials provided and maintained by the technical staff, and some sessions will present tools or environments developed and maintained by users. | Practical session will give participants a hands on experience of tools available for Grid'5000. Some sessions will be based on the online tutorials provided and maintained by the technical staff, and some sessions will present tools or environments developed and maintained by users. | ||
Participants to the School should register for the practical session they intend to take part in [https://helpdesk.grid5000.fr/g5kschool09/ here] | {{Warning|text=Participants to the School should register for the practical session they intend to take part in [https://helpdesk.grid5000.fr/g5kschool09/ here]}} | ||
{{Note|text=Details on [[School 2009 labs|computer access and configuration]] is accessible for participants with a Grid'5000 account}} | |||
__TOC__ | __TOC__ | ||
== Talk : Introduction to Grid'5000 == | == Talk: Introduction to Grid'5000 == | ||
This talk is intended as a general introduction of Grid'5000. People who have never used Grid'5000 should register for this talk to get an overview of the instrument, the concepts for its use and an overview of the available tools. | This talk is intended as a general introduction of Grid'5000. People who have never used Grid'5000 should register for this talk to get an overview of the instrument, the concepts for its use and an overview of the available tools. | ||
Line 15: | Line 15: | ||
=== Information === | === Information === | ||
* Contact author: Aurélien Cedeyn | * Contact author: Aurélien Cedeyn | ||
* Site used for developpment: | * Site used for developpment: Lyon/Orsay | ||
* Site restrictions during the school: none | * Site restrictions during the school: none | ||
* Tester : David Loureiro | * Tester : David Loureiro | ||
Line 23: | Line 23: | ||
** ssh to connect to Grid'5000 | ** ssh to connect to Grid'5000 | ||
== [ | == [[g5kss09_GRUDU_TP|GRUDU: Grid Reservation Utility for Deployment Usage]] == | ||
=== Description === | === Description === | ||
In a grid environment such as Grid’5000, we need several complex tools for the management of | In a grid environment such as Grid’5000, we need several complex tools for the management of resources, the allocation of resources to the job or even for the deployment of Kadeploy images. Most grid software systems use command-line interfaces without any Graphical User Interface (GUI). For the creation of a tool dedicated to the management of grid environments, different functions are mandatory. We can consider three main graphical interfaces for such framework: one for resource monitoring, one for resources allocation and one for distant usage (terminal and file access). [[GRUDU|GRUDU]] answers the need of an unified set of tools providing the user with a complete, modular, portable, and powerful way to manage grid resources. These three main parts of the software will be the heart of the Practical Session where we will explore the capabilities of [[GRUDU|GRUDU]] in a common usage of Grid’5000 (The software must of course first be installed and configured, and a correct ssh configuration must be setup on Grid’5000). | ||
=== Information === | === Information === | ||
* Contact author: David Loureiro | * Contact author: David Loureiro | ||
* Site used for developpment: | * Site used for developpment: Lyon | ||
* Site restrictions during the school: none | * Site restrictions during the school: none | ||
* Tester : | * Tester : Benjamin Depardon | ||
* Site used for tests: | * Site used for tests: Lyon | ||
* Duration : | * Duration : 2 hours | ||
* Prerequisites on the user's machines : | * Prerequisites on the user's machines : | ||
** JAVA (1.5.0 or higher) | ** JAVA (1.5.0 or higher) | ||
Line 49: | Line 49: | ||
** ssh to connect to Grid'5000 | ** ssh to connect to Grid'5000 | ||
== | == First steps on Grid'5000 == | ||
=== Description === | === Description === | ||
The practical session will focus on discovering the Grid'5000 infrastructure. At the end of the practical session, practitioners should be familiar with the different sources of information available to Grid'5000 users, and should be able to connect to all Grid'5000 sites. | The practical session will focus on discovering the Grid'5000 infrastructure. At the end of the practical session, practitioners should be familiar with the different sources of information available to Grid'5000 users, and should be able to connect to all Grid'5000 sites. This practical session will follow two steps: | ||
* [[Understanding_Grid5000|Understanding Grid'5000]] and | |||
* [[First_steps|First steps]] | |||
=== Information === | === Information === | ||
* Contact author: Benjamin Dexheimer | * Contact author: Benjamin Dexheimer | ||
* Site used for developpment: | * Site used for developpment: | ||
* Site restrictions during the school: none | * Site restrictions during the school: none | ||
* Tester : | * Tester : Romain Cavagna | ||
* Site used for tests: | * Site used for tests: Grenoble | ||
* Duration : | * Duration : About 2 hours, depending on Grid'5000 knowledges | ||
* Prerequisites on the user's machines : | * Prerequisites on the user's machines : | ||
** web browser | ** web browser | ||
Line 65: | Line 67: | ||
== [[Cluster_experiment|Resource allocation with OAR]] and [[Grid_experiment|with OARGRID]] == | == [[Cluster_experiment|Resource allocation with OAR]] and [[Grid_experiment|with OARGRID]] == | ||
=== Description === | === Description === | ||
This practice is about running jobs on a cluster. You will learn how to access a Grid'5000 cluster, how to install your data and how to run your jobs and visualize them. | This practice is about running jobs on a cluster. You will learn how to access a Grid'5000 cluster, how to install your data and how to run your jobs and visualize them. This is a two step practical session: | ||
* [[Cluster_experiment|Resource allocation with OAR]] and | |||
* [[Grid_experiment|with OARGRID]] | |||
=== Information === | === Information === | ||
* Contact author: Romain Cavagna | * Contact author: Romain Cavagna | ||
Line 93: | Line 97: | ||
** ideally, ruby v1.8.6 + irb + rubygems (with restclient gem) + libopenssl-ruby1.8 + libjson-ruby1.8 + python v2.4 + python-httplib2 + python-simplejson | ** ideally, ruby v1.8.6 + irb + rubygems (with restclient gem) + libopenssl-ruby1.8 + libjson-ruby1.8 + python v2.4 + python-httplib2 + python-simplejson | ||
== Deployment of [[FreeBSD]] using Kadeploy == | == Deployment of [[Non-Linux_system_deployment|FreeBSD]] using Kadeploy == | ||
=== Description === | === Description === | ||
While most users of Grid'5000 use Linux on the compute nodes, Kadeploy also allows to deploy other systems, by recording or writing all the data contained in a disk partition. In this practical session, you will learn more about Kadeploy internals, and, using FreeBSD as an example, you will deploy and record non-linux environments. A tour of recent FreeBSD enhancements, imported from Solaris (DTrace, ZFS) will also be provided. | While most users of Grid'5000 use Linux on the compute nodes, Kadeploy also allows to deploy other systems, by recording or writing all the data contained in a disk partition. In this practical session, you will learn more about Kadeploy internals, and, using FreeBSD as an example, you will deploy and record non-linux environments. A tour of recent FreeBSD enhancements, imported from Solaris (DTrace, ZFS) will also be provided. | ||
Line 107: | Line 111: | ||
** ssh to connect to Grid'5000 | ** ssh to connect to Grid'5000 | ||
== Running MPI on Grid'5000 | == [[Run MPI On Grid'5000|Running MPI on Grid'5000]] == | ||
=== Description === | === Description === | ||
In this practical session, we will explore different techniques to run mpi jobs on Grid'5000, focusing on automatic configuration of the runtime environment (openMPI, MPICH2) using the <code class="var">$OAR_NODE_FILE</code> | In this practical session, we will explore different techniques to run mpi jobs on Grid'5000, focusing on automatic configuration of the runtime environment (openMPI, MPICH2) using the <code class="var">$OAR_NODE_FILE</code> | ||
Line 118: | Line 122: | ||
* Duration : | * Duration : | ||
* Prerequisites on the user's machines : | * Prerequisites on the user's machines : | ||
** ssh to connect to Grid'5000 | ** ssh to connect to Grid'5000 | ||
== DIET Introductory Hands-On == | == [[g5kss09_DIET_TP|DIET Introductory Hands-On]] == | ||
=== Description === | === Description === | ||
The Distributed Interactive Engineering Toolbox | |||
([http://graal.ens-lyon.fr/DIET <span style='font-variant: small-caps;'>Diet</span>]) project is focused on the development of a scalable middleware with initial efforts focused on distributing the scheduling problem across multiple agents. This middleware is able to find an appropriate server according to the information given in the client's request (''e.g.,'' problem to be solved, size of the data involved), the performance of the target platform (''e.g.,'' server load, available memory, communication performance) and the local availability of data stored during previous computations. | |||
This practical session aims at presenting three aspects grid computing with <span style='font-variant: small-caps;'>Diet</span>: | |||
; <span style='font-variant: small-caps;'>Diet</span> basis | |||
: We first present how requests are handled by <span style='font-variant: small-caps;'>Diet</span>, through simple examples presenting the required basis to write a <span style='font-variant: small-caps;'>Diet</span> client/server for handling matrices multiplication. We will see how to: | |||
* <span style='font-variant: small-caps;'>Diet</span> installation. | |||
* We first write a client/server which send simple data types (double, matrix) within <span style='font-variant: small-caps;'>Diet</span>, and compute a scalar by matrix product. | |||
* Then, instead of relying on a matrix stored in memory, we send it using a file. | |||
* We also create a new service relying on the BLAS Library, in order to deal with matrix computation of the type [[Image:diet_tuto_eq_dgemm.png]]. | |||
* Finally, we see how to deploy a <span style='font-variant: small-caps;'>Diet</span> hierarchy, either by hand, or by using <span style='font-variant: small-caps;'>GoDiet</span>, the <span style='font-variant: small-caps;'>Diet</span> deployment tool. | |||
; Advanced data management | |||
: We introduce data management within <span style='font-variant: small-caps;'>Diet</span> application using the <span style='font-variant: small-caps;'>Dagda</span> API : | |||
* We start by introducing simple data management using <span style='font-variant: small-caps;'>Dagda</span>. | |||
* Then, we see how to perform these data management operations in asynchronous mode. | |||
* We show how to explicitely perform data replications on the different nodes of the <span style='font-variant: small-caps;'>Diet</span> hierarchy. | |||
* Finally, we use all these <span style='font-variant: small-caps;'>Dagda</span> features to make a <span style='font-variant: small-caps;'>Diet</span> application using some data management optimizations. | |||
; Workflow management | |||
: We present how <span style='font-variant: small-caps;'>Diet</span> handles workflows execution. | |||
* We start by running a simple DAG (Direct Acyclic Graph) of tasks. | |||
* Then we will submit multiple workflows, and modify the scheduling policy. | |||
* We also see how to display the task schedule using VizDiet | |||
=== Information === | === Information === | ||
* Contact author: eddy.caron@ens-lyon.fr | * Contact author: benjamin.depardon@ens-lyon.fr;eddy.caron@ens-lyon.fr | ||
* Site used for developpment: | * Site used for developpment: | ||
* Site restrictions during the school: | * Site restrictions during the school: none | ||
* Tester : | * Tester : david.loureiro@ens-lyon.fr | ||
* Site used for tests: | * Site used for tests: all can be used but Grenoble and Lille | ||
* Duration : | * Duration: | ||
* Prerequisites on the user's machines : | * Prerequisites on the user's machines: | ||
** ssh to connect to Grid'5000 | ** ssh to connect to Grid'5000 | ||
** | ** None as we will deploy an image containing all required dependencies for the tutorial | ||
[[Grid5000:School2009|back to main page]] | [[Grid5000:School2009|back to main page]] |
Latest revision as of 09:58, 17 April 2009
Practical session will give participants a hands on experience of tools available for Grid'5000. Some sessions will be based on the online tutorials provided and maintained by the technical staff, and some sessions will present tools or environments developed and maintained by users.
Warning | |
---|---|
Participants to the School should register for the practical session they intend to take part in here |
Note | |
---|---|
Details on computer access and configuration is accessible for participants with a Grid'5000 account |
Talk: Introduction to Grid'5000
This talk is intended as a general introduction of Grid'5000. People who have never used Grid'5000 should register for this talk to get an overview of the instrument, the concepts for its use and an overview of the available tools.
Customizing Grid'5000 with Kadeploy
Description
In this practical session, you will learn to use Kadeploy to deploy a preconfigured environment, customize it, register the customize version and script its deployment.
Information
- Contact author: Aurélien Cedeyn
- Site used for developpment: Lyon/Orsay
- Site restrictions during the school: none
- Tester : David Loureiro
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- ssh to connect to Grid'5000
GRUDU: Grid Reservation Utility for Deployment Usage
Description
In a grid environment such as Grid’5000, we need several complex tools for the management of resources, the allocation of resources to the job or even for the deployment of Kadeploy images. Most grid software systems use command-line interfaces without any Graphical User Interface (GUI). For the creation of a tool dedicated to the management of grid environments, different functions are mandatory. We can consider three main graphical interfaces for such framework: one for resource monitoring, one for resources allocation and one for distant usage (terminal and file access). GRUDU answers the need of an unified set of tools providing the user with a complete, modular, portable, and powerful way to manage grid resources. These three main parts of the software will be the heart of the Practical Session where we will explore the capabilities of GRUDU in a common usage of Grid’5000 (The software must of course first be installed and configured, and a correct ssh configuration must be setup on Grid’5000).
Information
- Contact author: David Loureiro
- Site used for developpment: Lyon
- Site restrictions during the school: none
- Tester : Benjamin Depardon
- Site used for tests: Lyon
- Duration : 2 hours
- Prerequisites on the user's machines :
- JAVA (1.5.0 or higher)
Deploying virtual machines with Kadeploy
Description
In this practical session, you will learn to deploy and configure Xen hypervisors and virtual machines on Grid'5000. You will also learn the basic tools and environment provided by Grid'5000 to ease the use of virtualization technology in Grid'5000 : MAC adress generation, routed IPs availbale to virtual machines, DNS for virtual machines, etc.
Information
- Contact author: Cyril Constantin
- Site used for developpment: Nancy
- Site restrictions during the school: only valid in Nancy
- Tester : Emmanuel Jeanvoine
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- ssh to connect to Grid'5000
First steps on Grid'5000
Description
The practical session will focus on discovering the Grid'5000 infrastructure. At the end of the practical session, practitioners should be familiar with the different sources of information available to Grid'5000 users, and should be able to connect to all Grid'5000 sites. This practical session will follow two steps:
Information
- Contact author: Benjamin Dexheimer
- Site used for developpment:
- Site restrictions during the school: none
- Tester : Romain Cavagna
- Site used for tests: Grenoble
- Duration : About 2 hours, depending on Grid'5000 knowledges
- Prerequisites on the user's machines :
- web browser
- ssh to connect to Grid'5000
Resource allocation with OAR and with OARGRID
Description
This practice is about running jobs on a cluster. You will learn how to access a Grid'5000 cluster, how to install your data and how to run your jobs and visualize them. This is a two step practical session:
Information
- Contact author: Romain Cavagna
- Site used for developpment: Nancy, Rennes and Lyon
- Site restrictions during the school: none
- Tester :
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- web browser
- ssh to connect to Grid'5000
Discovering Grid'5000's API
Description
Beginning with the ReferenceAPI tutorial, you will learn how to access the current description of the platform: the sites, clusters, nodes, etc. as well as the description of the platform as it was at a time in the past. We will also show you how to get the latest changes brought on a specific resource, and how to select nodes based on their properties. The MonitoringAPI tutorial builds on the first tutorial and introduces the MonitoringAPI, which returns the status of a specific Grid5000 resource so that you can monitor them or make decisions based on the availability of resources. In particular, you'll learn how to select nodes that match specific criteria in terms of properties and availability.
Information
- Contact author: Cyril Rohr and Bruno Bzeznik
- Site used for development: Rennes and Grenoble
- Site restrictions during the school: proxies only installed in Rennes and Bordeaux, but usable from all sites
- Tester : Pascal Morillon
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- web browser
- ssh to connect to Grid'5000
- curl with SSL support
- ideally, ruby v1.8.6 + irb + rubygems (with restclient gem) + libopenssl-ruby1.8 + libjson-ruby1.8 + python v2.4 + python-httplib2 + python-simplejson
Deployment of FreeBSD using Kadeploy
Description
While most users of Grid'5000 use Linux on the compute nodes, Kadeploy also allows to deploy other systems, by recording or writing all the data contained in a disk partition. In this practical session, you will learn more about Kadeploy internals, and, using FreeBSD as an example, you will deploy and record non-linux environments. A tour of recent FreeBSD enhancements, imported from Solaris (DTrace, ZFS) will also be provided.
Information
- Contact author: Lucas Nussbaum
- Site used for developpment: Orsay
- Site restrictions during the school: needs grub
- Tester :
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- web browser
- ssh to connect to Grid'5000
Running MPI on Grid'5000
Description
In this practical session, we will explore different techniques to run mpi jobs on Grid'5000, focusing on automatic configuration of the runtime environment (openMPI, MPICH2) using the $OAR_NODE_FILE
Information
- Contact author: Nicolas Niclausse
- Site used for developpment:
- Site restrictions during the school: none
- Tester :
- Site used for tests:
- Duration :
- Prerequisites on the user's machines :
- ssh to connect to Grid'5000
DIET Introductory Hands-On
Description
The Distributed Interactive Engineering Toolbox (Diet) project is focused on the development of a scalable middleware with initial efforts focused on distributing the scheduling problem across multiple agents. This middleware is able to find an appropriate server according to the information given in the client's request (e.g., problem to be solved, size of the data involved), the performance of the target platform (e.g., server load, available memory, communication performance) and the local availability of data stored during previous computations.
This practical session aims at presenting three aspects grid computing with Diet:
- Diet basis
- We first present how requests are handled by Diet, through simple examples presenting the required basis to write a Diet client/server for handling matrices multiplication. We will see how to:
- Diet installation.
- We first write a client/server which send simple data types (double, matrix) within Diet, and compute a scalar by matrix product.
- Then, instead of relying on a matrix stored in memory, we send it using a file.
- We also create a new service relying on the BLAS Library, in order to deal with matrix computation of the type .
- Finally, we see how to deploy a Diet hierarchy, either by hand, or by using GoDiet, the Diet deployment tool.
- Advanced data management
- We introduce data management within Diet application using the Dagda API :
- We start by introducing simple data management using Dagda.
- Then, we see how to perform these data management operations in asynchronous mode.
- We show how to explicitely perform data replications on the different nodes of the Diet hierarchy.
- Finally, we use all these Dagda features to make a Diet application using some data management optimizations.
- Workflow management
- We present how Diet handles workflows execution.
- We start by running a simple DAG (Direct Acyclic Graph) of tasks.
- Then we will submit multiple workflows, and modify the scheduling policy.
- We also see how to display the task schedule using VizDiet
Information
- Contact author: benjamin.depardon@ens-lyon.fr;eddy.caron@ens-lyon.fr
- Site used for developpment:
- Site restrictions during the school: none
- Tester : david.loureiro@ens-lyon.fr
- Site used for tests: all can be used but Grenoble and Lille
- Duration:
- Prerequisites on the user's machines:
- ssh to connect to Grid'5000
- None as we will deploy an image containing all required dependencies for the tutorial