Grid5000 Metadata Bundler: Difference between revisions

From Grid5000
Jump to navigation Jump to search
 
(3 intermediate revisions by one other user not shown)
Line 17: Line 17:
G5k-metadata-bundler is installed on every site frontend in Grid'5000. It can only be executed from the site frontends.
G5k-metadata-bundler is installed on every site frontend in Grid'5000. It can only be executed from the site frontends.


  g5k-metadata-bundler -s SITE -j JOBID [-o OUTPUT]
  g5k-metadata-bundler -s SITE -j JOBID [-o NAME]
   -v, --version                    Print g5k-metadata-bundler version
   -v, --version                    Print g5k-metadata-bundler version
   -s, --job-site SITE              [MANDATORY] Grid'5000 site from which to extract
   -s, --job-site SITE              [MANDATORY] Grid'5000 site from which to extract
Line 75: Line 75:
*: This can be used to look up information about nodes not directly used by the bundled oar jobs.
*: This can be used to look up information about nodes not directly used by the bundled oar jobs.
*: The resources files are a subset of this file.
*: The resources files are a subset of this file.
* '''g5k-monitoring-SITE-NODE-JID.json""": Monitoring files
* '''g5k-monitoring-SITE-NODE-JID.json''': Monitoring files
*: Contains all the monitoring measurements made by [[Monitoring_Using_Kwollect|Kwollect]] for a NODE during the oar job JID.
*: Contains all the monitoring measurements made by [[Monitoring_Using_Kwollect|Kwollect]] for a NODE during the oar job JID.
*: The contents will vary depending on how much monitoring was enabled for a given job. See default metrics in [[Monitoring_Using_Kwollect#Metrics_available_in_Grid.275000|Monitoring_Using_Kwollect]].
*: The contents will vary depending on how much monitoring was enabled for a given job. See default metrics in [[Monitoring_Using_Kwollect#Metrics_available_in_Grid.275000|Monitoring_Using_Kwollect]].
Line 87: Line 87:
*:* List of files included in the bundle with a short description
*:* List of files included in the bundle with a short description


User will also find at the end of every file a small bundler info segment. This segment contains the date at which the file was generated, warnings raise by the file generation and a list of references indicating how this file relates to other files in the bundle.
User will also find at the end of every file a small bundler info segment. This segment contains the date at which the file was generated, warnings raised by the file generation and a list of references indicating how this file relates to other files in the bundle.


= Planned Evolutions =
= Planned Evolutions =


As the blunder is still in alpha version, we welcome comments and feature request.
As the bundler is still in alpha version, we welcome comments and feature requests.


The following is a list of feature we are already looking at:
The following is a list of features we are already looking at:
*; Concerning bundle contents
*; Concerning bundle contents
** Bundling multiple jobs in a single archive
** Bundling multiple jobs in a single archive
** Bundling based on job names
** Bundling based on job names
** Better management of monitoring information when it is to big for download
** Better management of monitoring information when it is too big for download
** Adding information concerning the standard environment to the bundle
** Adding information concerning the standard environment to the bundle
** Adding image deployments to the bundle
** Adding image deployments to the bundle

Latest revision as of 10:27, 22 July 2021

Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

This page summarize what you need to know about g5k-metadata-bundler.

Introduction

When running experiments on Grid'5000, users generate metadata across multiple services. This metadata is useful for reproducibility purposes or scientific dissemination. The g5k-metadata-bundler is a tool designed to retrieve metadata across all the different services and bundle them in a single archive. The bundle only retrieves metadata generated by Grid'5000 services, the collection of data generated by the users experiment is beyond the scope of this application.

Warning.png Warning

This service is in alpha version and not yet feature complete. See the section on Planned_Evolutions.

Usage

G5k-metadata-bundler is installed on every site frontend in Grid'5000. It can only be executed from the site frontends.

g5k-metadata-bundler -s SITE -j JOBID [-o NAME]
  -v, --version                    Print g5k-metadata-bundler version
  -s, --job-site SITE              [MANDATORY] Grid'5000 site from which to extract
  -j, --job-id JID                 [MANDATORY] Job id of the OAR jod to extract
  -o, --output NAME                Bundle name to use for the directory/archive

Users do not need to operate the bundler on the same frontend as the site the jobs was executed on. The bundler download all data pertaining to the queried job and bundle in a archive named g5k-bundle-SITE-JID.tar.gz or if an output name has been provided NAME.tar.gz. The bundle is provided in as a tar.gz archive which can be manipulated by using the following commands:

  • Listing
    tar -tzf NAME.tar.gz lists all files contained within the bundle
    Extraction
    tar -xzf NAME.tar.gz extracts all files to a directory with the same name as the bundle

Users operating on older versions of Windows might require thrid party software to unpack the bundle. (often 7-zip)

Example usage

user@fsophia:~$ g5k-metadata-bundler -s nancy -j 3003030
 Running g5k-metadata-bundler for job 3003030 at nancy
 Downloading https://api.grid5000.fr/stable/sites/nancy/jobs/3003030
 Downloading https://api.grid5000.fr/stable/sites/nancy/clusters/graoully/nodes/graoully-1?version=7f6b81c2621c6ed3a4fac632f213436813495755
 Downloading https://api.grid5000.fr/stable/?version=7f6b81c2621c6ed3a4fac632f213436813495755&deep=true
 Downloading https://api.grid5000.fr/stable/sites/nancy/metrics?job_id=3003030&nodes=graoully-1
 Generating README
 Compressing bundle
 Bundle created at g5k-bundle-nancy-3003030.tar.gz
user@fsophia:~$ ls -lh g5k-bundle-nancy-3003030.tar.gz
 -rw-r--r-- 1 user g5k-users 456K Jul 19 09:50 g5k-bundle-nancy-3003030.tar.gz
user@fsophia:~$ tar -tzf g5k-bundle-nancy-3003030.tar.gz
 g5k-bundle-nancy-3003030/
 g5k-bundle-nancy-3003030/g5k-oarjob-nancy-3003030.json
 g5k-bundle-nancy-3003030/README
 g5k-bundle-nancy-3003030/g5k-resource-nancy-graoully-1-7f6b81c2621c6ed3a4fac632f213436813495755.json
 g5k-bundle-nancy-3003030/g5k-monitoring-nancy-graoully-1-3003030.json
 g5k-bundle-nancy-3003030/g5k-refapi-7f6b81c2621c6ed3a4fac632f213436813495755.json

Bundle contents

The bundle contains these different files types:

  • g5k-oarjob-SITE-JID.json: Job files
    Contains the information for a given OAR job JID at Grid'5000 site SITE such as:
    • submission, start, and end dates
    • user and group (group granting access) of the job
    • job types and properties
    • command executed by the job
    • list of resources attributed to the job
    • OAR events for the job
    This information is extracted from the jobs API
  • g5k-resource-SITE-NODE-VERSION.json: Resource files
    Contains information about a single NODE, such as:
    • Node architecture, bios, ram, and cpu information
    • network, storage, and monitoring devices
    • base configuration information
    The bundle will contain one such file for each of the nodes involved in a job.
    This information is extracted from the reference API
    The VERSION of the information contained in this file will match what it was on the day the job was executed.
  • g5k-refapi-VERSION.json: Reference API files
    Contains a full copy of the reference API at VERSION
    This can be used to look up information about nodes not directly used by the bundled oar jobs.
    The resources files are a subset of this file.
  • g5k-monitoring-SITE-NODE-JID.json: Monitoring files
    Contains all the monitoring measurements made by Kwollect for a NODE during the oar job JID.
    The contents will vary depending on how much monitoring was enabled for a given job. See default metrics in Monitoring_Using_Kwollect.
    Often the heaviest files in the bundle
    This information is extracted from Kwollect
  • README: The readme file
    Contains information pertaining to the execution of the bundler such as:
    • Bundler version
    • Execution date
    • List of warnings and errors that happened during bundling
    • List of files included in the bundle with a short description

User will also find at the end of every file a small bundler info segment. This segment contains the date at which the file was generated, warnings raised by the file generation and a list of references indicating how this file relates to other files in the bundle.

Planned Evolutions

As the bundler is still in alpha version, we welcome comments and feature requests.

The following is a list of features we are already looking at:

  • Concerning bundle contents
    • Bundling multiple jobs in a single archive
    • Bundling based on job names
    • Better management of monitoring information when it is too big for download
    • Adding information concerning the standard environment to the bundle
    • Adding image deployments to the bundle
      • Adding information concerning the deployed images
    Concerning bundler operation
    • No-compress mode where the bundle is left as a directory containing all files
    • Appending new files to an existing bundle
    • Reduce memory footprint
    • Assess viability of parallel downloads

Open questions

Additionally, we would welcome feedback concerning the following open questions:

  • What format should be used inside the bundle ? Are json files sufficient or should we convert files to a structured XML ?
  • Should we keep bundler segments at the end of every file ? Should we move all cross references to a separate file ? Not have cross references at all ?