Monitoring Using Kwollect: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Portal|User}}
{{Portal|Tutorial}}
{{TutorialHeader}}
This page describes the monitoring service available in Grid’5000 that uses [https://gitlab.inria.fr/grid5000/kwollect Kwollect] to retrieve environmental and performance metrics from nodes.
This page describes the monitoring service available in Grid’5000 that uses [https://gitlab.inria.fr/grid5000/kwollect Kwollect] to retrieve environmental and performance metrics from nodes.


The service currently provides metrics for:
The service currently provides metrics for:
* Energy consumption from dedicated “wattmetre” devices (currently available for some clusters in Lyon, Grenoble, Nancy)
* Energy consumption from dedicated “wattmetre” devices (currently available for some clusters in Lyon, Grenoble, Nancy)
* Metrics collected from nodes’ Board Management Controller (out-of-band management hardware, such as Dell iDRAC), such as ambient temperature, hardware component temperature, energy consumption from PSU, fan speed, etc.
* Metrics collected from nodes’ Board Management Controller ("BMC": out-of-band management hardware, such as Dell iDRAC), such as ambient temperature, hardware component temperature, energy consumption from PSU, fan speed, etc.
* Trafic collected from network devices
* Traffic collected from network devices
* Energy consumption from PDU, when available
* Energy consumption from PDU, when available
* Node metrics from Prometheus node exporter (and Nvidia DCGM exporter when GPU is available)


{{Warning| text=Monitoring with Kwollect under Grid'5000 is still in beta phase. It uses "sid" branch of the API, while [[Monitoring_deployed_nodes|legacy monitoring API]] (based on Ganglia and Kwapi) still uses the "stable" branch. Kwollect is intended to replace the legacy system in the future}}
{{Warning| text=Monitoring with Kwollect under Grid'5000 is still in beta phase. It uses "sid" branch of the API, while [[Monitoring_deployed_nodes|legacy monitoring API]] (based on Ganglia and Kwapi) still uses the "stable" branch. Kwollect is intended to replace the legacy system in the future}}


= Metrics available =
= Metrics available =
Line 15: Line 21:
  $ curl https://api.grid5000.fr/stable/sites/lyon/clusters/taurus | jq .metrics
  $ curl https://api.grid5000.fr/stable/sites/lyon/clusters/taurus | jq .metrics


This return a list where each element describes a single metric. Most important fields of that description are:
This returns a list where each element describes a single metric. Most important fields of that description are:
* ''name'': The name identifying the metric
* ''name'': The name identifying the metric
* ''description'': A human-readable description of the metric
* ''description'': A human-readable description of the metric
* ''labels'' (optional): A “label” that will be used to distinguish two metrics of the same kind, but targeting different objects (e.g. temperature of CPU #1 vs temperature of CPU #2)
* ''labels'' (optional): A “label” that will be used to distinguish two metrics of the same kind, but targeting different objects (e.g. temperature of CPU #1 vs temperature of CPU #2)
* ''period'' and ''optional_period'': The interval (in milli-seconds) under which the metric is collected. The former is the default interval, the latter is the interval used when “Full monitoring mode” is in use (see below). A metric with ''period'' value at 0 is not collected by default.
* ''period'' and ''optional_period'': The interval (in milli-seconds) under which the metric is collected. The former is the default interval, the latter is the interval used when the metric is activated “on demand” (see below). A metric with ''period'' value at 0 is not collected by default.


“Full monitoring mode” can be enabled for specific jobs by adding the “-t monitor” option to oarsub. Non-default metrics will be available for nodes belonging to this job. E.g:
$ oarsub -I -t monitor
''Note'': Dedicated wattmetre devices (metric “wattmetre_power_watt”) are able to perform one measurement every 20 milli-seconds. However, this high frequency is only provided under “Full monitoring mode”. By default, only the value averaged over one second is provided.


= Getting metrics values =
= Getting metrics values =


The metrics values are stored by Kwollect and available using the Metrology API (under “sid” version) by performing a GET request at URL:
The metrics values are stored by Kwollect and available using the Metrology API (under “sid” version) by performing a GET request with appropriate parameters at URL:


<pre>https://api.grid5000.fr/sid/sites/<site>/metrics</pre>
<pre>https://api.grid5000.fr/sid/sites/<site>/metrics</pre>
The following parameters are supported:
* ''nodes'': The list of nodes, separated by ‘,’ (comma), on which to obtain values
* ''start_time'': The time from which to obtain the values. By default it is one hour before the current time.
* ''stop_time'': The time until which to obtain the values. By default it is the current time.
* ''job_id'': A job identifier. This parameter is an alternative to the previous three as it will provide nodes list, start and stop time according to job characteristics.
* ''metrics'': The list of metrics name, separated by ‘,’ (comma), on which to obtain values (metrics name are described in Reference API, see above). By default all metrics are returned.


For instance, to:
For instance, to:
* get all metrics collected inside job 12345 at Lyon:
* get all metrics collected inside job 12345 at Lyon:


<pre>curl 'https://api.grid5000.fr/sid/sites/lyon/metrics?job_id=12345'</pre>
<pre>curl 'https://api.grid5000.fr/sid/sites/lyon/metrics?job_id=1157330'</pre>


* get all metrics from chifflot-5 and chifflot-6 between 2020-06-08 15:00 and 2020-06-08 17:00:
* get all metrics from chifflot-5 and chifflot-6 between 2020-06-08 15:00 and 2020-06-08 17:00:


<pre>curl 'https://api.grid5000.fr/sid/sites/lille/metrics?nodes=chifflot-5,chifflot-6&start_time=2020-06-08T15:00&stop_time=2020-06-08T17:00'</pre>
<pre>curl 'https://api.grid5000.fr/sid/sites/lille/metrics?nodes=chifflot-5,chifflot-6&start_time=2020-06-08T15:00&end_time=2020-06-08T17:00'</pre>


* get all values from Wattmetre for taurus-2, during last 15 minutes:
* get all values from Wattmetre for taurus-2, during last 15 minutes:
Line 53: Line 47:
<pre>curl "https://api.grid5000.fr/sid/sites/lyon/metrics?nodes=taurus-2&metrics=wattmetre_power_watt&start_time=$(date -d '15 min ago' +%s)"</pre>
<pre>curl "https://api.grid5000.fr/sid/sites/lyon/metrics?nodes=taurus-2&metrics=wattmetre_power_watt&start_time=$(date -d '15 min ago' +%s)"</pre>


The request will return a JSON-formatted list of metrics values and their associated timestamp.
For a complete description of parameters and returned fields, see the API specification at: https://api.grid5000.fr/doc/sid/#tag/metrics
'''Important note:''' To avoid overloading Kwollect servers, request duration is limited to 5 minutes. When too much metrics values are requested, you may hit that limit and receive an error message. In such case, try to reduce the size of your request by selecting less nodes or a shorter time period.
'''Tip''': If you need metrics formatted as CSV, you can use this command line :
curl <URL_to_request_some_metrics> | jq -r '.[] | [.timestamp, .device_id, .metric_id, .value] | @csv'
= On-demand metrics =
“On-demand” metrics can be enabled for specific jobs by adding the “-t monitor=<metric_to_enable>” option to oarsub. E.g:
$ oarsub -I -t monitor=bmc_cpu_temp_celsius
The provided argument can be a regular expression For instance, to enable all metrics related to temperature:
$ oarsub -I -t monitor='.*temp.*'
If no argument is provided, all “On-demand” metrics available are enabled.
'''Note''':
* Dedicated wattmetre devices (metric “wattmetre_power_watt”) are able to perform one measurement every 20 milli-seconds. However, this high frequency is only provided using “on demand” activation. For instance, submit your job using:
$ oarsub -I -t monitor='wattmetre_power_watt'
By default, only the value averaged over one second is provided.
* As prometheus metrics depend on node characteristics, they cannot be fully described. Only a subset of prometheus metrics will be collected by default (described in the API by the ''prom_default_metrics'' and ''prom_nvgpu_default_metrics'', when relevant). To enable collecting all prometheus metrics, use "on-demand" activation on ''prom_all_metrics'' or ''prom_nvgpu_all_metrics'' (for instance, use <code>monitor='prom_.*'</code>)
= Visualization Dashboard =
A simple dashboard to visualize the metrics is also available. Metrics can be displayed by job ID or by date and graphics can be updated in real time with new values.
Dashboards are available at:
<pre>https://api.grid5000.fr/sid/sites/<site>/metrics/dashboard</pre>
For instance, at Lyon: https://api.grid5000.fr/sid/sites/lyon/metrics/dashboard
= Pushing custom metrics =
A simple mechanism is available to let you push your own, arbitrary, custom metrics to Kwollect (internally it uses [https://github.com/prometheus/node_exporter#textfile-collector Prometheus Node Exporter "Textfile Collector"]). From a node, a custom metric will be collected by writing to a specific file (you currently need ''root'' privileges to do so):
$ echo 'kwollect_custom{_metric_id="my_metric"} 42' > /var/lib/prometheus/node-exporter/kwollect.prom
This will push a custom metric named ''my_metric'' and with value "42".
The associated timestamp will be the time when Kwollect fetches metrics on the node. You may override the timestamp by adding ''_timestamp="your_timestamp_in_unix_seconds"''. For instance:
$ echo 'kwollect_custom{_metric_id="my_metric", _timestamp="1606389005.1234"} 42' > /var/lib/prometheus/node-exporter/kwollect.prom
(You can also add your own labels. See Prometheus metrics [https://prometheus.io/docs/practices/naming/ naming] and [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md format] for more information).


The request will return a JSON-formatted list of values, each containing following information:
* ''timestamp'': The time when the value has been collected
* ''device_id'': The identifier of the device (such as the node name) where the value has been collected
* ''metric_id'': The name of the metric for that value
* ''value'': The value collected for that metric
* ''labels'': Some optional additional information (see labels description above)


Kwollect will continuously pull values from <code>/var/lib/prometheus/node-exporter/kwollect.prom</code> file. It is your responsibility to keep its content updated.


= Known problems =
= Known problems =


* Metrics from BMC are quite unreliable. They may not be available on some nodes from time to time.
* Metrics from BMC are quite unreliable. They may be inaccurate, or unavailable on some nodes from time to time.
* Some metrics are not available for every nodes of a cluster : For grcinq and hercule clusters, one every four nodes have more metrics available than the others ; on parasilo and paravance clusters, the measurement of the power consumption by PDUs is only available on some nodes
* Some metrics are not available for every nodes of a cluster: For grcinq and hercule clusters, one every four nodes have more metrics available than the others ; on parasilo and paravance clusters, the measurement of the power consumption by PDUs is only available on some nodes
* It may happen that few values of a metric are not collected quickly enough to comply with interval described in the Reference API (for instance, when the targeted device is overloaded).
* It may happen that few values of a metric are not collected quickly enough to comply with interval described in the Reference API (for instance, when the targeted device is overloaded).
* Electrical consumption reported by PDU are not always reliable, see [[Power_Monitoring_Devices#measurement_artifacts_and_pitfalls]]
* Electrical consumption reported by PDU is not always reliable, see [[Power_Monitoring_Devices#measurement_artifacts_and_pitfalls]]

Revision as of 15:06, 11 January 2021

Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

This page describes the monitoring service available in Grid’5000 that uses Kwollect to retrieve environmental and performance metrics from nodes.

The service currently provides metrics for:

  • Energy consumption from dedicated “wattmetre” devices (currently available for some clusters in Lyon, Grenoble, Nancy)
  • Metrics collected from nodes’ Board Management Controller ("BMC": out-of-band management hardware, such as Dell iDRAC), such as ambient temperature, hardware component temperature, energy consumption from PSU, fan speed, etc.
  • Traffic collected from network devices
  • Energy consumption from PDU, when available
  • Node metrics from Prometheus node exporter (and Nvidia DCGM exporter when GPU is available)
Warning.png Warning

Monitoring with Kwollect under Grid'5000 is still in beta phase. It uses "sid" branch of the API, while legacy monitoring API (based on Ganglia and Kwapi) still uses the "stable" branch. Kwollect is intended to replace the legacy system in the future


Metrics available

The list of metrics available for a given Grid’5000 cluster is described in the Reference API, under the “metrics” entry of the cluster description. For instance, to get the list of available metrics for nodes of taurus cluster, you can use:

$ curl https://api.grid5000.fr/stable/sites/lyon/clusters/taurus | jq .metrics

This returns a list where each element describes a single metric. Most important fields of that description are:

  • name: The name identifying the metric
  • description: A human-readable description of the metric
  • labels (optional): A “label” that will be used to distinguish two metrics of the same kind, but targeting different objects (e.g. temperature of CPU #1 vs temperature of CPU #2)
  • period and optional_period: The interval (in milli-seconds) under which the metric is collected. The former is the default interval, the latter is the interval used when the metric is activated “on demand” (see below). A metric with period value at 0 is not collected by default.


Getting metrics values

The metrics values are stored by Kwollect and available using the Metrology API (under “sid” version) by performing a GET request with appropriate parameters at URL:

https://api.grid5000.fr/sid/sites/<site>/metrics

For instance, to:

  • get all metrics collected inside job 12345 at Lyon:
curl 'https://api.grid5000.fr/sid/sites/lyon/metrics?job_id=1157330'
  • get all metrics from chifflot-5 and chifflot-6 between 2020-06-08 15:00 and 2020-06-08 17:00:
curl 'https://api.grid5000.fr/sid/sites/lille/metrics?nodes=chifflot-5,chifflot-6&start_time=2020-06-08T15:00&end_time=2020-06-08T17:00'
  • get all values from Wattmetre for taurus-2, during last 15 minutes:
curl "https://api.grid5000.fr/sid/sites/lyon/metrics?nodes=taurus-2&metrics=wattmetre_power_watt&start_time=$(date -d '15 min ago' +%s)"

The request will return a JSON-formatted list of metrics values and their associated timestamp.


For a complete description of parameters and returned fields, see the API specification at: https://api.grid5000.fr/doc/sid/#tag/metrics


Important note: To avoid overloading Kwollect servers, request duration is limited to 5 minutes. When too much metrics values are requested, you may hit that limit and receive an error message. In such case, try to reduce the size of your request by selecting less nodes or a shorter time period.


Tip: If you need metrics formatted as CSV, you can use this command line :

curl <URL_to_request_some_metrics> | jq -r '.[] | [.timestamp, .device_id, .metric_id, .value] | @csv'

On-demand metrics

“On-demand” metrics can be enabled for specific jobs by adding the “-t monitor=<metric_to_enable>” option to oarsub. E.g:

$ oarsub -I -t monitor=bmc_cpu_temp_celsius

The provided argument can be a regular expression For instance, to enable all metrics related to temperature:

$ oarsub -I -t monitor='.*temp.*'

If no argument is provided, all “On-demand” metrics available are enabled.


Note:

  • Dedicated wattmetre devices (metric “wattmetre_power_watt”) are able to perform one measurement every 20 milli-seconds. However, this high frequency is only provided using “on demand” activation. For instance, submit your job using:
$ oarsub -I -t monitor='wattmetre_power_watt'

By default, only the value averaged over one second is provided.

  • As prometheus metrics depend on node characteristics, they cannot be fully described. Only a subset of prometheus metrics will be collected by default (described in the API by the prom_default_metrics and prom_nvgpu_default_metrics, when relevant). To enable collecting all prometheus metrics, use "on-demand" activation on prom_all_metrics or prom_nvgpu_all_metrics (for instance, use monitor='prom_.*')


Visualization Dashboard

A simple dashboard to visualize the metrics is also available. Metrics can be displayed by job ID or by date and graphics can be updated in real time with new values.

Dashboards are available at:

https://api.grid5000.fr/sid/sites/<site>/metrics/dashboard

For instance, at Lyon: https://api.grid5000.fr/sid/sites/lyon/metrics/dashboard


Pushing custom metrics

A simple mechanism is available to let you push your own, arbitrary, custom metrics to Kwollect (internally it uses Prometheus Node Exporter "Textfile Collector"). From a node, a custom metric will be collected by writing to a specific file (you currently need root privileges to do so):

$ echo 'kwollect_custom{_metric_id="my_metric"} 42' > /var/lib/prometheus/node-exporter/kwollect.prom

This will push a custom metric named my_metric and with value "42".


The associated timestamp will be the time when Kwollect fetches metrics on the node. You may override the timestamp by adding _timestamp="your_timestamp_in_unix_seconds". For instance:

$ echo 'kwollect_custom{_metric_id="my_metric", _timestamp="1606389005.1234"} 42' > /var/lib/prometheus/node-exporter/kwollect.prom

(You can also add your own labels. See Prometheus metrics naming and format for more information).


Kwollect will continuously pull values from /var/lib/prometheus/node-exporter/kwollect.prom file. It is your responsibility to keep its content updated.

Known problems

  • Metrics from BMC are quite unreliable. They may be inaccurate, or unavailable on some nodes from time to time.
  • Some metrics are not available for every nodes of a cluster: For grcinq and hercule clusters, one every four nodes have more metrics available than the others ; on parasilo and paravance clusters, the measurement of the power consumption by PDUs is only available on some nodes
  • It may happen that few values of a metric are not collected quickly enough to comply with interval described in the Reference API (for instance, when the targeted device is overloaded).
  • Electrical consumption reported by PDU is not always reliable, see Power_Monitoring_Devices#measurement_artifacts_and_pitfalls