CPU parameters

From Grid5000
Revision as of 11:31, 6 September 2016 by Jgaidamour (talk | contribs)

Jump to: navigation, search

CPU parameters in Grid'5000 : Hyperthreading, C-State, P-State and Turboboost

This page describes the CPU configuration of Grid'5000 systems and explains how to change the CPU parameters by using either sysfs (with root privileges) or kernel boot parameters.

Grid'5000 configuration

This page focus on the following CPU parameters:

  • Hyperthreading (HT): for each physical core, the operating system addresses two logical cores.
  • C-States: Processors and cores idle states management.
  • P-States: Dynamic voltage and frequency scaling (DVFS).
  • Turboboost: allows cores to run faster than their original frequency if they are operating below power and temperature specification limits.

Note that the CPU configuration depends on both the BIOS configuration and the operating system.

Default BIOS configuration on Grid'5000:

  • HT and Turboboost are enabled on every clusters installed since 2012. This corresponds to clusters with Intel CPU using Sandy Bridge, Haswell or later archi­tectures.
  • P-States and C-States (including C1E) are enabled but managed by the operating system (see below).

Grid'5000 reference images:

  • The min reference images is a standard Debian installation. Therefore (with Debian Jessie) the P-States governor is powersave for clusters using the intel_pstate module and ondemand for clusters using the acpi-cpufreq module.
  • Starting with base, the Grid'5000 reference images are tuned for performance. In particular, the P-States governor is 'performance' and C-States are limited to C0, C1 and C1E (see changes). This applies to xen, nfs, big and std environments. See Tuning_CPU_parameters for more information.

The following table presents the status of the platform as of 09/2016.

Installation date Site Cluster CPU model Server type HT enabled Turboboost enabled P-State driver C-State driver
2007-01-01 lyon sagittaire Opteron 250 2.40GHz Sun Fire V20z Fail.png Fail.png none none
2007-02-23 sophia sol Opteron 2218 2.60GHz Sun Fire X2200 M2 Fail.png Fail.png none none
2007-11-08 luxembourg granduc Xeon L5335 Merom 2.00GHz Dell PowerEdge 1950ES Fail.png Fail.png none none
2008-03-01 lille chinqchint Xeon E5440 Penryn 2.83GHz SGI Altix Xe 310 Fail.png Fail.png none none
2008-10-01 grenoble genepi Xeon E5420 Penryn 2.50GHz Bull R422-E1 Fail.png Fail.png acpi-cpufreq none
2009-03-01 nancy griffon Xeon L5420 Penryn 2.50GHz Carri System CS-5393B Fail.png Fail.png acpi-cpufreq none
2009-08-01 nancy talc Xeon L5420 Penryn 2.50GHz Carri System CS-5393B Fail.png Fail.png acpi-cpufreq none
2009-10-15 rennes parapide Xeon X5570 Nehalem 2.93GHz SUN FIRE X2270 Fail.png Check.png acpi-cpufreq intel_idle
2009-12-01 grenoble edel Xeon E5520 Nehalem 2.27GHz Bullx Blade B500 Fail.png Fail.png acpi-cpufreq intel_idle
2009-12-01 sophia suno Xeon E5520 Nehalem 2.27GHz Dell R410 Fail.png Fail.png acpi-cpufreq intel_idle
2010-03-19 grenoble adonis Xeon E5520 Nehalem 2.27GHz Bullx R422-E2 Fail.png Check.png acpi-cpufreq intel_idle
2010-10-06 rennes parapluie Opteron 6164HE 1.70GHz HP Proliant DL165 G7 Fail.png Fail.png acpi-cpufreq none
2011-01-01 lille chirloute Xeon E5620 Westmere 2.40GHz Dell PowerEdge C6100 Fail.png Fail.png none intel_idle
2011-01-03 nancy graphene Xeon X3440 Nehalem 2.53GHz Carri System CS-5393B Fail.png Check.png acpi-cpufreq intel_idle
2011-03-28 lille chimint Xeon E5620 Westmere 2.40GHz Dell PowerEdge R410 Fail.png Fail.png none intel_idle
2011-04-20 reims stremi Opteron 6164HE 1.70GHz HP Proliant DL165 G7 Fail.png Fail.png acpi-cpufreq none
2012-07-26 lyon hercule Xeon E5-2620 Sandy Bridge 2.00GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2012-07-26 lyon orion Xeon E5-2630 Sandy Bridge 2.30GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2012-07-26 lyon taurus Xeon E5-2630 Sandy Bridge 2.30GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2012-11-29 nantes econome Xeon E5-2660 Sandy Bridge 2.20GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2013-09-11 luxembourg petitprince Xeon E5-2630L Sandy Bridge 2.00GHz Dell PowerEdge M620 Check.png Check.png intel_pstate intel_idle
2013-12-01 nancy graphite Xeon E5-2650 Sandy Bridge 2.00GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2013-12-02 rennes paranoia Xeon E5-2660 Sandy Bridge 2.20GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2014-08-01 nancy graphique Xeon E5-2620 v3 Haswell 2.40GHz Dell PowerEdge R730 Check.png Check.png intel_pstate intel_idle
2015-01-01 rennes parasilo Xeon E5-2630 v3 Haswell 2.40GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2015-01-01 rennes paravance Xeon E5-2630 v3 Haswell 2.40GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-01-01 nancy graoully Xeon E5-2630 v3 Haswell 2.40GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-01-05 nancy grimoire Xeon E5-2630 v3 Haswell 2.40GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-01-05 nancy grisou Xeon E5-2630 v3 Haswell 2.40GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle

Up-to-date information can be found by querying the Grid'5000 Reference-API:

See also Special:G5KHardware for more information about Grid'5000 hardware.

Checking the configuration

  • The CPU configuration is checked automatically by g5k-checks and nodes are disabled if the configuration is wrong.
  • You can also retrieve the CPU configuration of a node by running g5k-checks manually:
Terminal.png fnancy:
oarsub -I -l nodes=1,walltime=00:30 -t deploy
Terminal.png fnancy:
kadeploy3 -f $OAR_NODE_FILE -e jessie-x64-base -k
Terminal.png node:
g5k-checks -m api
Terminal.png node:
cat /tmp/*.yaml
  • The Execo user guide also provides an example for checking the CPU performance settings of Grid5000 clusters.

Setting CPU parameters: Hyperthreading, C-State, P-State and Turboboost

Hyperthreading (HT)

On Intel Xeon, HT provides two logical cores per physical core. With HT, one physical core appears as two processors to the operating system.

Checking the configuration

  • Is the processor HT-capable?
    • cat /proc/cpuinfo | grep flags: flag list should include ‘ht’
  • Is HT enabled?
    • lscpu | grep 'Thread(s) per core' 1: HT is disable, 2: HT is enabled

Enabling/Disabling HT

  • HT must be turned on in the BIOS configuration to enable changing the HT configuration with kernel boot parameters or as root at runtime. Every cluster installed since 2012 is configured like that (see CPU_parameters_in_Grid5000 for more information).
  • Note that on some clusters, the BIOS parameters might be ignored. Ex: on adonis à Grenoble, see Bug #5229.

Setting up HT using root privilege (recommended)

  • HT can be turned ON or OFF using the cpu-hotplug capability of the linux kernel.
  • Interface: /sys/devices/system/cpu/cpu*
  • To disable HT:
    • for i in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -d ',' -f 2 | sort -u); do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
  • To enable HT:
    • for i in $(ls /sys/devices/system/cpu/cpu*/online); do echo $i; echo 1 > $i; done

Note: HT must be turned ON in the BIOS configuration and the kernel boot parameters should not restrict the visibility of the logical cores (ie. no boot parameters or maxcpus+additional_cpus set to number of logical cores. See below).

Kernel boot command-line parameters (alternative)

Relevant Parameters

HT settings can also be controlled using Kernel boot parameters:

  • maxcpus=n restricts boot time cpus to n. It can be used to turn off HT:
    • For example, using maxcpu=4 at the boot of a quad-core machine disables HT properly because each of the 4 first logical cores correspond to one physical cores. See bug #5229 for additional information.
  • additional_cpus=n allows to bring the other logical cores online later.
  • noht does not always work (See for an example).
  • acpi=ht is a misleading option: It disables ACPI but keeps the components required for HT (See Ubuntu BootOptions).
Setting kernel parameters with Kadeploy

With Kadeploy, you can specify kernel boot parameters on the environment description file:

  • Get the description of jessie-x64-base:
kaenv3 -p jessie-x64-base -u deploy > myjessie-x64-base.env
  • Edit the myjessie-x64-base.env and add a kernel_params entry within the existing boot entry of the YAML file:
  kernel: "/vmlinuz"
  initrd: "/initrd.img"
  kernel_params: maxcpus=4
  • Deploy the environment:
oarsub -I -t deploy -l nodes=1,walltime=1
kadeploy3 -f $OAR_NODEFILE -a myjessie-x64-base.env -k
  • Note that the kernel parameters specified in the environment description file are appended to the default kernel parameters of the cluster.
  • You can check the kernel parameters with cat /proc/cmdline



C-States are power modes that put various processor subsystems to sleep when the CPU is idle.

The deeper C-States levels save more power but require more time to get the CPU active again. The C-States are:

  • C0: the CPU is actively running code (ie. it is the non-idle state)
  • C1: the CPU is idle but quick to wake-up
  • C2 and up: extra power saving states

There is also a C-State called C1E: C1E replaces C1 when C1E is enabled on the BIOS and OS configuration. C1E allows lower CPU’s speed and voltage.

The Dell whitepaper about Controlling Processor C-State Usage in Linux is a great resource to learn about C-States.

C-States are managed by the operating system using the cpuidle subsystem and an idle driver (either intel_idle or acpi_idle). The default driver is intel_idle on new kernel and hardware.

C-States Drivers

intel_idle driver

  • This driver does not use ACPI. It directly uses knowledge of Intel CPU hardware.
  • The C-States list of this driver might differ from the list provided by ACPI.
  • C1E can be disable via the driver (as it is view as one of the C-State in the C-State list).

acpi_idle driver

  • This driver is used when intel_idle is disabled.
  • It takes into account both the BIOS parameters and the kernel parameters.
  • Procfs interface: cat /proc/acpi/processor/CPUx/power
  • Companion tool: acpitool -c

Checking the configuration (Sysfs)

  • Driver is in use:
    • cat /sys/devices/system/cpu/cpuidle/current_driver
  • Name and Latency of C-States:
    • cat /sys/devices/system/cpu/cpu*/cpuidle/state*/name
    • cat /sys/devices/system/cpu/cpu*/cpuidle/state*/latency
  • Idle state statistics:
    • /sys/devices/system/cpu/cpu*/cpuidle/state*/usage
    • Idle state statistics can be retrieve more easily with cpupower (see below).
  • Sysfs also provides an interface to know if the C-States are disabled but as there is multiple ways to disable C-States, you cannot relies on it. The more reliable way to check if C-States are enabled or disabled is to monitor the CPU idle state statistics. Note also that C-States can be disabled independently of each other. For the record, here are the Sysfs interface to know if C-States are disabled:
    • Are C-States disabled ? cat /sys/devices/system/cpu/cpu*/cpuidle/state*/disable.
      • This is only correct if the C-States were disabled with the sysfs interface. It is not correct when the CPU latency is limited with /dev/cpu_dma_latency.
    • Max C-States allowed by the intel_idle driver: cat /sys/module/intel_idle/parameters/max_cstate.
      • It only provides the value corresponding to the intel_idle.max_cstate kernel parameter.
  • The /proc/acpi/processor/*/power interface has been removed from the kernel.

Checking the configuration (Tools)

Cpupower retrieves CPU information from the sysfs interface (apt-get install linux-cpupower):

  • cpupower idle-info works for both intel_idle and acpi_idle and provides the same information as /sys/devices/system/cpu/cpu*/cpuidle/state*/.
  • cpupower monitor -m Idle_Stats gives idle state statistics and is a reliable way to check if C-States are enabled. On the following example, C-States are fully enabled and the CPU is idle:
root@graphene-143:~# cpupower monitor -m Idle_Stats
CPU | POLL | C1-N | C1E- | C3-N | C6-N 
   0|  0.00|  0.00|  0.00|  0.00| 53.04
   1|  0.00|  0.00|  0.00|  0.00| 95.43
   2|  0.00|  0.00|  0.00|  0.04| 76.96
   3|  0.00|  0.00|  0.00|  0.00| 99.97
    Shows  statistics  of  the  cpuidle  kernel  subsystem.  Values  are  retrieved  from   /sys/devices/sys‐
    tem/cpu/cpu*/cpuidle/state*/.   The  kernel  updates  these values every time an idle state is entered or
    left. Therefore there can be some inaccuracy when cores are in an idle state for some time when the  mea‐
    sure starts or ends. In worst case it can happen that one core stayed in an idle state for the whole mea‐
    sure time and the idle state usage time as exported by the kernel did not get updated.  In  this  case  a
    state residency of 0 percent is shown while it was 100.

Extra tools you might find useful:

  • i7z: can be used to check C-state usage regardless of which idle driver is being used. It uses MSR information.
  • powertop: tool to find out what is using power
  • turbostat: report processor frequency and idle statistics
  • hwloc: detect the hierarchical topology of the hardware architectures

Enabling/Disabling C-States

Note that disabling entirely C-States (ie. only allowing C0) interferes with HT and HT should be disable when the CPU is forced to stay on the C0 state.

Grid'5000 specifics

On the base, xen, nfs, big and std environments, a service daemon is limiting the C-States to C0-C1-C1E using the sysfs interface. C-States can be fully enabled back by stopping the service: systemctl status cstates

Dynamic Control of the C-States using root privilege (recommended)

  • C-States can be disabled on a per core and per C-State basis:
    • echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state3/disable
    • or cpupower idle-set -d 3
  • To enable back a C-State:
    • echo 0 > /sys/devices/system/cpu/cpu0/cpuidle/state3/disable
    • or cpupower idle-set -e 3
  • You can also limit the allowed C-States by using the Power management Quality of Service (PM QOS) interface. Indeed, requesting a low latency prevents the processor from entering deep sleep states.
    • The file /dev/cpu_dma_latency can be used to set a maximum allowable latency: Write a number to this file representing the maximum allowed response time in microseconds.
    • This file must be kept open as long as you want to limit the latency.
    • The latency of C-States are given by /sys/devices/system/cpu/cpu*/cpuidle/state*/latency. "0" means only allowing C0.
    • Setting a maximum latency does not update /sys/devices/system/cpu/cpu0/cpuidle/state*/disable or /sys/module/intel_idle/parameters/max_cstate.
    • More information can be found here and here.
    • Here is a ruby example for using /dev/cpu_dma_latency:

if ARGV.empty?
  puts "Usage: sudo ./limit_ctates.rb <latency>"

latency = ARGV[0]

['INT', 'TERM'].each { |sig|
  Signal.trap(sig) {

$f = File.open("/dev/cpu_dma_latency", "w")

On taurus, it seems that limiting latency to 80 ms (C3) also enables C-States up to C7. However, on Graphene, limiting latency to C3 does work.

Kernel boot command-line parameters

intel_idle driver
  • This driver mostly ignores BIOS settings and kernel parameters but idle=halt automatically disable cpuidle including intel_idle, in newer kernels
  • intel_idle.max_cstate=0 disables intel_idle and fall back on acpi_idle.
  • intel_idle.max_cstate=[1-6] specifies the maximum depth of C-states.
acpi_idle driver
  • The depth of C-States can be set with processor.max_cstate=n
  • Note that when processor.max_cstate=0 is used, the kernel actually silently sets it to 1.
Other kernel parameters
  • idle=halt: allows C0-C1(E) (Halt means C1). It allows for low latency.
  • idle=poll: CPU will stay in C0 (Poll means C0). It allows for extremely low latency: The processor will stay in C0 and kept busy in a loop. It increases power usage considerably.
  • idle=mwait: has been removed from 3.x kernels.

Influence of C-States on power usage, network latency and P-States

C-States are a power saving feature of the CPU. Here are some power usage measurement performed on Taurus (bug #6570):

  • C0/POLL (0ms): 183 W
  • C1S (2 ms): 144 W
  • C1E (10ms): 113 W

C-States impact CPU wake-up latencies and have therefore an impact on network latency when the nodes are not busy (see bug #5368 for an example). You should disable C-States for maximum performances. Here are some ping latency measurements on the Infiniband interface of Graphene when nodes are idle:

  • C0 (0 ms): ping in ~ 0.030 ms
  • C1 (3 ms): ping in ~ 0.030 ms
  • C1E (10 ms): ping in ~ 0.045 ms
  • C3 (20 ms): ping in ~ 0.140 ms
  • C6 (200 ms): ping in ~ 0.200 ms



Modern CPU supports dynamic frequency scaling to reduce both the electric energy consumption and the heat generated by the processor. The P-States are the various frequency settings supported by the CPU. C-States and P-States are independent from each other. To understand the difference between C-States and P-States, you can read this or this.

P-States are managed by cpu-freq and kernel drivers (either intel_pstate or acpi-cpufreq). The default driver is intel_pstate on new kernels and new Intel hardwares. Each driver implements several governors, ie. policies for the CPU frequency scaling algorithms.

P-States drivers

acpi_cpufreq driver

This driver supports five governors. The governors performance, powersave and userspace set the frequency statically whereas the governors ondemand and conservative set the CPU depending on the current CPU usage (dynamic scaling).

Governor Description
performance Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
powersave Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
userspace Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_setspeed
ondemand Set the CPU frequency to scaling_max_freq when the CPU load threshold (default is 95%) is reached.
conservative Same as ondemand but increases the frequency by steps. The default threshold is 80%.

Those governors are described in the Linux kernel documentation. The governors ondemand and conservative can be finely tuned using the sysfs file accessible parameters described in the documentation (/sys/devices/system/cpu/cpufreq/<governor>/* appears when the governor is in used).

intel_pstate driver

  • This driver supports two governors: performance and powersave.
  • The performance governor is similar to the acpi_cpufreq performance governor, but the policy of the intel-pstate powersave governor depends on the CPU usage. In fact, it corresponds to the ondemand governor of the acpi_cpufreq driver.
  • The meanings of the data in /sys/devices/system/cpu/cpu*/cpufreq/ differs from the acpi_cpufreq driver. See intel-pstate.txt for more information.
  • The driver also provides its own sysfs interface within /sys/devices/system/cpu/intel_pstate/.

Checking the configuration

cpupower frequency-info provides the cpufreq kernel information in a consolidated manner. Information are gathered from the sysfs /sys/devices/system/cpu/cpu*/cpufreq/ interface. The sysfs interface is described here.

Driver and hardware capability:

  • scaling_driver is the driver in use.
  • scaling_available_governors lists the available governors for the driver in use.
  • scaling_available_frequencies lists the frequencies that are available with your CPU model (acpi-cpufreq only).
  • cpuinfo_min_freq and cpuinfo_max_freq gives the frequency range capability of the hardware.

C-States configuration:

  • scaling_governor is the current governor.
  • scaling_max_freq and scaling_min_freq: current frequency range limit used by the governor. When setting a policy you need to first set scaling_max_freq, then scaling_min_freq.
  • scaling_cur_freq: current frequency of the CPU as obtained from the hardware, in KHz. The Intel driver shows the frequency in used but acpi-cpufreq displays the requested frequency (it might change in the future). The same is true for /proc/cpuinfo.
  • scaling_cur_freq: frequency the kernel thinks the CPU runs at.
  • scaling_setspeed (write-only): is used by the userspace governor of the acpi-cpufreq driver. Irrelevant for other governors or the intel driver.

cpufrequtils are unmaintained/deprecated/dead and sys-power/cpupower should be used instead.

Setting up P-States using root privileges

The P-States configuration can be modified using cpupower frequency-set or by echoing into the following files: scaling_governor, scaling_max_freq, scaling_min_freq and scaling_setspeed. Modifying some parameters might be irrelevant for the current driver or the current governor.

$ cpupower frequency-set -g ondemand # set governor
$ cpupower frequency-set -u 2.60GHz    # set the maximum allowed frequency
$ cpupower frequency-set -d 1.80GHz    # set the minimum allowed frequency
$ cpupower frequency-info # check the configuration

On Debian system, you can make changes permanent by editing /etc/default/cpufrequtils. This is the configuration file of the cpufrequtils daemon. Note that other distribution might use another daemon (like cpupower and /etc/default/cpupower). You can also use sysfsutils and /etc/sysfs.conf for managing the configuration.

# /etc/default/cpufrequtils

Intel driver specificities:

  • The scaling_governor is the same for each CPU. The last requested policy is applicable to all CPUs.
  • The scaling_max_freq and scaling_min_freq can be used to set the P-State range of the CPUs but as frequencies are converted to the nearest possible P-State available, this is subject to rounding errors. You can use instead the /sys/devices/system/cpu/intel_pstate/[min_perf_pct,max_perf_pct] parameters of Intel P-State Sysfs.


Related bugs

  • #5327 Low CPU frequency after deployment (fixed)
  • #5368 IPoIB latency on graphene (fixed)
  • #6281 Econome problem on P-State (fixed)
  • #6397 RefAPI CPU Mhz (fixed)
  • #6570 Power consumption with jessie-std (fixed)


Turboboost allows cores to run above their normal operating frequency when the CPU demand is high and as long as the electrical or thermal limits stays below the specification limits.

Checking the configuration

The Sysfs interface depends on the P-State driver in use:

  • To check the driver in use: cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver
  • With the intel pstate driver: cat /sys/devices/system/cpu/intel_pstate/no_turbo (1 == disabled)
  • With the acpi-cpufreq driver: cat /sys/devices/system/cpu/cpufreq/boost (1 == enabled)

The cpupower frequency-info tool also provides information about Turboboost:

  boost state support:
    Supported: yes
    Active: yes

Enabling/Disabling Turboboost

Using the Sysfs interface:

  • With the acpi-cpufreq driver: echo 0 > /sys/devices/system/cpu/cpufreq/boost (0 == disabled)
  • With the intel pstate driver: echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo (1 == disabled)

With the acpi-cpufreq, Turboboost can also be disabled by setting manually the maximum CPU frequency as Turboboost is represented by a CPU frequency in the scaling_available_frequencies list.

In any case, Turboboost can also be disabled by using MSR. See this page or this page for more information.