CPU parameters

From Grid5000
Jump to: navigation, search

CPU parameters in Grid'5000 : Hyperthreading, C-State, P-State and Turboboost

This page describes the CPU configuration of Grid'5000 systems and explains how to change the CPU parameters.

This page focus on the following CPU parameters:

  • Hyperthreading (HT): for each physical core, the operating system addresses two logical cores.
  • C-States: Processors and cores idle states management.
  • P-States: Dynamic voltage and frequency scaling (DVFS).
  • Turboboost: allows cores to run faster than their original frequency if they are operating below power and temperature specification limits.

Grid'5000 configuration

Note that the CPU configuration depends on both the BIOS configuration and the operating system.

Default BIOS configuration on Grid'5000:

  • HT and Turboboost are enabled on every clusters installed since 2012. This corresponds to clusters with Intel CPU using Sandy Bridge, Haswell or later archi­tectures.
  • P-States and C-States (including C1E) are enabled but managed by the operating system (see below).

Grid'5000 reference images:

  • The min reference images is a standard Debian installation. Therefore (with Debian 9) the P-States governor is powersave for clusters using the intel_pstate module and ondemand for clusters using the acpi-cpufreq module.
  • Starting with base, the Grid'5000 reference images are tuned for performance. In particular, the P-States governor is 'performance' and the Linux kernel's default of allowing all C-States . This applies to xen, nfs, big and std environments.

The following table presents the status of the platform:

Installation date Site Cluster CPU Family CPU Version Microarchitecture Frequency Server type HT enabled Turboboost enabled P-State driver C-State driver
2006-07-01 lyon sagittaire AMD Opteron 250 K8 2.4 GHz Sun Fire V20z Fail.png Fail.png none none
2010-01-25 rennes parapide Intel Xeon X5570 Nehalem 2.95 GHz SUN FIRE X2270 Fail.png Check.png acpi-cpufreq intel_idle
2010-01-27 sophia suno Intel Xeon E5520 Nehalem 2.25 GHz Dell PowerEdge R410 Fail.png Fail.png acpi-cpufreq intel_idle
2010-11-02 rennes parapluie AMD Opteron 6164 HE K10 1.7 GHz HP ProLiant DL165 G7 Fail.png Fail.png acpi-cpufreq none
2011-01-04 sophia uvb Intel Xeon X5670 Westmere 2.95 GHz Dell PowerEdge C6100 Check.png Fail.png none intel_idle
2011-12-01 luxembourg granduc Intel Xeon L5335 Clovertown 2.0 GHz Dell PowerEdge 1950 Fail.png Fail.png none none
2012-09-14 lyon orion Intel Xeon E5-2630 Sandy Bridge 2.3 GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2012-09-14 lyon taurus Intel Xeon E5-2630 Sandy Bridge 2.3 GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2012-10-02 lyon hercule Intel Xeon E5-2620 Sandy Bridge 2.0 GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2013-04-09 nancy grcinq Intel Xeon E5-2650 Sandy Bridge 2.0 GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2013-09-10 luxembourg petitprince Intel Xeon E5-2630L Sandy Bridge 2.0 GHz Dell PowerEdge M620 Check.png Check.png intel_pstate intel_idle
2013-12-05 nancy graphite Intel Xeon E5-2650 Sandy Bridge 2.0 GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2014-02-21 rennes paranoia Intel Xeon E5-2660 v2 Sandy Bridge 2.2 GHz Dell PowerEdge C6220 II Check.png Check.png intel_pstate intel_idle
2014-04-16 nantes econome Intel Xeon E5-2660 Sandy Bridge 2.2 GHz Dell PowerEdge C6220 Check.png Check.png intel_pstate intel_idle
2015-01-13 rennes parasilo Intel Xeon E5-2630 v3 Haswell 2.4 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2015-01-13 rennes paravance Intel Xeon E5-2630 v3 Haswell 2.4 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2015-05-12 nancy graphique Intel Xeon E5-2620 v3 Haswell 2.4 GHz Dell PowerEdge R720 Check.png Check.png intel_pstate intel_idle
2016-01-04 nancy graoully Intel Xeon E5-2630 v3 Haswell 2.4 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-01-04 nancy grisou Intel Xeon E5-2630 v3 Haswell 2.4 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-01-22 nancy grimoire Intel Xeon E5-2630 v3 Haswell 2.4 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-08-30 nancy grimani Intel Xeon E5-2603 v3 Haswell 1.6 GHz Dell PowerEdge R730 Fail.png Fail.png intel_pstate intel_idle
2016-12-01 lille chetemi Intel Xeon E5-2630 v4 Broadwell 2.2 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2016-12-01 lille chifflet Intel Xeon E5-2680 v4 Broadwell 2.4 GHz Dell PowerEdge R730 Check.png Check.png intel_pstate intel_idle
2016-12-01 lyon nova Intel Xeon E5-2620 v4 Broadwell 2.1 GHz Dell PowerEdge R430 Check.png Check.png intel_pstate intel_idle
2017-06-26 nancy grele Intel Xeon E5-2650 v4 Broadwell 2.2 GHz Dell PowerEdge R730 Check.png Check.png intel_pstate intel_idle
2017-10-16 nantes ecotype Intel Xeon E5-2630L v4 Broadwell 1.8 GHz Dell PowerEdge R630 Check.png Check.png intel_pstate intel_idle
2018-01-16 grenoble yeti Intel Xeon Gold 6130 Skylake 2.1 GHz Dell PowerEdge R940 Check.png Check.png intel_pstate intel_idle
2018-03-22 grenoble dahu Intel Xeon Gold 6130 Skylake 2.1 GHz Dell PowerEdge C6420 Check.png Check.png intel_pstate intel_idle
2018-04-11 nancy grvingt Intel Xeon Gold 6130 Skylake 2.1 GHz Dell PowerEdge C6420 Check.png Check.png intel_pstate intel_idle
2018-08-01 lille chifflot Intel Xeon Gold 6126 Skylake 2.6 GHz Dell PowerEdge R740 Check.png Check.png intel_pstate intel_idle
2018-08-06 lille chiclet AMD EPYC 7301 Zen 2.2 GHz Dell PowerEdge R7425 Check.png Check.png acpi-cpufreq acpi_idle
Last generated from the Grid'5000 Reference API on 2019-03-22 (commit a2028cf)

Up-to-date information can be found by querying the Grid'5000 Reference-API:

See also Hardware for more information about Grid'5000 hardware.

Checking the configuration

  • The CPU configuration is checked automatically by g5k-checks and nodes are disabled if the configuration is wrong.
  • You can also retrieve the CPU configuration of a node by running g5k-checks manually:
Terminal.png fnancy:
oarsub -I -l nodes=1,walltime=00:30 -t deploy
Terminal.png fnancy:
kadeploy3 -f $OAR_NODE_FILE -e debian9-x64-base -k
Terminal.png node:
g5k-checks -m api
Terminal.png node:
cat /tmp/*.yaml
  • The Execo user guide also provides an example for checking the CPU performance settings of Grid5000 clusters.

Setting CPU parameters: Hyperthreading, C-State, P-State and Turboboost

The following explains how to change the CPU parameters by using either sysfs (with root privileges) or kernel boot parameters.

Hyperthreading (HT)

On Intel Xeon, HT provides two logical cores per physical core. With HT, one physical core appears as two processors to the operating system.

Checking the configuration

  • Is the processor HT-capable?
    • cat /proc/cpuinfo | grep flags: flag list should include ‘ht’
  • Is HT enabled?
    • lscpu | grep 'Thread(s) per core' 1: HT is disable, 2: HT is enabled

Enabling/Disabling HT

  • HT must be turned on in the BIOS configuration to enable changing the HT configuration with kernel boot parameters or as root at runtime. Every cluster installed since 2012 is configured like that.
  • Note that on some clusters, the BIOS parameters might be ignored, see Bug #5229.

Setting up HT using root privilege (recommended)

  • HT can be turned ON or OFF using the cpu-hotplug capability of the linux kernel.
  • Interface: /sys/devices/system/cpu/cpu*
  • To disable HT:
    • for i in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | awk -F',' '{print $2}' | sort -u); do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
  • To enable HT:
    • for i in $(ls /sys/devices/system/cpu/cpu*/online); do echo $i; echo 1 > $i; done

Note: HT must be turned ON in the BIOS configuration and the kernel boot parameters should not restrict the visibility of the logical cores (ie. no boot parameters or maxcpus+additional_cpus set to number of logical cores. See below).

Kernel boot command-line parameters (alternative)

Relevant Parameters

HT settings can also be controlled using Kernel boot parameters:

  • maxcpus=n restricts boot time cpus to n. It can be used to turn off HT:
    • For example, using maxcpu=4 at the boot of a quad-core machine disables HT properly because each of the 4 first logical cores correspond to one physical cores. See bug #5229 for additional information.
  • additional_cpus=n allows to bring the other logical cores online later.
  • noht does not always work (See for an example).
  • acpi=ht is a misleading option: It disables ACPI but keeps the components required for HT (See Ubuntu BootOptions).
Setting kernel parameters with Kadeploy

With Kadeploy, you can specify kernel boot parameters on the environment description file:

  • Get the description of debian9-x64-base:
kaenv3 -p debian9-x64-base -u deploy > mydebian-x64-base.env
  • Edit the mydebian-x64-base.env and add a kernel_params entry within the existing boot entry of the YAML file:
  kernel: "/vmlinuz"
  initrd: "/initrd.img"
  kernel_params: maxcpus=4
  • Deploy the environment:
oarsub -I -t deploy -l nodes=1,walltime=1
kadeploy3 -f $OAR_NODEFILE -a mydebian-x64-base.env -k
  • Note that the kernel parameters specified in the environment description file are appended to the default kernel parameters of the cluster.
  • You can check the kernel parameters with cat /proc/cmdline



C-States are power modes that put various processor subsystems to sleep when the CPU is idle.

The deeper C-States levels save more power but require more time to get the CPU active again. The C-States are:

  • C0: the CPU is actively running code (ie. it is the non-idle state)
  • C1: the CPU is idle but quick to wake-up
  • C2 and up: extra power saving states

There is also a C-State called C1E: C1E replaces C1 when C1E is enabled on the BIOS and OS configuration. C1E allows lower CPU’s speed and voltage.

The Dell whitepaper about Controlling Processor C-State Usage in Linux is a great resource to learn about C-States.

C-States are managed by the operating system using the cpuidle subsystem and an idle driver (either intel_idle or acpi_idle). The default driver is intel_idle on new kernel and hardware.

C-States Drivers

intel_idle driver

  • This driver does not use ACPI. It directly uses knowledge of Intel CPU hardware.
  • The C-States list of this driver might differ from the list provided by ACPI.
  • C1E can be disable via the driver (as it is view as one of the C-State in the C-State list).

acpi_idle driver

  • This driver is used when intel_idle is disabled.
  • It takes into account both the BIOS parameters and the kernel parameters.
  • Procfs interface: cat /proc/acpi/processor/CPUx/power
  • Companion tool: acpitool -c

Checking the configuration (Sysfs)

  • Driver is in use:
    • cat /sys/devices/system/cpu/cpuidle/current_driver
  • Name and Latency of C-States:
    • cat /sys/devices/system/cpu/cpu*/cpuidle/state*/name
    • cat /sys/devices/system/cpu/cpu*/cpuidle/state*/latency
  • Idle state statistics:
    • /sys/devices/system/cpu/cpu*/cpuidle/state*/usage
    • Idle state statistics can be retrieve more easily with cpupower (see below).
  • Sysfs also provides an interface to know if the C-States are disabled but as there is multiple ways to disable C-States, you cannot relies on it. The more reliable way to check if C-States are enabled or disabled is to monitor the CPU idle state statistics. Note also that C-States can be disabled independently of each other. For the record, here are the Sysfs interface to know if C-States are disabled:
    • Are C-States disabled ? cat /sys/devices/system/cpu/cpu*/cpuidle/state*/disable.
      • This is only correct if the C-States were disabled with the sysfs interface. It is not correct when the CPU latency is limited with /dev/cpu_dma_latency.
    • Max C-States allowed by the intel_idle driver: cat /sys/module/intel_idle/parameters/max_cstate.
      • It only provides the value corresponding to the intel_idle.max_cstate kernel parameter.
  • The /proc/acpi/processor/*/power interface has been removed from the kernel.

Checking the configuration (Tools)

Cpupower retrieves CPU information from the sysfs interface (apt-get install linux-cpupower):

  • cpupower idle-info works for both intel_idle and acpi_idle and provides the same information as /sys/devices/system/cpu/cpu*/cpuidle/state*/.
  • cpupower monitor -m Idle_Stats gives idle state statistics and is a reliable way to check if C-States are enabled. On the following example, C-States are fully enabled and the CPU is idle:
root@graphene-143:~# cpupower monitor -m Idle_Stats
CPU | POLL | C1-N | C1E- | C3-N | C6-N 
   0|  0.00|  0.00|  0.00|  0.00| 53.04
   1|  0.00|  0.00|  0.00|  0.00| 95.43
   2|  0.00|  0.00|  0.00|  0.04| 76.96
   3|  0.00|  0.00|  0.00|  0.00| 99.97
    Shows  statistics  of  the  cpuidle  kernel  subsystem.  Values  are  retrieved  from   /sys/devices/sys‐
    tem/cpu/cpu*/cpuidle/state*/.   The  kernel  updates  these values every time an idle state is entered or
    left. Therefore there can be some inaccuracy when cores are in an idle state for some time when the  mea‐
    sure starts or ends. In worst case it can happen that one core stayed in an idle state for the whole mea‐
    sure time and the idle state usage time as exported by the kernel did not get updated.  In  this  case  a
    state residency of 0 percent is shown while it was 100.

Extra tools you might find useful:

  • i7z: can be used to check C-state usage regardless of which idle driver is being used. It uses MSR information.
  • powertop: tool to find out what is using power
  • turbostat: report processor frequency and idle statistics
  • hwloc: detect the hierarchical topology of the hardware architectures

Enabling/Disabling C-States

Note that disabling entirely C-States (ie. only allowing C0) interferes with HT and HT should be disable when the CPU is forced to stay on the C0 state.

Dynamic Control of the C-States using root privilege (recommended)

  • C-States can be disabled on a per core and per C-State basis:
    • echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state3/disable
    • or cpupower idle-set -d 3
  • To enable back a C-State:
    • echo 0 > /sys/devices/system/cpu/cpu0/cpuidle/state3/disable
    • or cpupower idle-set -e 3
  • You can also limit the allowed C-States by using the Power management Quality of Service (PM QOS) interface. Indeed, requesting a low latency prevents the processor from entering deep sleep states.
    • The file /dev/cpu_dma_latency can be used to set a maximum allowable latency: Write a number to this file representing the maximum allowed response time in microseconds.
    • This file must be kept open as long as you want to limit the latency.
    • The latency of C-States are given by /sys/devices/system/cpu/cpu*/cpuidle/state*/latency. "0" means only allowing C0.
    • Setting a maximum latency does not update /sys/devices/system/cpu/cpu0/cpuidle/state*/disable or /sys/module/intel_idle/parameters/max_cstate.
    • More information can be found here and here.
    • Here is a ruby example for using /dev/cpu_dma_latency:

if ARGV.empty?
  puts "Usage: sudo ./limit_ctates.rb <latency>"

latency = ARGV[0]

['INT', 'TERM'].each { |sig|
  Signal.trap(sig) {

$f = File.open("/dev/cpu_dma_latency", "w")

On taurus, it seems that limiting latency to 80 ms (C3) also enables C-States up to C7. However, on Graphene, limiting latency to C3 does work.

Kernel boot command-line parameters

intel_idle driver
  • This driver mostly ignores BIOS settings and kernel parameters but idle=halt automatically disable cpuidle including intel_idle, in newer kernels
  • intel_idle.max_cstate=0 disables intel_idle and fall back on acpi_idle.
  • intel_idle.max_cstate=[1-6] specifies the maximum depth of C-states.
acpi_idle driver
  • The depth of C-States can be set with processor.max_cstate=n
  • Note that when processor.max_cstate=0 is used, the kernel actually silently sets it to 1.
Other kernel parameters
  • idle=halt: allows C0-C1(E) (Halt means C1). It allows for low latency.
  • idle=poll: CPU will stay in C0 (Poll means C0). It allows for extremely low latency: The processor will stay in C0 and kept busy in a loop. It increases power usage considerably.
  • idle=mwait: has been removed from 3.x kernels.

Influence of C-States on power usage, network latency and P-States

C-States are a power saving feature of the CPU. Here are some power usage measurement performed on Taurus (bug #6570):

  • C0/POLL (0ms): 183 W
  • C1S (2 ms): 144 W
  • C1E (10ms): 113 W

C-States impact CPU wake-up latencies and have therefore an impact on network latency when the nodes are not busy (see bug #5368 for an example). You should disable C-States for maximum performances. Here are some ping latency measurements on the Infiniband interface of Graphene when nodes are idle:

  • C0 (0 ms): ping in ~ 0.030 ms
  • C1 (3 ms): ping in ~ 0.030 ms
  • C1E (10 ms): ping in ~ 0.045 ms
  • C3 (20 ms): ping in ~ 0.140 ms
  • C6 (200 ms): ping in ~ 0.200 ms



Modern CPU supports dynamic frequency scaling to reduce both the electric energy consumption and the heat generated by the processor. The P-States are the various frequency settings supported by the CPU. C-States and P-States are independent from each other. To understand the difference between C-States and P-States, you can read this or this.

P-States are managed by cpu-freq and kernel drivers (either intel_pstate or acpi-cpufreq). The default driver is intel_pstate on new kernels and new Intel hardwares. Each driver implements several governors, ie. policies for the CPU frequency scaling algorithms.

P-States drivers

acpi_cpufreq driver

This driver supports five governors. The governors performance, powersave and userspace set the frequency statically whereas the governors ondemand and conservative set the CPU depending on the current CPU usage (dynamic scaling).

Governor Description
performance Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
powersave Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
userspace Sets the CPU frequency to the value defined in /sys/devices/system/cpu/cpu*/cpufreq/scaling_setspeed
ondemand Set the CPU frequency to scaling_max_freq when the CPU load threshold (default is 95%) is reached.
conservative Same as ondemand but increases the frequency by steps. The default threshold is 80%.

Those governors are described in the Linux kernel documentation. The governors ondemand and conservative can be finely tuned using the sysfs file accessible parameters described in the documentation (/sys/devices/system/cpu/cpufreq/<governor>/* appears when the governor is in used).

intel_pstate driver

  • This driver supports two governors: performance and powersave.
  • The performance governor is similar to the acpi_cpufreq performance governor, but the policy of the intel-pstate powersave governor depends on the CPU usage. In fact, it corresponds to the ondemand governor of the acpi_cpufreq driver.
  • The meanings of the data in /sys/devices/system/cpu/cpu*/cpufreq/ differs from the acpi_cpufreq driver. See intel-pstate.txt for more information.
  • The driver also provides its own sysfs interface within /sys/devices/system/cpu/intel_pstate/.

Checking the configuration

cpupower frequency-info provides the cpufreq kernel information in a consolidated manner. Information are gathered from the sysfs /sys/devices/system/cpu/cpu*/cpufreq/ interface. The sysfs interface is described here.

Driver and hardware capability:

  • scaling_driver is the driver in use.
  • scaling_available_governors lists the available governors for the driver in use.
  • scaling_available_frequencies lists the frequencies that are available with your CPU model (acpi-cpufreq only).
  • cpuinfo_min_freq and cpuinfo_max_freq gives the frequency range capability of the hardware.

C-States configuration:

  • scaling_governor is the current governor.
  • scaling_max_freq and scaling_min_freq: current frequency range limit used by the governor. When setting a policy you need to first set scaling_max_freq, then scaling_min_freq.
  • scaling_cur_freq: current frequency of the CPU as obtained from the hardware, in KHz. The Intel driver shows the frequency in used but acpi-cpufreq displays the requested frequency (it might change in the future). The same is true for /proc/cpuinfo.
  • scaling_cur_freq: frequency the kernel thinks the CPU runs at.
  • scaling_setspeed (write-only): is used by the userspace governor of the acpi-cpufreq driver. Irrelevant for other governors or the intel driver.

cpufrequtils are unmaintained/deprecated/dead and sys-power/cpupower should be used instead.

Setting up P-States using root privileges

The P-States configuration can be modified using cpupower frequency-set or by echoing into the following files: scaling_governor, scaling_max_freq, scaling_min_freq and scaling_setspeed. Modifying some parameters might be irrelevant for the current driver or the current governor.

$ cpupower frequency-set -g ondemand # set governor
$ cpupower frequency-set -u 2.60GHz    # set the maximum allowed frequency
$ cpupower frequency-set -d 1.80GHz    # set the minimum allowed frequency
$ cpupower frequency-info # check the configuration

On Debian system, you can make changes permanent by editing /etc/default/cpufrequtils. This is the configuration file of the cpufrequtils daemon. Note that other distribution might use another daemon (like cpupower and /etc/default/cpupower). You can also use sysfsutils and /etc/sysfs.conf for managing the configuration.

# /etc/default/cpufrequtils

Intel driver specificities:

  • The scaling_governor is the same for each CPU. The last requested policy is applicable to all CPUs.
  • The scaling_max_freq and scaling_min_freq can be used to set the P-State range of the CPUs but as frequencies are converted to the nearest possible P-State available, this is subject to rounding errors. You can use instead the /sys/devices/system/cpu/intel_pstate/[min_perf_pct,max_perf_pct] parameters of Intel P-State Sysfs.


Related bugs

  • #5327 Low CPU frequency after deployment (fixed)
  • #5368 IPoIB latency on graphene (fixed)
  • #6281 Econome problem on P-State (fixed)
  • #6397 RefAPI CPU Mhz (fixed)
  • #6570 Power consumption with jessie-std (fixed)


Turboboost allows cores to run above their normal operating frequency when the CPU demand is high and as long as the electrical or thermal limits stays below the specification limits.

Checking the configuration

The Sysfs interface depends on the P-State driver in use:

  • To check the driver in use: cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver
  • With the intel pstate driver: cat /sys/devices/system/cpu/intel_pstate/no_turbo (1 == disabled)
  • With the acpi-cpufreq driver: cat /sys/devices/system/cpu/cpufreq/boost (1 == enabled)

The cpupower frequency-info tool also provides information about Turboboost:

  boost state support:
    Supported: yes
    Active: yes

Enabling/Disabling Turboboost

Using the Sysfs interface:

  • With the acpi-cpufreq driver: echo 0 > /sys/devices/system/cpu/cpufreq/boost (0 == disabled)
  • With the intel pstate driver: echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo (1 == disabled)

With the acpi-cpufreq, Turboboost can also be disabled by setting manually the maximum CPU frequency as Turboboost is represented by a CPU frequency in the scaling_available_frequencies list.

In any case, Turboboost can also be disabled by using MSR. See this page or this page for more information.