The kidle_inj processes, one per CPU, are one way to achieve thermal throttling to help keep your processor cooler. Typically, invoked by thermald, the actual method used is a function of the CPU frequency scaling driver, with possible overides via the /etc/thermald/thermal-cpu-cdev-order.xml file.
Let's work through a couple of examples. Even under 100% utilization on all CPUs, my test server doesn't overheat, so I will set a low thermal trip point of 55 degrees for this.
First using the intel_pstate CPU frequency scaling driver and the powersave governor:
doug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driverintel_pstateintel_pstateintel_pstateintel_pstateintel_pstateintel_pstateintel_pstateintel_pstatedoug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governorpowersavepowersavepowersavepowersavepowersavepowersavepowersavepowersaveNow use turbostat to monitor things, and also watch the maximum performance setting (maximum allowed CPU frequency as a percent). It starts un-throttled:
doug@s15:~$ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct100.
doug@s15:~$ sudo turbostat --quiet --Summary --show Busy%,Bzy_MHz,PkgTmp,PkgWatt --interval 5Busy% Bzy_MHz PkgTmp PkgWatt0.02 1600 26 3.700.03 1600 26 3.702.21 3737 38 6.8738.89 3564 48 42.7094.64 3500 50 58.54 <<< Load being ramped up.100.00 3500 52 58.49 <<< Processor package temperature going up.100.00 3500 53 58.78100.00 3500 56 59.04100.00 3500 56 59.27100.00 3123 53 51.18 <<< Notice throttling via clock frequency100.00 2969 56 47.32100.00 2693 52 41.90100.00 2009 53 28.98100.00 2489 55 37.90100.00 2431 54 36.82100.00 2620 54 40.50100.00 2409 55 36.39100.00 2511 54 38.47100.00 2569 57 39.61100.00 2301 53 34.57100.00 1682 53 23.64100.00 2089 54 30.52100.00 2569 56 39.59100.00 2301 52 34.5587.08 1671 53 22.9848.70 2037 52 24.045.58 2318 44 7.500.02 1603 40 3.880.03 1600 40 3.870.02 1600 39 3.85^C0.04 1600 38 3.86And during the above the max percent was reduced, until after the load was removed, and the processor temperature dropped:
doug@s15:~$ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct60doug@s15:~$ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct60doug@s15:~$ cat /sys/devices/system/cpu/intel_pstate/max_perf_pct100Second, use the acpi-cpufreq CPU scaling driver and ondemand governor:
doug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driveracpi-cpufreqacpi-cpufreqacpi-cpufreqacpi-cpufreqacpi-cpufreqacpi-cpufreqacpi-cpufreqacpi-cpufreqdoug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governorondemandondemandondemandondemandondemandondemandondemandondemandNow use turbostat to monitor things, and also watch the kidle_inject threads.Notice that the turbostat output has an added column , %C6 which is the deepest idle state my processor goes into (done via "hide" instead of via "show" because "show" method doesn't work).
doug@s15:~$ sudo turbostat --Summary --quiet --hide IRQ,Avg_MHz,SMI,GFXMHz,TSC_MHz,GFXWatt,CorWatt,POLL%,CPU%c1,CPU%c3,CPU%c7,CoreTmp,GFX%rc6,Pkg%pc2,Pkg%pc3,Pkg%pc6,POLL,C1,C1E,C3,C6,C1%,C1E%,C3%,C6% --interval 5Busy% Bzy_MHz CPU%c6 PkgTmp PkgWatt0.05 1602 99.83 26 3.710.04 1600 99.87 28 3.710.05 1600 99.83 26 3.710.05 1601 99.84 26 3.7124.67 3591 52.79 45 30.2493.87 3500 0.00 47 58.30 <<< Load ramped up100.00 3500 0.00 50 58.42100.00 3500 0.00 53 58.70100.00 3500 0.00 55 58.99100.00 3500 0.00 56 59.2393.72 3424 6.18 56 54.44 <<< Now some C6 idle time is forced via kidle_inj81.41 3223 18.32 54 44.4977.81 3179 21.82 56 42.0283.82 3348 15.90 57 48.1478.87 3278 20.78 54 44.5266.34 3061 33.15 54 35.0262.65 2898 36.78 54 30.8061.20 2856 38.20 53 29.6363.71 3051 35.73 54 33.3661.67 2938 37.76 54 30.9061.92 2929 37.53 52 30.9563.47 3039 35.97 55 33.1760.87 2862 38.52 56 29.6062.90 3073 36.56 53 33.4062.36 2964 37.09 55 31.6161.16 2866 38.24 53 29.7863.98 3099 35.43 55 34.2856.37 2708 43.01 52 25.8052.01 2616 47.29 53 23.0758.24 2738 41.15 53 26.8665.39 3143 34.05 56 35.6068.01 3209 31.50 56 38.0958.62 2949 40.79 53 29.8358.43 2730 40.95 53 26.8848.87 3158 36.84 53 33.6814.74 2642 70.22 43 14.770.37 1602 99.10 42 4.020.29 1601 99.30 40 3.970.23 1602 99.43 40 3.940.17 1601 99.58 39 3.910.17 1686 99.56 38 3.910.06 1601 99.79 38 3.850.04 1602 99.87 36 3.830.04 1600 99.88 36 3.830.09 1750 99.75 35 3.850.04 1600 99.89 35 3.820.04 1600 99.85 36 3.820.04 1600 99.88 34 3.810.04 1600 99.86 35 3.80^C0.04 1600 99.87 33 3.80And during throttling, the kidle_inj tasks force the deep idle state and the processor power goes down.
doug@s15:~$ ps aux | grep kidleUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 3005 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/0]root 3006 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/1]root 3007 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/2]root 3008 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/3]root 3009 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/4]root 3010 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/5]root 3011 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/6]root 3012 33.2 0.0 0 0 ? S 00:19 0:32 [kidle_inj/7]doug@s15:~$ ps aux | grep kidleUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 3005 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/0]root 3006 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/1]root 3007 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/2]root 3008 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/3]root 3009 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/4]root 3010 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/5]root 3011 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/6]root 3012 33.5 0.0 0 0 ? S 00:19 0:35 [kidle_inj/7]doug@s15:~$ ps aux | grep kidleUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 3005 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/0]root 3006 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/1]root 3007 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/2]root 3008 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/3]root 3009 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/4]root 3010 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/5]root 3011 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/6]root 3012 29.1 0.0 0 0 ? S 00:19 0:44 [kidle_inj/7]doug@s15:~$ ps aux | grep kidle... throttling over, processes gone ...You should not disable whatever thermal protection method you are using, but you could clean your fans and such to help keep the processor cooler. Also, if you have the option to use the pstate method of throttling, the remaining performance is typically higher than the kidle_inj method. For example, and for the workflow used for the above, the pstate method outperforms the intel_powerclamp kidle_inj method by 33%.
Now, if for whatever reason your processor is capable of using the intel_pstate CPU frequency scaling driver, but you have chosen not to, then the suggestion is to use the intel_cpufreq driver (which is just the intel_pstate driver in passive mode) and the ondemand governor. Why? Because then the pstate throttling method will be used. On my system, resulting in about 28% performance improvement over the kidle_inject method under the same throttling conditions.
How to change from intel_pstate to intel_cpufreq?
doug@s15:~$ cat /sys/devices/system/cpu/intel_pstate/statusactivedoug@s15:~$ echo passive | sudo tee /sys/devices/system/cpu/intel_pstate/statuspassiveAnd set the governor:
doug@s15:~$ sudo suroot@s15:/home/doug# for file in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo "ondemand" > $file; doneroot@s15:/home/doug# exitexitdoug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driverintel_cpufreqintel_cpufreqintel_cpufreqintel_cpufreqintel_cpufreqintel_cpufreqintel_cpufreqintel_cpufreqdoug@s15:~$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governorondemandondemandondemandondemandondemandondemandondemandondemandWhy is there such a significant difference in performance? Because the kidle_inj method wastes a lot of time and energy going into and exiting the deep idle state, whereas the pstate method does not.
And for users seeing "idle_inject" instead of, or in addition to, "kidle_inj":
doug@s15:~$ ps aux | grep idle USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 12 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/0]root 16 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/1]root 22 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/2]root 28 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/3]root 34 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/4]root 40 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/5]root 46 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/6]root 52 0.0 0.0 0 0 ? S 09:07 0:00 [idle_inject/7]That is relatively recent, as of kernel 4.19, and the kernel configuration parameter is "CONFIG_IDLE_INJECT", which is set for Ubuntu kernels, but I don't yet know the purpose.
EDIT (2019.08.09, Aug 9th):
Readers, please be aware that for the intel_cpufreq CPU frequency scaling driver (intel_pstate in passive mode), and the acpi-cpufreq driver and schedutil governor some bugs may cause thermald to not work correctly. A reduction in the maximum allowed CPU frequency might not be honored by the system. Patches to fix these issues are in progress, but will take awhile to propagate.
