Friday, 10 May 2013

High CPU usage due to kworker

I recently solved a problem with a kworker process taking up more than 60% of one of my eight CPUs. What is kworker? Here is a simple yet good explanation. I forgot to take snapshots, but the scenario was more or less as shown in figure (source).
Before starting, please note that this is not a OS-specific issue. Read this stuff again if you're not fully convinced about it; the above links could suggest this is only a Ubuntu issue, but it's not (I'm using Fedora for example).
 $ uname -a  
 Linux pc-lillo 3.8.11-100.fc17.x86_64 #1 SMP Wed May 1 19:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux  
I don't really like to say I "solved" the problem as I see the following as a workaround, but it actually works good. I just followed what I could find here and here. So, here is what you should do if you see that something is taking up too much CPU.
Make sure the offender is indeed a kworker process. Use top, or ps aux | grep kworker if you prefer.
Find out the gpe (general purpose event) that is causing all of this:
 $ grep . -r /sys/firmware/acpi/interrupts  
Normally you should get a list of this kind:
 /sys/firmware/acpi/interrupts/sci:   162  
 /sys/firmware/acpi/interrupts/error:    0  
 /sys/firmware/acpi/interrupts/gpe00:    0  invalid  
 /sys/firmware/acpi/interrupts/gpe01:    0  invalid  
 /sys/firmware/acpi/interrupts/gpe02:    0  disabled  
 /sys/firmware/acpi/interrupts/gpe03:    0  enabled  
 /sys/firmware/acpi/interrupts/gpe04:    0  disabled  
 ........  
 /sys/firmware/acpi/interrupts/gpe_all:   162  
 /sys/firmware/acpi/interrupts/ff_gbl_lock:    0  enabled  
 /sys/firmware/acpi/interrupts/ff_pwr_btn:    0  enabled  
 /sys/firmware/acpi/interrupts/ff_slp_btn:    0  invalid  
You should see a gpe with a very high value. In my case the value of gpe1B was 162, which is ok; gpe06 had instead a value of (approximately) 170.000. This is definitely NOT ok: that's the proof the problem is with ACPI interrupts.
Let's say the offender gpe is called gpe[XX] (replace the "[XX]" accordingly). Now:
 $ su  
 Password:  
 # cp /sys/firmware/acpi/interrupts/gpe[XX] /pathtobackup  
Note that some commands will be executed even without administrator rights, but the trick will not work (or at least not completely). So it's good to keep the admin rights until the end.
Now we should schedule a task through crontab. Such task should disable the offender gpe, and it must be performed every startup or reboot. To open crontab with vi:
 # crontab -e  
If you want to use another editor (e.g., gedit), just specify it:
 # env EDITOR=gedit crontab -e  
Add the following line to the crontab file:
 @reboot echo "disable" > /sys/firmware/acpi/interrupts/gpe[XX]  
Save, close.
To make it work even after suspend (optional):
 # touch /etc/pm/sleep.d/30_disable_gpe[XX]  
 # chmod +x /etc/pm/sleep.d/30_disable_gpe[XX]  
 # gedit /etc/pm/sleep.d/30_disable_gpe[XX]  
Copy this script in 30_disable_gpe[XX]:
 #!/bin/bash  
 case "$1" in  
   thaw|resume)  
     echo disable > /sys/firmware/acpi/interrupts/gpe[XX] 2>/dev/null  
     ;;  
   *)  
     ;;  
 esac  
 exit $?  
Save, close, and we're done. Sweet.

Note: no luck with the cron job? Have a look at Myk's comments below...

19 comments:

  1. I have the same problem in Ubuntu 14.04 on my new laptop. I tried your tutorial to no avail. Seems that "disable" is not being appended to the file. Here are the lines I added to crontab:

    # edit to hopefully correct kworker high cpu issue
    @reboot echo "disable" > /sys/firmware/acpi/interrupts/gpe10


    I did this as sudo.. Should it have been done as the user instead? Grasping at straws, looking for options.

    Thanks

    ReplyDelete
  2. Hi Myk,

    nope, it has to be done with admin rights... have you tried to execute the commands in a root shell (sudo -s)?

    ReplyDelete
  3. Thanks for the reply. Yes, I tried with root crontab, user crontab, and just issuing the command in terminal like this:


    myk@E45-B4200:~$ sudo -s echo "disable" > /sys/firmware/acpi/interrupts/gpe10
    bash: /sys/firmware/acpi/interrupts/gpe10: Permission denied
    myk@E45-B4200:~$

    Any ideas?
    Thanks

    ReplyDelete
  4. Me again.
    I found that if i do this, it works:

    myk@E45-B4200:~$ sudo su
    root@E45-B4200:/home/myk# echo "disable" > /sys/firmware/acpi/interrupts/gpe10
    root@E45-B4200:/home/myk# cat /sys/firmware/acpi/interrupts/gpe10
    46093046 disabled


    and my cpu drops like a rock.

    So, how do I get this to work properly in a cron job?

    thanks

    ReplyDelete
    Replies
    1. Dunno why, but I thought that "su" was not a valid command in Ubuntu, sorry.. that's exactly what you want, yes.
      Now, delete your crontab and repeat all the procedure as root.

      $ sudo su
      # crontab -e
      # ....

      Let me know how it goes.

      Delete
    2. Still no luck. I entered the cronjob as explained, and it doesn't disable.

      Interesting to note, if i put that command in once in terminal, it still shows as enabled. If i put it in a second time, it does disable. I think I remember someone stating that they had to do it twice on another thread.

      That said, what is the correct way to pass that in a cron job twice? Maybe that will work.

      Delete
    3. Sorry but I'm not the "cron job man", no idea.. share if you get it to work!

      Delete
    4. Got it figured out. I just entered it twice in the cron job, and it works like a charm.

      Thanks

      Delete
    5. Odd but gold :) thank YOU for sharing!

      Delete
  5. Now that's weird. What do you see if you type the following?
    $ ls -l /sys/firmware/acpi/interrupts

    ReplyDelete
    Replies
    1. Results are too long to paste. Here you are.

      http://paste.ubuntu.com/8029138/

      Thanks

      Delete
  6. I did exaclty what you said... and it works perfectly to me.

    Thanks!!

    ReplyDelete
  7. many thanks, i had the same problem with debian 8.2 and the solution worked!

    ReplyDelete
    Replies
    1. You're very welcome, glad to hear it :)

      Delete
  8. This worked for me in Ubuntu 15.10 like a treat

    ReplyDelete
  9. Try also to plug in say a USB stick and watch the CPU drop like a stone. This seems the only for certain work around that works 00% of the time for me

    ReplyDelete
  10. Thanks a lot! This works for me in Debian 8.3,i5-4200m,GT720M,Fujitsu.
    I have this problem since i brought the pc in 2014 in all of win7,win8.1,win10,debian , And finall sloved in debian today.

    ReplyDelete
  11. Thanks for posting this, it solved the kworker/overheating issue for me on an 11" 2013 Macbook Air.

    ReplyDelete
  12. Thanks solved my MBP 11.2 either under Manjaro or Ubuntu

    ReplyDelete