Origin of a kworker-thread

Question

On my newly installed system using kernel 3.2 I see a kworker-thread which is constantly consuming CPU. I\'d like to find out which part of kernel/module has created this workqu

Answer 1

An additional solution (that I've already posted in different thread not a while ago and maybe you can find there additional solutions to your problem) for these kinds of problems is to use the perf tool (It's not always enabled by default and you may need to install perf on your device).

Step 1: Set perf to record workqueue events:

perf record -e 'workqueue:*' -ag -T

Step 2: Run it as long as you think you need to catch the event (10 seconds should be ok if this event is frequent enough, but you can let it run longer, depending on the available free space you have left on your device) and then stop it with Ctrl + C.

Step 3: Print the captured events (on Linux versions < 4.1 I think it should be -f and not -F):

perf script -F comm,pid,tid,time,event,trace

This will display something like this: 

      task-name    pid/tid     timestamp    event 
------------------------------------------------------------------------------------------------------------------------------------------------------------------- 
    turtle        9201/9201   1473.339166:  workqueue:workqueue_queue_work:    work struct=0xef20d4c4 function=pm_runtime_work workqueue=0xef1cb600 req_cpu=8 cpu=1
    turtle        9201/9201   1473.339176:  workqueue:workqueue_activate_work: work struct 0xef20d4c4
    kworker/0:3  24223/24223  1473.339221:  workqueue:workqueue_execute_start: work struct 0xef20d4c4: function pm_runtime_work
    kworker/0:3  24223/24223  1473.339248:  workqueue:workqueue_execute_end:   work struct 0xef20d4c4

Step 4: Analyzing the table above:

In the first row, a task named turtle (pid 9201) is pushing the work pm_runtime_work to the workqueue. In the third row, we can see that the kworker/0:3 (pid 24223) is executing that work.

Summary: Now back to your questions, we see that kworker/0:3 has been requested by turtle task to run the pm_runtime_work function. So let's go back to the code and see what this pm_runtime_work function does! Maybe here we'll manage to understand why it consumes so much CPU.

Answer 2

So, after some time I found the solution. In fact Anthon is right, it is the ACPI-subsystem which sends interrupts. On my system I disabled the following interrupts and the kworker-thread is calmed down.

echo disable > /sys/firmware/acpi/interrupts/gpe1B
echo disable > /sys/firmware/acpi/interrupts/gpe08

However until now haven't identified what the bogus IRQs coming from gpe08 and gpe1B.

Answer 3

kworker is a kernel thread which processes workqueues.This thread is created in the file linux/kernel/workqueue.c file.

Answer 4

(Seems to me this is rather off topic here, but here's the answer I posted on unix.stackexchange.com.)

I found this thread on lkml that answers your question a little. (It seems even Linus himself was puzzled as to how to find out the origin of those threads.)

Basically, there are two ways of doing this:

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)

For this you will need ftrace to be compiled in your kernel.

This will output what threads are all doing, and is useful for tracing multiple small jobs.

cat /proc/THE_OFFENDING_KWORKER/stack

This will output the stack of a single thread doing a lot of work. It may allow you to find out what caused this specific thread to hog the CPU (for example). THE_OFFENDING_KWORKER is the pid of the kworker in the process list.

Answer 5

kworker / watchdog causing high load regularly when not on cable power - workaround

Had kworker (cause kernel_thread_helper+0x6/0x10 ?) thread 1 spiking to 100% of a cpu for 1 second every 5 seconds only if laptop powered by cable. No problems on battery power. Did not matter if battery fully charged.

It more or less came out of the blue, so I guess the recent Ubuntu update was the cause (3.2.0-55-generic-pae now).

Was looking for half a day, trying to figure out wft the root cause is and endet up trying to disable all acpi interrupts, which did not matter.

Adding GRUB_CMDLINE_LINUX_DEFAULT=”acpi=off” to /etc/defaults/grub helped in the end.

As I discovered right now I added it with these exact special unicode quotes, which might explain the incompatibility with "quiet splash" (with default quotes) which puzzled me. I have to look into some more grub issues (boot menu not being displayed however I try, default entry config not used ...) so I will leave that testing for later.

PS: a week later, the problem is back (without restart, only suspend in between). There might be a correlation to using an usb mouse. Power down/power up helped (restart did not). Threw all grub options away. I am expecting the problem to show up again sometime.

PPS: Took some time (half a year), but it is back again after a resume from ram. No recent updates, just plugging/unplugging power, as I often do. No USB devices plugged/unplugged. Had a totem running (but nothing playing) during suspend. Power down/power up with power cable plugged did not help, power down/power up with power cable unplugged did.