[Discuss] khugepaged + vmware = massive CPU load

[dbarrett at blazemonger.com (Daniel Barrett)]

I just spent 3 weeks trying to solve a performance problem that showed
up when I upgraded to Ubuntu 15.10 (from 14.04). Here are the details
in case anybody else runs into it.

Every time I tried to display a large image (using ImageMagick's
"display" command, geeqie, Google Chrome, etc.), if I was also running
VMware Workstation 12 at the time, the entire system's performance
plummeted. The vmware-vmx process would hog the CPU (100-200%,
according to "top").  My VM guests would slow to a crawl, the host's
window manager would become unresponsive, and so on. Strangely, the
CPU load reported by the VM guest OSes was nearly zero... they weren't
doing anything. But the second I shut down my VM guests, the problem
vanished.

A process called "khugepaged" was also visible sometimes as the
problem occurred. (Ominous music plays....)

VMware tech support spent 2 weeks debugging with me over email, trying
all sorts of options. It's a beefy computer (12 cores, 32 GB RAM), so
this slowdown was really weird. The problem remained no matter what we
tried.

Then I ran across this article tonight:

  http://unversioned.blogspot.com/2015/10/vmware-workstation-11-ubuntu-high-cpu-utilization.html

Sure enough, when I made the recommended khugepaged-related change,
disabling "transparent hugepages" in the host as suggested, the
symptoms vanished.

  # echo never > /sys/kernel/mm/transparent_hugepage/enabled
  # echo never > /sys/kernel/mm/transparent_hugepage/defrag

I don't understand much about khugepaged except that it's supposed to
be (ironically) a performance optimization:

  http://www.phoronix.com/scan.php?page=article&item=linux_transparent_hugepages

Can anybody explain what might have been happening here, and whether I
am losing anything important by disabling transparent hugepages?

Thank you,
Dan

--
Dan Barrett
dbarrett at blazemonger.com