[Discuss] khugepaged + vmware = massive CPU load

[me at mattgillen.net (Matthew Gillen)]

On 12/13/2015 3:54 PM, Shankar Viswanathan wrote:
> On Fri, 11 Dec 2015 06:42:14 -0500
> Dan Ritter <dsr at randomstring.org> wrote:
> 
>> Normally your system keeps track of memory in 4KB pages. They're
>> analogous to filesystem blocks. Performance is improved when you
>> can satisfy memory allocation requests with less overhead, so
>> hugepages were created. A hugepage is a 2MB page.
> 
> On x86_64 (aka amd64) systems, hugepage can also be 1GB. But barring large databases and playback of large media files, there isn't any benefit from using 1GB pages (it will actually harm performance for most other applications).
> 
> The reason large page sizes are useful is that one 2MB page only uses a single TLB entry in the processor whereas the same 2MB space will need 512 entries if mapped as 4KB pages. So depending on TLB sizes in the processor, you could have lots of TLB misses if you use smaller pages, and each TLB miss takes a large number of processor cycles to do a "table walk" and fetch the translation. TLB pressure is even higher if virtualization is being used as both guest and host translations have to share the TLB.

That explains why hugepages are useful, but it doesn't explain why it
would be detrimental in some cases. The best I could come up with is
that certain applications try to be too cute with their memory
management for efficiency reasons (e.g. image processing), and only deal
with 4KB chunks anyway (i.e. because they "know" how the underlying
memory management works, so they over-optimized based on it).  So
loading that 1MB image, which should take 256 4KB pages now takes 256
2MB pages, which comes out to ~536MB.  Which is probably more memory
than you have spare on your system.

Does that sound plausible (excusing any arithmetic errors)?

Matt