[Discuss] Limiting amount of memory

[shankar.viswan at gmail.com (Shankar Viswanathan)]

On Fri, Sep 13, 2013 at 10:10:07PM -0400, Richard Pieri wrote:
> Some Googling about lead me to the PCI memory hole:
> http://en.wikipedia.org/wiki/PCI_hole
> 
> Short version: in order to boot 32-bit operating systems the BIOS or
> EFI firmware needs to map all PCI devices to the first 4GB of
> address space. This leads to "holes" in the memory map between
> what's available in the lower 4GB and everything above 4GB. That's
> the "missing" RAM.

That is correct, but there a couple of details to this that the PCI_hole Wikipedia article does not 
mention. Refer to this one for some more details: 
http://en.wikipedia.org/wiki/3_GB_barrier

There are two aspects to the address space (here inbound and outbound are from the CPU/host 
perspective):

1. Inbound DMA addresses: legacy 32-bit PCI devices cannot send DMA traffic to addresses greater 
than (2^32 - 1) and therefore addresses for these must be allocated below 4GB. Modern systems 
generally get around this limitation by using an IOMMU to remap those addresses into general 64-bit 
space upto the amount of RAM available in the system. Alternately the OS might implement a "bounce 
buffer" under 4GB for the device that is then immediately memcpy'ed to another higher address and 
the application is told to use the new address.

2. Outbound memory access and register accesses: these are addresses used by the CPU to send writes 
to the device (MMIO accesses). If these addresses are below 4GB as required for many devices or 
when the OS is a 32-bit OS, then these addresses in RAM are effectively unreachable. So if you 
allocate the 3GB to 4GB address range to these devices (common in many OSes), then the DRAM between 
3GB and 4GB is unusable. So, even if you have 4GB of RAM, the applications will only address 3GB. 
AMD processors (and likely Intel ones too) allow an OS that is 64-bit (or 32-bit OSes with PAE 
support) to remap this 1GB region to above 4GB so that you can still use the entire RAM. e.g. if 
you have 4GB in your machine, DRAM addresses in the 3GB to 4GB range will be remapped to 4GB to 5GB 
range. This is known as "memory hole hoisting" in AMD parlance, and AFAIK both 64-bit Windows and 
Linux support this.

Anway, as was mentioned elsewhere in this thread, this by itself doesn't answer the "missing 
memory" question. I too would suspect some kernel driver grabbing a chunk of memory for itself. 
Integrated graphics will usually grab anywhere from 128MB to 1GB depending on configuration. On my 
laptop with an AMD A8-3500M and 8GB of installed RAM (Ubuntu 12.04), /proc/meminfo shows 
MemTotal=7641140kB which means that the integrated Radeon HD GPU has likely reserved the rest for 
itself.

-Shankar