[Discuss] Whence distributed operating systems?

[bogstad at pobox.com (Bill Bogstad)]

On Thu, Apr 28, 2016 at 3:17 PM, Rich Pieri <richard.pieri at gmail.com> wrote:
> On 4/28/2016 4:28 AM, Bill Bogstad wrote:
>> So memory was shared? between the QBBs?   This sounds more like a NUMA
>> architecture environment.  What would you say are the differences
>> between this definition
>> of SSI and NUMA?
>
> In a NUMA machine, memory is directly attached to the CPUs but not all
> of that memory is local to each CPU. Galaxy wasn't NUMA. Each QBB was
> NUMA: 4 processors with 512MB local to each with the rest being
> non-local but still directly attached. Memory in one QBB was not
> directly attached to the processors in other QBBs; it was shared via
> software over a fibre data bus.

Can you clarify what you mean by "shared via software"?   Did the
virtual memory system, page fault data from remote QBBs as needed or
was there a fibre bus transaction every time a local process accessed
remote memory?   I understood your original note to mean that from a
programs perspective it could allocate/use memory in other QBBs
transparently (except for possible performance differences).  The fact
that this was done via a fibre data bus vs. a faster local bus would
seem to me to be an implementation detail.  It still sounds like a
NUMA with three levels of memory access (CPU local, QBB local, remote
QBB).   But all of the memory transparently visible in a program's
address space.

Bill Bogstad