[HH] AoE controller

[Tom Metro]

Bill Bogstad wrote:
> Haven't thought about AoE in a long time.

Yeah, it never really took off.

While writing that message I took a look at a wiki I bookmarked a few
years ago, and it hasn't or has hardly changed. Those Linux AoE server
projects have similarly stagnated, it appears.

Coraid still seems to be the only significant vendor pushing the technology.


> Unless you go with 10Gbit Ethernet (fairly pricey host adapters &
> switches), you aren't going to have the same burst speed as current
> SATA/SAS interfaces.

True. SATA burst speed is either 1.5 Gbps or 3 Gbps, so a single drive
could saturate your Gb Ethernet. That's probably why AoE has stagnated.


> ...multi-port (8 device) host adapters are pretty
> cheap as well.

Sure. You can get 4-port adapters for under $50 and install several of
them, and thus handle a bunch of drives, as long as they will live close
to the motherboard. I'm sure there are pricier 8-port boards...


> If that isn't enough, apparently you can get SATA/SAS multiplexers
> which have decent fanouts (1:8?).

Yes. Like:
http://www.amazon.com/Addonics-eSATA-Port-Multiplier-AD5SARPM-E/dp/B000VEMNAU/

5-drive fanout is the most I've seen, and these fairly small, simple
circuits run >$50. But spread across 5 drives, that's cheap.

This is actually the most promising alternative to AoE.


> What are your use cases?

One scenario I have in mind is a large pool of non-RAID storage used
strictly for backup and "near-line" storage.

A BLU discussion a while back included some statistics on the cost of
tape storage, and it was near or past the cost of disk storage, yet
others in the thread dismissed this as not being a meaningful comparison
as you have to add the cost of a big NAS appliance and additional drives
for RAID. My thought is that the latter isn't a fair comparison to tape,
which doesn't offer RAID, and in some cases doesn't even approach the
reliability of a single drive.

The objective would be a low cost controller/server from which would
could hang an arbitrary pool of drives, most of which would be spun down
at any given moment, and some management software that could cope with
drives being swapped in and out, while tracking what was stored on them.
Performance would only need to be comparable to tape, which shouldn't be
hard to beat.

Qualcom makes a commercial drive-based backup appliance that tries to do
 some of this, but isn't low cost.


> One upside I could see to AoE would be that you could place
> the drives fairly far away from the host systems whereas SATA/SAS is
> likely to restrict you to the same rack or perhaps the same room.

Yes, but I'm not sure that's important...it does, however, let you scale
to more drives than you can fit in close physically proximity to your
server.


> Your point about arbitrary numbers of drives might matter if you want
> truly large amounts of data accessible from a single system and don't
> care too much about how fast you can get to it ("cold" data).

Yes, and yes.


> Why not do some aggregation and serve it up as either large RAIDed
> disks or maybe as a networked file server rather then seeing each
> drive directly?

If you want the ability to remove drives from the pool, know what was
stored on them, and add drives without rebuilding arrays and file
systems, then each drive needs to be individually addressable, at least
at some level. An abstraction layer could be placed over it.

I think there is some crossover with the functionality of unRAID[1]. I'd
have to dig up my BLU posting on it to remember what it was I didn't
like about it.

 -Tom

1. http://www.lime-technology.com/