[Discuss] Redundant array of inexpensive servers: clustering?

[tmetro+blu at gmail.com (Tom Metro)]

Rich Braun wrote:
> Jabr observed:
>> adding HA to a legacy application after the fact is a lot like adding
>> security to an application after it's been developed, instead of
>> addressing security as part of the application development process.
> 
> Very astute.

Isn't this just a natural indicator of immaturity in this market? As the
tech matures, the core needs will be figured out and productized.
Whether that means creating common libraries to implement HA
functionality at the application level, or packaging applications
specifically for HA environments.

We've already seen this happening at the infrastructure level. After
all, a project like OpenStack is just existing VM tech packaged up with
some management UI added.


> Wouldn't it be nice if there were some OpenStack-like HA
> framework that was (a) easy to set up, (b) worked the same way on every
> hardware config ranging from a single laptop to a global cluster of data
> centers, and (c) enjoyed full support from most people in the FOSS-developer
> community.

Maye it is just marketing, but it seems like this is what Canonical has
been working on. Frederico, I'm sure, will say more at the next meeting.

Whether they pull it off, and in particular whether they accomplish "c"
- community acceptance - is yet to be seen.


> ...has AWS developed such a strategic advantage that all alternative
> infrastructure solutions will be absorbed or destroyed in the coming
> shakeout of cloud-service providers?

I actually think trends are suggesting the reverse of what your question
implies. I think we are actually at or near the peak of Amazon as a
cloud provider.

Earlier attempts at competing with Amazon didn't seem to gain a lot of
traction. There were various small hosting providers that tried creating
similar or semi-compatible clouds. There was Eucalyptus for private
clouds. And Google made a half-hearted attempt at being a cloud provider.

In recent years RackSpace has grown much bigger, and sponsored OpenStack
development. Community interest in private cloud tech has mostly aligned
behind OpenStack. A bunch of hosting providers have also jumped on the
OpenStack bandwagon. (Dreamhost, for example, in addition to using
OpenStack internally, spun off an OpenStack support company, and
developed the Ceph filesystem.)

Google is now taking a more serious run at being a cloud provider, with
recently announced new more competitive pricing. When they started out,
what you could run in their cloud was quite limited. Now they let you
run arbitrary VMs, like Amazon.

For reference, see "This Week in Google" #242, where at time offset 1:16:00:
http://www.youtube.com/watch?list=UUM4IHEg4jh7X_dJhzw_O4Gw&feature=player_detailpage&v=mgD57YnyrKg#t=4574

Kevin Marks says "this is good...the fact that we've got 3 suppliers
[referring to I believe Amazon, Microsoft, and Google] that let us run
stuff in the cloud, and don't require us to use their technology as much
as much as they did, is good." And he goes on to explain how Google's
cloud offering evolved from their restrictive "App Engine" to their
current product where you can run your own VMs.

(Google's Dev channel on YouTube had a whole slew of videos posted last
week detailing how their cloud tech works.)

I think the Google change is significant, but if OpenStack gains wide
acceptance, it'll be far more significant for lower-end buyers. It'll be
like what happened when Linux + Apache facilitated hundreds of commodity
web hosting providers to spring up.

So if you are designing apps to run in the cloud that are tightly
coupled to AWS, a year or two from now, that might look like you're at a
disadvantage.


> That has serious consequences to the careers of anyone with ambition in
> systems/network infrastructure.

In the same video above, at time offset 1:18:00 Gina Trapani says, "[a
Google] product manager was saying to me that at Google they don't have
ops teams. They've automated the crap out of everything, right. And
everything just kinda works. They've got fleets of site reliability
engineers (SREs), but this ops stuff they don't do - nobody wants to do
the ops stuff. And nobody should have to do the ops stuff. And because
they automated the crap out of it, and automated it all, they want to
offer these services to everyone else so that other startups don't have
to to have ops folks..."

So yeah, this probably means a percentage of these sorts of jobs will be
lost to automation. But there will still be a need for systems engineers
at places like Google, and working for customers of the cloud providers,
just fewer of them.

 -Tom

-- 
Tom Metro
The Perl Shop, Newton, MA, USA
"Predictable On-demand Perl Consulting."
http://www.theperlshop.com/