enterprise distribution

[Tom Metro]

Here's another "which distribution" question, which is going to sound
a bit redundant with the one posted a few weeks ago, as I'm also
looking specifically for a distribution ideal for a web server. (I had
actually started researching the topic before that question was
posted, but got sidetracked on other projects.)

My second criteria is to select a distribution that has good
commercial appeal so that I can gain exposure to a distribution that
future clients/customers will be using. (I'm currently using Debian, 
which I like, but it has somewhat limited commercial appeal. I've used 
Red Hat in the past, but in the pre-enterprise days.) Although I have 
neither the budget nor the interest in paying $1000+ for a commercial 
enterprise distribution, so it seems my best alternative would be to 
approximate the chosen commercial enterprise distribution by using a clone.

A minor criteria is that the distribution should have licensing terms
that make it friendly for royalty free distribution as part of an
appliance. (I think most of the official commercial distributions won't 
qualify for this. Red Hat does have a program for appliances 
(http://www.redhat.com/solutions/partners/runtime/), which sounds good, 
but costs about $25K/year.)

I'd also be interested to know how consultant/reseller friendly the 
commercial distributions are. Does anyone have experience reselling 
commercial support for Red Hat or SUSE? (If I'm following the Red Hat 
site correctly, it appears they don't offer any partner programs 
appropriate for a consultant/small VAR. Novell, up at the "brochure 
level," seems inclusive of consultants/small VARs, but they keep the 
cost/requirements of the partnering programs more hidden.)


I think the answer to what I'm looking for is probably CentOS, but I'll 
include my research below for discussion. In summary, SUSE looks good, 
but is way behind Red Hat in terms of commercial adoption (see below). 
Red Hat Enterprise is too expensive, so a clone is probably the best 
alternative for building experience with the platform. CentOS appears to 
be leading the pack of RHE clones at the moment. Lineox, another RHE 
clone, is a possible second choice.

If I wanted to be more speculative, I'd consider trying something like 
Gentoo, which unexpectedly seems to have a growing commercial appeal, 
and I've read might offer technical advantages when it comes to package 
management.

I'd be interested to hear if anyone thinks there is another minor 
distribution that they think could be a break out hit with corporate 
users. (Looking at the stats below, Debian comes in 3rd behind Red Hat 
and SUSE in corporate use, but a distant 3rd, and for reasons mentioned 
below, I'm not sure it'll ever catch up in the corporate space.)


Here's the research:

To get a feel for distribution popularity, I figured I'd run some 
general web search engine queries. I was expecting Red Hat would be at 
the top, but I was curious to see how far behind SUSE would be. And more 
importantly, how the Red Hat Enterprise clones would compare.

Then I had the thought that job postings might actually be a better 
proxy for commercial adoption than a general web search would, so I ran 
the same terms through the monster.com job search engine (no location 
specified; job category: "Information Technology"). More recently I ran 
across indeed.com, another - possibly more comprehensive - job search 
engine, and repeated the queries there. Here's what I found:


			Google		Monster		Indeed
redhat			24,100,000	120		1,280
"red hat"		17,700,000	199		1,471
suse			26,400,000	34		343

suse -redhat -rhce					186
rhce -suse						161

"red hat linux"		 6,270,000	76		752
"redhat linux"		 2,740,000	42		646
"suse linux"		 5,450,000	9		110

"red hat enterprise"	 1,100,000	9		110
"redhat enterprise"	   120,000	6		70
"SUSE LINUX Enterprise"	   464,000	1		2
"SUSE Enterprise"	    53,400	1		14

debian			64,400,000	10		62
Gentoo			 8,800,000	4		18
UserLinux		   168,000	0		0

centos			   945,000	0		1
Lineox			   170,000	0		0
X/OS -mac		   126,000	0		0
"white box linux"	    52,700	0		0
"Tao Linux"		    40,500	0		0
ThinTao			        40	0		0

(Notes: "X/OS" is pretty much impossible to isolate, as Google and 
Indeed ignore the slash and consider things like "Mac OS 7.x-OS-X" to be 
a match, and even with the "-mac" to filter out the Mac stuff,  it still 
mostly turns up false positives.)


Looks like Red Hat has a substantial lead on SUSE. Perhaps as many as 10 
times the number of job mentions (combining "redhat" and "red hat": 
about 300 vs. 30 at Monster; about 3000 vs. 300 at Indeed). And of the 
jobs mentioning SUSE, it looks like almost half also mention RedHat or 
RHCE (Red Hat Certified Engineer, I presume).

On the enterprise side of things, the gap isn't quite as big at Monster 
(Red Hat gets 15 vs. 2 for SUSE), though the count is probably too low 
to be statistically significant. At Indeed (Red Hat 180 vs. 16 for SUSE) 
the difference is again in the 10 to 1 ballpark.

So as expected, it seems Red Hat is the safe bet for what businesses 
will be using in the near future.

As for the clones, the job market numbers are less significant, as you 
wouldn't expect them to be highly adopted within large organizations, 
not to mention that they're all relatively new. The more general web 
search is the more interesting figure, and for that CentOS clearly has 
the lead.

Also, according to DistroWatch:

http://distrowatch.com/stats.php?section=popularity

CentOS has climbed from 29th to 14th place in the last year (actually a 
step ahead of Red Hat), and is the highest ranked distro that claims to 
be an "enterprise" distribution, with the except of SUSE, which comes in 
at #5 currently. (Though I imagine most of SUSE's popularity at 
DistroWatch comes from its non-enterprise version.)

The only enterprise distros to show up at DistroWatch are CentOS, Lineox 
(currently at #31, up from #74 a year ago), and White Box (#43), aside 
from Red Hat and SUSE. (Some of the others listed there might classify 
themselves as enterprise grade, though they aren't clones of Red Hat or 
SUSE.)


As far as I can tell, there are no SUSE LINUX Enterprise clones. The 
only thing I could turn up was this article from about a year ago:

http://www.snowblink.co.uk/archives/2004/03/19/yast_gpled/

which says that YaST has been released under the GPL, and speculates 
that this makes a SUSE clone more likely.

The lack of any SUSE clones is another strike against SUSE from my 
perspective. (Though I've heard you can use SUSE distributions beyond 
their trial period. If this is legal, and they allow you access to 
updates (though of course no tech support), then this might be an 
alternative to using a clone.)


Raimo Koski, the creator of Lineox, another RHEL clone, has some 
interesting things to say in a whitepaper:

http://developer.lineox.org/white_paper.html

   ...the very aspects that make Linux desirable, its low cost, Open
   Source nature, and the way it gives customers more control over
   their software, are under attack by Linux vendors bent on increasing
   shareholder value. Businesses are paying more as Linux distributions
   demand a per-seat cost and service lock-in for software that they
   didn't develop and that others support.
[...]
   This has hampered the adoption of Linux. For example, a very large
   multinational bank recently informed that they had called off a
   10,000-system Linux deployment because "Linux is now more expensive
   than Windows". An ISP complained that the cost of Enterprise Linux
   is greater than the annual profit of one of his servers.
[...]
   ...the $1000 per year or greater that many customers now pay for
   their Linux systems goes not for service, but for a brand and the
   endorsement of a few application providers like Oracle.

An interesting point, though I'm not so sure he can convincingly argue 
that Lineox can provide the same level of support as Red Hat or Novell. 
In fact currently Lineox only provides an update subscription service, 
and no actual technical support.


   White Box Linux is a one man show... For whatever reason, [John
   Morris] has refused all offers of help, stating that White Box Linux
   is made for the library and will stay that way. The current (30.
   Oct. 2004) situation is that there are still packages from Update 3,
   which Red Hat released the 3rd Sep., which are not built for White
   Box Linux.

Seems like a valid criticism of White Box Linux, which doesn't seem to 
be gaining any ground (stuck at #43 for the last year per DistroWatch), 
though I don't see any good evidence that Lineox is much more than a 
"one man show." (According to their web site there are two founders and 
no mention of employees. The counter argument Lineox provides is that 
they get their updates out within days of Red Hat releasing them.)


   So CentOS (RHEL rebuild project hosted by cAos) should be more
   reliable than White Box because there is no single point of failure
   in human resources. There are however some uncertainties and
   questions when your operating system provider is a volunteer
   organization.
[...]
   A volunteer organization can't make any binding promises about the
   release speed and availability of products and updates to them. One
   has just to trust that the previous performance level is maintained.
   There is also the somewhat related inability to sign papers like
   NDAs, which are almost always required when dealing with commercial
   companies.

Valid points, but not a concern if CentOS is being used essentially as a 
stand in for RHEL. When the installation requires not just the software 
characteristic of enterprise software, but the business characteristics 
of it, then you pay the price and buy Red Hat.

Depending on your level of confidence in Lineox, a certain demographic 
might find that their low-cost (about $20/yr/machine) update 
subscription service is a good alternative.

It doesn't seem to provide any compelling advantage over CentOS from my 
perspective, with the one exception that Lineox might prove to be a far 
better business partner than a Red Hat or a Novell if you're a small VAR 
or consultant. For a client that couldn't afford Red Hat, I could see 
Lineox possibly being a better solution than CentOS.


   All other current RHEL clones are built by doing the changes to
   source files by hand. ... Lineox Enterprise Linux 4.0 will be built
   completely by a script set. This means that the build process is
   repeatable and the scripts act as a documentation. This approach
   enhances the quality of Lineox Enterprise Linux and makes the build
   process faster because the scripts can be prepared in advance using
   beta versions of RHEL.

Good point, though I'd be surprised if projects like CentOS were not 
heading in that direction as well.

However...

   Compared to Lineox approach all other clones have inherently lower
   quality. As an other example, CentOS was the first with Update 3
   after Lineox. Lineox had Update 3 based Always Current 4 days after
   source availability, CentOS was ready two weeks later. So Lineox is
   currently the only one with professional quality.

I wouldn't necessarily equate speed of releases with quality. Is 4 days 
really enough for Lineox to have confirmed that their cloning process 
hasn't introduced any unforeseen side effects?


   Where UserLinux fails first and foremost in my view is that it uses
   Debian as it's base. This just isn't palatable for the existing
   Enterprise Linux users because there is no upgrade path or even a
   possibility to create one. Debian uses dep package format while
   existing Enterprise Linux deployments use almost exclusively rpm
   format and there are currently no existing tools to upgrade rpm
   based installation to deb based one.
[...]
   It might help making Debian enterprise ready, but the response has
   been very lame so far (see the article Perens readies old-school
   Linux, but who wants it? for example).

I read more about UserLinux, which aims to be an enterprise distribution 
for Debian, at the UserLinux site (http://userlinux.com/) and some other 
articles. I agree with the gist of the above comments.

In some ways UserLinux seems to miss the point of what an enterprise 
distribution is. In my view the software characteristics (release cycle; 
duration of maintenance releases; etc.) are only a small part of what 
draws companies to enterprise distributions. The big attraction is the 
traditional large company desire to have some other party be responsible 
(legally and otherwise).

That means a community developed enterprise distribution isn't 
particularly appealing. They're good for small organizations that simply 
want enterprise-style software characteristics, but that's about it. The 
exception would be a community developed clone of a commercial 
enterprise distribution. This might have an appeal to some organizations 
as a lower cost alternative for non-critical systems, while providing 
uniformity with the official corporate distribution, so administration 
practices don't need to change, and systems can always be upgraded to 
the commercial distro if needed.

UserLinux might have some success at larger organizations if a strong 
network of VARs and consultants forms around it to provide that 
commercial grade support.


Other quotes on UserLinux:

http://www.vnunet.com/news/1157304

   And Gary Hein, analyst with the Burton Group, expressed doubt that
   there would be much demand for the product. "What is a justification
   for an enterprise to run UserLinux over Red Hat or SuSE? I don't
   think cost is a factor," he said.

   The price users pay for Red Hat and SuSE are "very palatable", he
   added. "Especially if you see companies like IBM and Oracle standing
   behind these distributions."


And:
http://www.linuxworld.com/story/37872.htm

   Bruce Perens writes: The solution here is obviously some front-end
   on top of apt, and Debian packages. It's really strange that people
   still complain about RPM dependencies, I don't understand why Debian
   was able to solve this so many years ago and Red Hat still has a
   problem.

The article was from a few years back, but is package management still 
an issue on non-Debian distributions these days? Doesn't YaST on SUSE 
and Yum on Red Hat provide a comparable solution to apt?


URLs for the clones:

http://www.centos.org/projects/centos
http://www.lineox.net/
http://www.xoslinux.org/
http://www.whiteboxlinux.org/
http://www.taolinux.org/

  -Tom