[Discuss] Big Data obsession

[markw at mohawksoft.com (Mark Woodward)]

In our drawn out discussion about databases, and most discussions about 
databases this happens as well, the subject of HUGE scalability was 
trotted out. Now I am by no means dismissing "big data" as a real 
problem. Seriously, I worked on some pretty large systems -- hundreds of 
servers. It is a complicated problem. Not only do you need to get the 
most out of each system, but you have to make many systems work as a 
single logical one.

There are some businesses that really need this functionality and scale, 
but, the problem is that 99.99% of the software being developed will 
NEVER EVER scale to that size. The developers are so eager to solve that 
problem that they forgo more practical designs. Also, no matter what you 
do, it will take a lot of time to grow to that size! So, even if it is 
envisioned in the business plan, you'll have probably cashed out your 
stock options and be living on an island before you need to develop it.

My favorite example is facebook. Yes, they are a big data show case. 
OMFG they have a lot of data and a lot of computational requirements. 
They did not start out dreaming of big data. It started small and grew. 
I believe that this inadvertent strategy helped them greatly. By 
focusing on the site and what it did and *not* how to make it scale 
until scalability was needed they were able to be attractive to more 
users more quickly.

Opinions?