[Discuss] Big Data obsession

[gaf at blu.org (Jerry Feldman)]

On 08/04/2012 11:24 AM, Mark Woodward wrote:
> In our drawn out discussion about databases, and most discussions
> about databases this happens as well, the subject of HUGE scalability
> was trotted out. Now I am by no means dismissing "big data" as a real
> problem. Seriously, I worked on some pretty large systems -- hundreds
> of servers. It is a complicated problem. Not only do you need to get
> the most out of each system, but you have to make many systems work as
> a single logical one.
>
> There are some businesses that really need this functionality and
> scale, but, the problem is that 99.99% of the software being developed
> will NEVER EVER scale to that size. The developers are so eager to
> solve that problem that they forgo more practical designs. Also, no
> matter what you do, it will take a lot of time to grow to that size!
> So, even if it is envisioned in the business plan, you'll have
> probably cashed out your stock options and be living on an island
> before you need to develop it.
>
> My favorite example is facebook. Yes, they are a big data show case.
> OMFG they have a lot of data and a lot of computational requirements.
> They did not start out dreaming of big data. It started small and
> grew. I believe that this inadvertent strategy helped them greatly. By
> focusing on the site and what it did and *not* how to make it scale
> until scalability was needed they were able to be attractive to more
> users more quickly. --
Small companies that grow certainly have scalability issues. I work for
IBM and they certainly have Big Data issues.