How to best do zillions of little files?

[Nathan Meyers]

On Wed, Oct 02, 2002 at 10:53:58AM -0400, Scott Prive wrote:
> I haven't done a bit of work in this area, but I have read how the embedded and floppy Linux systems work: they conserve space (other filesystem reasons also?) by creating a monolithic file that handles everything, and just create links from the "files" to the file that has everything.

Sounds like a DBMS :-).

> > -----Original Message-----
> > From: John Chambers [mailto:jc at trillian.mit.edu]
> > Sent: Wednesday, October 02, 2002 10:40 AM
> > To: discuss at blu.ORG; discuss at blu.ORG
> > Subject: How to best do zillions of little files?
> > 
> > ...
> > 
> > There's also the possibility of trying the DB systems, but it'd be  a
> > bit  disappointing  to spend months doing this and find that the best
> > case is an order of magnitude slower than the  dumb  nested-directory
> > approach.   (I've  seen this already so many times that I consider it
> > the most likely outcome of storing files as records in a DB.  ;-)

At the risk of apostasy... have you tried this with commercial DBMSes
or only free ones?

There are many ways to make DBMSes a slow solution, including bad design
of the database, naive management of connections and other resources, and
failure to do routine DB maintenance. A well designed and administered
DBMS solution should do very well on this sort of problem.

Nathan Meyers
nmeyers at javalinux.net