[Discuss] File Systems vs.Databases Was: Re: SSH options (was Future of X11 (was Trying to connect to internet in Debian))

[dsr at randomstring.org (Dan Ritter)]

Kent Borg wrote: 
> They are different. A common case is to use them together. Store the
> metadata in a database (title, genre, date, running time, director, actors,
> screenwriter, MPEG path). And index most of that metadata to make it easy to
> search for things. But the actually fundamental data itself, the stuff we
> really care about (in this example MPEG data) is probably not going to be
> stored in the database, but will be opaque blobs---files--hashed into a
> directory path, stored in a file system.
> 
> Use the database for the stuff it is good at, use the file system for the
> stuff it is good at. Appreciate the difference.

Yup. This is a very common and useful pattern. Examples:

notmuch, mairix, and maildir-utils all do about the same job:
generate an indexed database of selected mail headers, then make
them available for searching. The output is a list of maildir
message paths, often symlinked into a temporary maildir so you
can act on it as though it were a new folder full of the
messages you just selected.

owntone is a music server which builds an indexed database of
metadata about your music files, then gives you a dynamically
generated webpage for your searches, plus player and control
tools. Files remain where they are.

jellyfin does the same for music, video, and audiobooks.

ubooquity does the same for ebooks and comic books.

plocate (successor to mlocate, locate, etc...) builds a database
of every file in your system, so that you can query it when you
don't know where a file is in the tree.

The primary advantage of all of these systems is that when
something better comes along, re-indexing the files with the new
system is *much easier* than trying to export and import all the
data from one database to another. And, of course, they all get
a free layer of cross-compatibility for dealing with the actual
data - moves, name changes, copies, backups, deletions.

-dsr-