[Discuss] File Systems vs.Databases Was: Re: SSH options (was Future of X11 (was Trying to connect to internet in Debian))

[slitt at troubleshooters.com (Steve Litt)]

Dan Ritter said on Fri, 23 Jan 2026 09:50:40 -0500

>Kent Borg wrote: 
>> They are different. A common case is to use them together. Store the
>> metadata in a database (title, genre, date, running time, director,
>> actors, screenwriter, MPEG path). And index most of that metadata to
>> make it easy to search for things. But the actually fundamental data
>> itself, the stuff we really care about (in this example MPEG data)
>> is probably not going to be stored in the database, but will be
>> opaque blobs---files--hashed into a directory path, stored in a file
>> system.
>> 
>> Use the database for the stuff it is good at, use the file system
>> for the stuff it is good at. Appreciate the difference.  
>
>Yup. This is a very common and useful pattern. Examples:
>
>notmuch, mairix, and maildir-utils all do about the same job:
>generate an indexed database of selected mail headers, then make
>them available for searching. The output is a list of maildir
>message paths, often symlinked into a temporary maildir so you
>can act on it as though it were a new folder full of the
>messages you just selected.
>
>owntone is a music server which builds an indexed database of
>metadata about your music files, then gives you a dynamically
>generated webpage for your searches, plus player and control
>tools. Files remain where they are.
>
>jellyfin does the same for music, video, and audiobooks.
>
>ubooquity does the same for ebooks and comic books.
>
>plocate (successor to mlocate, locate, etc...) builds a database
>of every file in your system, so that you can query it when you
>don't know where a file is in the tree.
>
>The primary advantage of all of these systems is that when
>something better comes along, re-indexing the files with the new
>system is *much easier* than trying to export and import all the
>data from one database to another. And, of course, they all get
>a free layer of cross-compatibility for dealing with the actual
>data - moves, name changes, copies, backups, deletions.
>
>-dsr-


Not only that, but from my Maildir root I just did a find . | xargs
grep -ir golug * and in 21 minutes and 53 seconds it found the 662606
emails out of 814739 total emails containing "golug" in any case
pattern. Yeah, one wouldn't want to do this often, but it's a pretty
good fallback I use when worst comes to worst. Also, after the first
time I ran this command, doing it again happened in less than a minute.
And it took 27.2 seconds to subsequently find the 11187 files containing
"nvidia". Of course I have 64GB RAM, but how many people do you know
who store over 800K emails on a 7200RPM spinning rust?

So unless you're storing millions of emails for lots of people, this is
a fallback when something like plocate fails or isn't feasible.

The beauty of text files is you can get at them from several different
ways.

SteveT

Steve Litt 

http://444domains.com