[Discuss] Stalwart E-mail; Was: Re: SSH options

[ron at bclug.ca (Ron)]

Rich Pieri wrote on 2026-01-22 08:02:

> You dropped the "arbitrary". When everything fits neatly into
> tables or key/value stores then sure, a database might work. Email
> messages are not neat. They are very much like medical records:
> arbitrary in size and structure. Few databases can deal well with
> this kind of data.

Quite a few actually *can* deal with this kind of data; it's been a
fairly mature field for a decade or more already.


You can read about databases designed to work with unstructured data at
your leisure:

https://en.wikipedia.org/wiki/NoSQL


> You can read up on all of Oracle's failures in the EMR space, and 
> the history of WinFS, at your own pace, as examples of this.

So, two companies that are bad at software had failures, so in this
field a database is doomed to failure?  Disagree.


What about Gmail? They use a database (from which RocksDB was forked) 
and it undeniably works:

> Google Infrastructure Supporting Gmail
> 
> Technology    Purpose
> 
> MapReduce     Processes large volumes of data, such as email indexing
> 
> BigTable      Stores structured metadata and user preferences

https://www.dhiwise.com/post/understanding-gmail-architecture-a-
comprehensive-guide








In this case, Stalwart is attempting, with funding from the European 
Commission?s Next Generation Internet programme? and GitHub's OSSF (Open 
Source Secure Fund)?, to replace reliance on Gmail / Yahoo / Outlook / 
iCloud / etc.





With the goal of competing on a scale of

> to host hundreds of millions of email accounts reliably? How do they
> store petabytes of messages, survive hardware failures without 
> losing data, and keep spam at bay across billions of daily 
> deliveries?

the thought of billions (or millions) of Maildir files per day is laughable.


A *lot* of people use web mail, so searching those millions / billions 
of messages *must* be fast.




I'm not yet sold on converting to Stalwart - it's extremely promising 
and that feels uncommon these days.


But I've encountered some issues that may be its fault, or the reverse 
proxy's fault, or KDE's fault, or my fault,... Undiagnosed.


And the schema for their data is all serialized key/value storage, even 
in PostgreSQL. That bothers me a bit and I'll have to evaluate some more.


What these developers have created in a short time is so highly polished 
and feature complete that I have nothing but respect for their abilities 
and reasonable confidence in their skills & choices.

We'll see.




Their discussion / recommendations on data storage are here:

https://stalw.art/docs/install/store/


? https://stalw.art:8443/blog/github-ossf#about-githubs-ossf

? https://stalw.art/blog/nlnet-grant-collaboration/