[Discuss] Mirroring of files to all nodes in a cluster

[richb at pioneer.ci.net (Rich Braun)]

David Rosenstrauch <darose at darose.net> asked:
> I'm looking for a Linux utility that will allow me to mirror/replicate a
> directory tree onto each and every node in a cluster

I had to figure this out first for my home email server (pair of physical
boxes) and then for a Docker swarm cluster. I've published the resulting
script after years of doing this (as a plug&play docker container) at:

  https://github.com/instantlinux/docker-tools/tree/master/images/swarm-sync

It uses unison, a package similar to rsync but works bidirectionally. I have
to set up each node as a peer with one of the other members: there's not a
single point of failure, but it's not quite as slick as a real clustering file
system.

Alas all the real-world clustering filesystems are either impractical or
pricey. Of the open-source ones, I've spent weeks trying out Ceph, XtreemFS,
OCFS-2, and a few others; and I ran GlusterFS for about a year. (At a past
workplace, I used DRBD but concluded it's far more trouble to maintain than
the benefit it offers.) GlusterFS just plain sucks, there's really nothing
good at all about it: performance is beyond terrible, it won't scale, it has a
tendency to silently fail (drops files randomly, and you discover it only by
adding your own monitoring scripts), recovering from events often leads to a
split-brain scenario, etc.

Unison just-plain-works. I've never had an issue in half a decade of running
it. And its performance is wire-speed. I wish it could do many-to-many
clustering, but given the complexities of the clustering systems, maybe it's a
good thing they never tried clustering. One-to-one peering is drop-dead simple
to understand.

-rich