[Discuss] Gluster startup, small-files performance

[richb at pioneer.ci.net (Rich Braun)]

Greetings...after concluding cephfs and ocfs2 are beyond-the-pale too
complicated to get working, and xtreemfs fails to include automatic healing,
I'm back to glusterfs.

None of the docs seem to address the question of how to get the systemd (or
sysvinit) scripts working properly so I've had to customize my own. 
Basically, the boot sequence has to be this:

  Mount local volumes (including the "bricks" which are xfs)
  Start up networking
  Initialize glusterd
  Wait for glusterd to come up
  Mount gluster volumes
  Launch LXC and/or VirtualBox instances

It's that 4th step which isn't obvious:  if I don't insert some sort of a
wait, the mount(s) will fail because glusterd isn't fully up-and-running. 
Also, in step 5 if I don't wait for all the volumes to mount, one or more
instances come up without their dependent volumes.

I'll include my cobbled-together systemd scripts below.  Systemd is new enough
that I'm not completely familiar with it; for example it's easy to get a
circular dependency and so far the only way I've found to debug it is
trial-and-error, rebooting each time.

Googling for performance-tuning on glusterFS makes me shake my head,
ultimately the technology is half-baked if it takes 500ms or more just to
transfer an inode from a source path into a glusterFS volume.  I should be
able to "cp -a" or rsync any directory on my system, whether it's got 100
files or 1 million files, into a gluster volume without waiting for hours or
days.  It falls apart completely after about 10,000 files, which means I can
only use the technology for a small portion of my deployments.  That said, if
any of y'all are *actually using* glusterfs for *anything* I'd love to share
stories.

-rich

-----------------------glusterd.service-----------------------
[Unit]
Description=Gluster elastic volume management daemon

[Service]
ExecStart=/usr/sbin/glusterd -N

[Install]
WantedBy=multi-user.target


-----------------------gluster-vols.service-------------------
[Unit]
Description=Mount glusterFS volumes
Conflicts=shutdown.target
After=glusterd.service
ConditionPathExists=/etc/fstab

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c "sleep 10;/usr/bin/mount -a -O _netdev"

[Install]
WantedBy=multi-user.target


-----------------------lxc at .service----------------------------
[Unit]
Description=Linux Container
After=network.target gluster-vols.service

[Service]
Type=forking
ExecStart=/usr/bin/lxc-start -dn %I
ExecStop=/usr/bin/lxc-stop -n %I

[Install]
WantedBy=multi-user.target