[Discuss] systemd reboot

[smallm at sdf.org (Mike Small)]

Steve Litt <slitt at troubleshooters.com> writes:

> On Sat, 03 Mar 2018 02:09:21 +0000
>> I see behaviour where if I change something under /etc/grub.d/, run
>> update-grub and then immediately run /sbin/reboot, upon start up grub
>> sees the old grub.cfg not the new one. 
>
> How do you know that? Did you back up the old grub.cfg, run update-grub
> and reboot, and then do a diff on the current and backed up grub.cfg
> and find them the same? If not, perhaps there are other reasons
> alterations of contents of /etc/grub.d don't appear to be recognized.

This is what made me think shutdown write cache syncing was an issue:

- what's involved is a script that updates a file under /etc/grub.d with
a new menu entry, runs update-grub and sets the new entry as the default
using grub-reboot.
- Intermittently (on some runs and not others, but usually within 4 or 5
tries, if I have my environment just so) the grub.cfg on reboot lacks
that menu entry. I.e. it looks like it did before.
- a debugging message after update-grub is run shows an md5sum for
/boot/grub/grub.cfg different from that which I see for the file after
rebooting.
- when run with additional debugging messages added later in the script the
problem no longer happens. i.e. grub.cfg will have the new entry and
across 90 tries.
- making the script run sync(1) just before it runs reboot similarly makes the
problem go away.
- this script had previously run on a Ubuntu Trusty derived system (but
with sysvinit used in place of upstart) without this issue as far as I
know.

It's curious though, systemctl does seem to call sync(2). When run as
the reboot symlink it eventually hits this code path:

static int halt_now(enum action a) {
        int r;

        /* The kernel will automaticall flush ATA disks and suchlike
         * on reboot(), but the file systems need to be synce'd
         * explicitly in advance. */
        if (!arg_no_sync)
                (void) sync();
...

It's not quite clear to me yet how arg_no_sync gets set. The code
suggests -n but the man page says differently. Regardless, I'm not seeing
how running reboot gets systemctl any extra args.

Or at least that's how it is in systemd 232 (what's in Debian
Stretch). At work I would have a different version perhaps, though I
think Xenial and Stretch are closely related.

I had read sync(2) on some systems only queues up the writes and
returns, but Linux seems to give stronger guarantees:

  According  to  the  standard specification (e.g., POSIX.1-2001), sync()
  schedules the writes, but may return before the actual writing is done.
  However  Linux  waits  for I/O completions, and thus sync() or syncfs()
  provide the same guarantees as fsync called on every file in the system
  or filesystem respectively.

So it's all a bit puzzling.  sync; sync; is better than sync? Or what,
should I do what was suggested in olden time and sync; sync; sync?

-- 
Mike Small
smallm at sdf.org