[Discuss] Issuing the 'sync' command more than once (and a tangent on how not to run a high-tech company)

[gaf at blu.org (Jerry Feldman)]

On 06/16/2012 12:59 PM, MBR wrote:
> On 6/12/2012 11:22 PM, Jack Coats wrote:
>> In old SunOS days, we could issue the 'sync' command, twice, to ensure
>> all system
>> buffers had been written to disk.  You could experiment to see if
>> issuing it occasionally
>> in your script helps.  Or issue it outside the script, even in a chron
>> might help.
>
> Actually, calling 'sync' multiple times from a script really won't
> help.  To the best of my knowledge, no Unix kernel has ever contained
> code that counts the number of times sync() (the system call that the
> 'sync' command issues) has been called.  But I do know something about
> the history of the myth that typing 'sync' multiple times guarantees
> that all system buffers have been written to disk.  I'm afraid I
> couldn't keep myself from digressing a bit into the history of an
> obscure and thankfully long-defunct Unix desktop company.  Feel free
> to skip the next two paragraphs.
>
> I first encountered Unix in 1980, when I went to work for Fortune
> Systems in Foster City, CA. A little company history here. Sorry for
> diverging so far from the 'sync' command, but Fortune is a cautionary
> tale in how not to run a high-tech company.  Fortunewas one of the
> first companies to build a Unix desktop computer. The company had some
> of the best Unix technical people I've ever met, and three founders
> who had very little interest in building a viable company or product. 
> Their primary interest was in having a company to take public when
> market hysteria for high tech was at its peak.  (See
> http://www.old-computers.com/museum/computer.asp?c=767 and
> http://articles.latimes.com/1985-01-08/business/fi-7514_1_computer-systems
> for more.  Oh, and there's also
> http://www.fundinguniverse.com/company-histories/Itel-Corporation-Company-History.html
> which recounts how before starting Fortune, its president had managed
> to lead his previous company, Itel (not to be confused with I*n*tel),
> into what at the time was the largest bankruptcy ever in U.S.
> history.)  The head of Fortune's Unix kernel group knew Bill Joy
> lobbied hard to hire him.  But he couldn't convince Fortune's
> foundersto do so!  A year or two later, Joy founded Sun Microsystems!
>
> While the hardware guys were still designing the Motorola 68000-based
> architecture that would eventually become Fortune's product, we
> software guys did all our development on a VAX running 4.1bsd.  (YAT
> (Yet Another Tangent): 4.1bsd was before Unix had the networking
> functionality currently provided by socket calls.  If I remember
> correctly, the socket API didn't appear until Berkeley's 4.2bsd
> release around 1982 or 1983.)  But the kernel Fortune was porting to
> their own hardware was not 4.1 bsd.  It was Version 7 straight out of
> Bell Labs, which was the code that the Berkeley guys had enhanced to
> create 4.1bsd. Shortly after the language tools guys managed to modify
> the C compiler on the VAX into a cross-compiler that could emit 68000
> code, the hardware guys produced some prototype hardware for us to try
> running our code on.
>
> _And now, back to 'sync' (finally ...)_
>
> I vividly remember sitting in front of a breadboard on a workbench
> with one of the OS guys, a recent Berkeley computer science grad,
> who'd just gotten the cross-compiled kernel working on the new
> hardware, while I debugged the word processor code application I was
> writing.  At the end of that session he told me that it was important
> to type the 'sync' command _*three times*_ before shutting the system
> down.  That seemed odd to me.  "Why *_three_* times?" I asked him.  He
> explained to me that 'sync' at the command line issues the sync()
> system call, and if I read the sync(2) manpage carefully I'd find that
> sync() does /not/ cause the kernel's buffers to be written out to
> disk.  Instead, it tells the kernel to /schedule its buffers to be
> written out to disk as soon as the kernel can get around to it./ 
> There /was/ no system call to tell the kernel to write its buffers out
> to disk right away and not return to the calling application until
> they were written out.  What happens if you run 'sync' more than
> once?  The first 'sync' tells the kernel to schedule the buffers to be
> written out.  Any subsequent 'sync' before the kernel has gotten
> around to writing the buffers just sets the "flush buffers to disk
> when the kernel has time" flag, that's already set to true, to true
> again. Any subsequent 'sync' after the kernel has flushed the buffers
> schedules any as-yet-unflushed buffers, of which there are now none,
> to be written to disk.  All syncs after the first one are effectively
> no-ops.
>
> So, why type 'sync' three times?  Turns out, somebody at Berkeley had
> timed how long it took the average typist to type 'sync' and compared
> that to how long it typically took the kernel to finish the scheduled
> write operations.  They'd figured out that typing 'sync' three times
> gave you enough margin for error that you could have a high level of
> confidence that all kernel buffers would be written out to disk before
> the user finished typing.
>
> The following is from the "Bugs" section of
> http://linux.die.net/man/2/sync:
>
>    According to the standard specification (e.g., POSIX.1-2001), sync()
>    schedules the writes, but may return before the actual writing is
>    done. However, since version 1.3.20 Linux does actually wait. (This
>    still does not guarantee data integrity: modern disks have large
>    caches.)
>
>
>
The sync() system call causes the buffers to flush to disk immediately.
On older Unix files systems and possibly on EXT2, you could lose data
when you issued a halt(1). Issuing a sync from a script makes very
little sense at all on any modern Unix and Linux file system. There used
to be a daemon that did this every 30 seconds, but I believe that is now
a kernel daemon kdmflush.

-- 
Jerry Feldman <gaf at blu.org>
Boston Linux and Unix
PGP key id:3BC1EB90 
PGP Key fingerprint: 49E2 C52A FC5A A31F 8D66  C0AF 7CEA 30FC 3BC1 EB90