[Discuss] du question

[dan moylan]

john writes:
> I encountered a different situation that I feel
> illustrates du's behavior even more clearly. When
> transcoding a video with ffmpeg and monitoring progress in a
> separate xterm, I noticed that du reported the size as a
> series of doublings. It started out reporting that the
> output file was 4M, it stayed at 4M for a while, then jumped
> to 8M, then 16M, 32M, 64M, 128M, and 256M, and when the
> transcoding completed. it dropped to 212M. It seems that
> ffmpeg preallocates disk blocks for the output file,
> presumably to maximize the efficiency of processing
> streaming media, and it does so by doubling the number of
> preallocated blocks whenever it's close to running out.

> du reports on the number of disk blocks allocated, not the
> file size.

> Another example of this would be a sparse file. You could
> have, for example, a database file that ls -l reports as 100
> GB but where most of the zero blocks are not allocated.
> While the file size is 100 GB, as ls -l reports, it may
> happen that only 512M of disk blocks have been allocated, in
> which case du would show 512M.

> On Fri, Jun 9, 2017 at 3:52 AM, John Abreau <abreauj at gmail.com> wrote:

> > du measures disk usage, not individual file size. Two hard links to the
> > same underlying file in two different directories don't use twice the disk
> > space of one instance.

> > If you run two separate instances of du on each of the directories
> > containing the hard links to that file, each would report 35G. If you run a
> > single instance of du to measure both, it would report an aggregate total
> > of 35G; the first one displayed would show as 35G of disk space used and
> > the second as zero additional disk space used.

> > On Thu, Jun 8, 2017 at 5:35 PM, dan moylan <jdm at moylan.us> wrote:

> > > dan ritter writes:
> > > > On Thu, Jun 08, 2017 at 02:11:22PM -0400, dan moylan wrote:

> > > > > dan ritter writes:

> > > > > > > I don't know backintime, but I'm guessing it uses links to
> > > > > > > do file-level snapshots.   Bet there's a FAQ.

> > > > > > > 35G     20170607-120002-419
> > > > > > > 86M     20170607-230005-357

> > > > almost as if there were 86MB of changes in between the
> > > > first and second snapshots.

> > > of course, that's the whole point of backintime.  it
> > > was/is/has been obvious, even to me, but it is not the
> > > question and never has been.

> > > my question was and is: why does
> > >   du -s *
> > > show 86MG for the second and now third backups, and
> > >   du -s 20170607-230005-357
> > > as well as
> > >   du -hd1 20170607-230005-357
> > > show 35GB?

> > > this is not obvious to me -- if it is to you, please
> > > explain.

john, thanks for your insights -- most interesting.

ole dan

j. daniel moylan
84 harvard ave
brookline, ma 02446-6202
617-777-0207 (cel)
jdm at moylan.us
www.moylan.us
[no html pls]