[Discuss] time-warp

[worley at alum.mit.edu (Dale R. Worley)]

About a decade ago, I set up a custom-made backup system.  It was
based on an idea a friend told me about: A friend of his had set up
his home directory as a working directory under the Subversion version
control system, and he would periodically add commits to the
Subversion repository that were snapshots of his entire home
directory.

Since I was learning Git at the time, I implemented the idea using
Git.  It turns out to be quite straightforward, though you likely want
to add gitignores for particular files and directories that change
frequently and/or contain low-value data.

Initially, I manually created commits roughly daily.  After about four
months, I set up a cron job that created a new commit each *minute*.
It turns out Git has some sort of timestamping system that allows it
to check a lot of directories and files for changes very quickly, so
the load from doing "git add --all; git commit" each minute is
minimal.  Also, by default, Git will refuse to create a new commit if
the file contents are the same as the previous commit, so commits are
created only for minutes during which logged files change.

I wrote a script that once a week goes through the archive and prunes
out some of the older commits.  The rule is that the spacing between
successive commits should be no more than 1/N of the age of the
commits.  I've set N to 365, so commits from a year ago are spaced at
least a day apart and commits back in the beginning are spaced at
least 10 days apart.  This allows the archive to keep a reasonably
thorough history without letting the archive grow without limit.
Theoretically, the archive grows as the log of the total age of the
archive -- log(age of the archive / commit interval) * N.

Currently, I've been running time-warp since Feb 2013, just over 10
years ago.  Theoretically it should have 5,600 commits; in reality, it
has 2,800.  Presumably the difference is because I haven't changed
files every minute for the past 10 years.  The archive is now 3.4 GiB.
I burn a copy of the archive to DVD-ROM each week.

I can send you a copy of the scripts if you want to use time-warp.

Dale