[Discuss] Cloud backup

[David Kramer]

I apologize I had on my list to send you this information since the 
Installfest and failed to do so.  I'll post it here to share with the 
larger audience.

I found the very cheapest storage of backup files you will hopefully 
never need turns out to be Google Cloud Platform.  Right now I'm only 
backing up my Linode server there automatically but I want to get 
regularly scheduled backups from other machines too. The cost is like... 
20 cents a month for daily backups of most of the important parts of the 
system.  You can configure the buckets to automatically delete files 
over a certain age so you don't even need to script that. HOWEVER, and 
this is a little non-intuitive, the cheaper the storage rate you choose, 
the longer you must keep the files there, because they want to reduce 
transactions.  It literally won't let your remove a file from cold 
storage that's been there for less than (IIRC) 90 days.  Fine by me.

I back up my database and mail directories daily, and do file level 
backups of the important directories weekly.  In all cases, I tar up the 
files, then encrypt them with a password using openssl AES256.  Then I 
used GCP's gsutil to copy the files into the right bucket.  I found 
gsutil very easy to use after some initial experimentation.  You do need 
to do some magic to connect your install to your account so you don't 
need to specify credentials with each call but that wasn't too bad. 
https://cloud.google.com/storage/docs/gsutil

Being a software engineer, I moved a lot of the functionality into one 
script, backuplib.sh, so there's very little duplication between the 
scripts, and it all just works once I get it working for one script.  I 
can post more examples if you want but here are some of the key 
functions in that file:

function makeBackupName {
     /bin/echo "${BACKUP_DIR}/backup_${1}_${HOSTNAME}_${DATE}.${2}"
}

function makeBucketName {
     /bin/echo "gs://dkramer_${HOSTNAME}_backups/"
}

# encryptBackup INPUTFILE OUTPUTFILE
function encryptBackup {
     testBackupExists ${1}
     /usr/bin/openssl enc -aes-256-cbc -salt -pbkdf2  -in ${1} -out ${2} 
-pass env:ENC_PASSWORD
     testBackupExists ${2}
     /bin/rm ${1}
     /bin/chmod 700 ${2}
}

# uploadBackup FILENAME
function uploadBackup {
     bucketName=`makeBucketName`
     /usr/bin/gsutil cp ${1}  "${bucketName}"
     echo "Space used:"
     gsutil du -sh "${bucketName}"
}

# removeOld DAYS WILDCARD
function removeOld {
     /usr/bin/find ${BACKUP_DIR} -name "${2}" -mtime +${1} -delete
}

Given that, the entire script to back up my system files (minus some 
other things I do to document the current state of the system):

backup=`makeBackupName system tgz`
backupenc="${backup}.enc"
tar cvzf ${backup} --exclude='var/spool/mail/' 
--exclude='var/log/journal'  etc home root usr/local var/www var/spool 
var/cache var/log
encryptBackup ${backup} ${backupenc}
uploadBackup ${backupenc}
removeOld ${BACKUP_DAYS} 'backup_system_*.tgz.*'

Backing up mail is similar but just under /var/mail, and skipping some 
large non-critical files.

Backing up the databases is a little more complicated because my server 
is using MySQL for some things and Postgresql in others, and in both 
cases I query the list of databases dynamically then generate a dump of 
each one, so there's no central list of databases to maintain.

The backups are called by cron of course.

If there's anything you would like me to dig into deeper, let me know.


On 12/14/19 10:17 AM, Jerry Feldman wrote:
> I currently have Dropbox professional and Google drive (free). I snapshot
> backup my tower to a local hard drive. I'm thinking of backing up the most
> recent snapshot to either Dropbox or Google. The advantage of Dropbox is
> that it supports Linux so I could either tar or cp or rsync to a folder in
> my local Dropbox directory, and it would automatically sync to the Dropbox
> cloud. With Google I could use deja dup. I certainly could use aws, but I
> already have Dropbox and Google accounts. I'm just looking for opinions on
> both.
>
> --
> Jerry Feldman <gaf.linux at gmail.com>
> Boston Linux and Unix http://www.blu.org
> PGP key id: 6F6BB6E7
> PGP Key fingerprint: 0EDC 2FF5 53A6 8EED 84D1  3050 5715 B88D 6F6
> B B6E7
> _______________________________________________
> Discuss mailing list
> Discuss at lists.blu.org
> http://lists.blu.org/mailman/listinfo/discuss