The Cult of Gary

I started backing up our servers to S3 a few weeks ago. I had been doing weekly tar based backups to my laptop over SSH. Some of the backups are starting to get big, so it was time to move to something else.

My key requirements were to back up to S3 and to use encryption on the backups. I found a lot of apps/scripts/products out there that can do this. I had two “nice to have” requirements as well. I wanted to be able to automated the backups as much as possible. Also, I should be able to download and store the backups on my laptop as an additional copy.

I first tried Brackup. It worked OK and has some really nice features. For example, it can do synthetic backups, so you only need to do one full backup.  There are a few design issues with Brackup though.

For one, you can’t purge old backup files when using GPG encrypted backups. The catalog files are encrypt. The backup agent can’t decrypt the catalog files and figure out what files can be deleted. I also found my configuration files were huge. Directories and destinations get configuration blocks. Each directory you want to back up needs its configuration. You can have a single S3 destination, but if you want to be able to purge backups, you need to back up each directory to it’s own destination. I had 5-10 directories on each server to back up.

I gave up on Brackup and figured I’d write a shell script to tar and upload backups. Before I started coding, googled to see if anyone else had already written an app. I found s3napback. It’s a little more traditional; s3napback treats S3 like a tape device (using js3tream) and chains tar, gzip and gpg together to create full/incremental backups. 

It uses apache style configs that are nice and concise. S3napback is also much faster that Brackup. It doesn’t need to perform a new S3 transaction for each file. Plus, I can download the stream files and restore without any special tools on any machine that has tar and gpg.