Backing up with 'Obnam'

I've recently been looking at my backup strategy, as I now believe that its very wasteful of resources to have 14 copies of something as I have at present. That's 14 copies of an email, document, or anything really. So I've been looking about for something

that will allow me to have just one copy of something
no incremental backups
not to use a database for its backend
and for it to be free.

A pretty tall order I thought.

I browsed through many sites on my quest, but none fulfilled my requirements, except for one called Obnam. This billed itself as "Obnam is an easy, secure backup program. Backups can be stored on local hard disks, or online via the SSH SFTP protocol. The backup server, if used, does not require any special software, on top of SSH.

Some features that may interest you:

Snapshot backups. Every generation looks like a complete snapshot, so you don't need to care about full versus incremental backups, or rotate real or virtual tapes.
Data de-duplication, across files, and backup generations. If the backup repository already contains a particular chunk of data, it will be re-used, even if it was in another file in an older backup generation. This way, you don't need to worry about moving around large files, or modifying them.
Encrypted backups, using GnuPG."

Now this sounds interesting, so I downloaded it from the Debian repos, and installed it and started to play. The documentation is a bit sparse, but the man page is very good and easily readable, but there is no sample configuration which shows the various options or ways of writing something that obnam would understand. But, they have a very good mailing list which with their knowledgeable people mean that you can get answers to most of your questions.

Obnam has its own site at Obnams home page http://liw.fi/obnam/, and has as its requirements - libc6 (>= 2.6), python (>= 2.6.6-7~), python (<< 2.8), python-larch (>= 1.20120527~), python-ttystatus (>= 0.19~), python-paramiko, python-tracing (>= 0.6~), python-cliapp (>= 1.20120630~) which are all easily fulfilled from the Debian repos.

When deleting an old backup recently I was shocked to discover that it maxed out at 81 Gigabytes! Multiply that by the 14 other backups that you're saving, and you begin to see how wasteful it actually is.

One of my gripes against obnam is that I cant find any configuration files online, neither on its site or anywhere else through google. So, this is my conf file;

Obnam conf

[config]

repository = /media/backup/obnam-home/
root = /home/boudiccas/
#exclude = .mp2$, .part$, .rar$, .nfo$, .m4v$
exclude-caches = false
keep = 8h,14d,10w,12m
log = /home/boudiccas/cron/obnam.txt
log-level = info
log-keep = 5
log-max = 50 mb
log-mode = 0600

As can be seen this backs up my $HOME every 2 hours from a cron job, and is quite happy just running in the background and firing itself off as required. The biggest backup is the first one, after that it only saves the file changes. Nothing is excluded from this backup, but I have commented out the 'exclude' section so that you can see what format it should take. I also back up my /var and /etc but this has a separate conf file as its using root permissions;

[config]

repository = /media/backup/obnam-etc-var/
root = /etc, /var
#exclude = .mp2$, .part$, .rar$, .nfo$, .m4v$
keep = 8h,14d,10w,12m
log = /home/boudiccas/cron/obnam-etc.txt
log-level = info
log-keep = 5
log-max = 50 mb
log-mode = 0600

The difference here is on line 5 'root = /etc, /var'. This is the only way I've found of having 2 directories to be backed up, it is impossible to add /var to an existing backup of /etc, you would have to restart with a new config and repository.

The backup of /home has this as its cron line - 00 */2 * * * DISPLAY=:0 /home/boudiccas/cron/notify-send and of /var and /etc - 30 05,23 * * * DISPLAY=:0 /home/boudiccas/cron/notify-send2. From both these lines you can immediately see something new and different, that of DISPLAY=;0. Having this as part of your cron line it shows a pop-up bubble when it starts, and another one telling you that its finished at the end. Its also calling 'notify-send' which is -

Obnam shell script

#!/bin/sh

notify-send "Starting main backup..."

obnam backup

if [ "$?" -ne 0 ]; then
  notify-send "Unable to finish main backup."
  exit 1
else
  notify-send "Finished main backup."
fi

This tells the pop-up bubble what to say, and runs obnam backup.

But obnam can also be run from the command line, and these are some of the commands that I've found useful -

obnam --config=~/cron/.obnam.conf backup Note this is just for testing purposes when deciding whether I want to use it or not

obnam ls>~/cron/backedup-obnam.txt Note gives you a hardcopy of obnams generations/backups

I have also extended the usage of obnam for my own benefit. I don’t like having to generate a file to show me where some item is in the repos so that I can restore it, So I've written two scripts, which are here;-

obnam-today

#!/usr/bin/env bash
#: Title            : obnam-today
#: Date             : June 2013
#: Author           : Sharon Kimble
#: Version          : 2.0
#: Description      : to give a complete dated list of all of obnams savings, run from $USERS cron
#: Options          : 2, all listed in variables
#: License          : GNU GPL 3.0 or later
####################################################
# Variables
NUM="7"     #number of backup files to keep
LOGFILE="/home/boudiccas/cron/obnam-running.txt"
####################################################

# Change into new directory
cd ~/cron/obnam

# Generate new file and save it
obnam ls>"obnam-"`date +"%Y-%m-%d.txt"`

# Delete old backups!
find ~/cron/obnam -type f -mtime +$NUM -name 'obnam-*.txt' -exec rm -v {} + >>$LOGFILE 2>&1

This creates a new file in ~/cron/obnam/ along the format of 'obnam-DATE', and saves it for 7 days, and then its auto deleted by the script. The script name is show in 'Title'.

obnam-etc

#!/usr/bin/env bash
#: Title            : obnam-etc
#: Date             : June 2013
#: Author           : Sharon Kimble
#: Version          : 2.0
#: Description      : to give a complete dated list of all of obnams savings, run from $ROOT cron
#: Options          : 2, all listed in variables
#: License          : GNU GPL 3.0 or later
####################################################
# Variables
NUM="7"     #number of backup files to keep
LOGFILE="/home/boudiccas/cron/obnametc-running.txt"
####################################################

# Change into new directory
cd ~/cron/obnam

# Generate new file and save it
obnam ls>"obnametc-"`date +"%Y-%m-%d.txt"`

# Delete old backups!
find ~/cron/obnam -type f -mtime +$NUM -name 'obnametc-*.txt' -exec rm -v {} + >>$LOGFILE 2>&1

Obnam log file

This is just the end of my 'obnam.txt' which shows the output of backing up my $HOME, which shows how quick it is and how much information its giving you ;-

2013-06-30 10:05:03 INFO * files found: 101946
2013-06-30 10:05:03 INFO * files backed up: 11043
2013-06-30 10:05:03 INFO * uploaded data: 286216866 bytes (272.957674026 MiB)
2013-06-30 10:05:03 INFO * duration: 299.039324999 s
2013-06-30 10:05:03 INFO * average speed: 934.688634026 KiB/s
2013-06-30 10:05:03 INFO Backup finished.
2013-06-30 10:05:03 INFO obnam version 1.4 ends normally
2013-06-30 10:05:06 INFO obnam version 1.4 starts
2013-06-30 10:05:06 INFO Forcing lock
2013-06-30 10:05:06 INFO Repository: /media/backup/obnam-home/
2013-06-30 10:05:06 INFO Client: london
2013-06-30 10:05:06 INFO clientlist is not locked
2013-06-30 10:05:06 INFO chunksums is not locked
2013-06-30 10:05:06 INFO chunklist is not locked
2013-06-30 10:05:06 INFO chunks is not locked
2013-06-30 10:05:06 INFO . is not locked
2013-06-30 10:05:06 INFO 11926674304779044483 is not locked
2013-06-30 10:05:06 INFO obnam version 1.4 ends normally

And from the email that it sends you ;

Obnam email

Subject: Cron  DISPLAY=:0 /home/boudiccas/cron/notify-send
Date: Sun, 30 Jun 2013 10:05:07 +0100

Backed up 11043 files (of 101946 found), uploaded 273.0 MiB in 4m59s at 934.7 KiB/s average speed

This is being backed up to a external USB hard drive, so its quite fast and doesn’t slow up what I'm doing on the computer. Mostly I don’t notice when its backing up, I just carry on with what I'm doing, which is very different from the 'backup v5.5' that I used previously. This for me is my backup solution for the foreseeable future, it just works on its own with very minimal input from me, which is how it should be.