Install Duplicity

Requirements for Duplicity 0.4.3

• Python v2.4 or later (see the Python homepage)
• Librsync v0.9.6 or later (downloadable from the librsync download page)
• GnuPG for encryption (see the GnuPG homepage).
• GnuPGInterface 0.3.2 or later (Sourceforge Home)
• Boto 0.9b or later (Boto Home)
• pexpect 2.1 or later (Pexpect Home) or python-pexpect from your distro's repository.

Reference http://duplicity.nongnu.org

Install on Ubuntu

On Ubuntu you simply can use apt to install Ubuntu. All the above dependancies will be installed automatically.

apt-get install duplicity

Create GPG Key

Gnupg should already be installed as a dependance of Duplicity. To create a key run the following:

$ gpg --gen-key
gpg (GnuPG) 1.4.6; Copyright (C) 2006 Free Software Foundation, Inc.

The default type of key is fine here.

Please select what kind of key you want:
    (1) DSA and Elgamal (default)
    (2) DSA (sign only)
    (5) RSA (sign only)
Your selection? 1

The default keysize is more then sufficient.

DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 2048

For backup use, there's no real reason to have the key expire, so select 0 here.

Please specify how long the key should be valid.
    0 = key does not expire
    <n>  = key expires in n days
    <n>w = key expires in n weeks
    <n>m = key expires in n months
    <n>y = key expires in n years
Key is valid for? (0) 0

Enter your name, email address and a comment.

Real name: John Doe
Email address: jon@sysadminschronicles.com
Comment: Key for Duplicity

Confirm the key details and enter O for Okay.

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

Enter a phasehrase. Make sure you remember this as you will need it later on, as well as to unencrypt any files from a restore. In the example script $ENV{'PASSPHRASE'} is where you use this password.

Enter passphrase: password
Repeat passphrase: password

At the end of the key being generated GPG will print the key name, copy down the eight character name. In the example script $gpg_key is where you use this key name.

gpg: key F1A3D642 marked as ultimately trusted
public and secret key created and signed.

Your private and public key are now located in ~/.gnupg folder.

Copy this folder to the home directory of the user you want to run the backup as. For example its likely you are going to have to run the backup as root because of permissions on the files you plan to backup, so copy this folder to /root. Make sure root doesn't already have a GPG key used for other purposes.

sudo cp -R ~/.gnupg /root

You can delete the original .gnupg folder as there is no longer any use for it.

Note: You really need to backup your GPG key offsite. Meaning back it up somewhere other then Amazon S3 and the machine you are using the key on. If you loose your GPG key, your backup is useless, as you will not be able to unencrypted it.

Sign up for Amazon S3

Goto Amazon Web Services and sign up for an account. They will want a credit card for billing. Once you login, navigate to the Amazon Simple Storage Service page and sign up for the service. Now using the top right button labeled Your Web Services Account, click the AWS Access Identifiers link. Copy down your Access Key ID and Secret Access Key for use in the below script. $ENV{'AWS_ACCESS_KEY_ID'} and $ENV{'AWS_SECRET_ACCESS_KEY'} is where these two values are used in the example script.

Example Perl Script for Backup

Now that you have a GPG key created, Duplicity installed, and your AWS account created you can start a backup. The following is a Perl script I created to backup a set of specified directories as well as dump our entire PostgreSQL and back it up. If you don't need the PostgreSQL functionality delete or comment out the section that starts with the comment #Dump database.

Note: If you did as above and copied the GPG key to the root home folder, you will need to run this script as root. If you use sudo to gain root privileges make sure you use the - H flag to set the HOME environmental variable to root's. Otherwise either login as root or use su - to gain root privileges.

Requirements

• MIME::Lite perl module
• Duplicity
• pg_dumpall
• Amazon S3 Account
• GPG key
• SMTP server that can relay mail

backup_s3.pl

#!/usr/bin/perl

use strict;
use MIME::Lite;
my $subject;
my $backupOut;

#Where to backup the database too.
my $db_backup_dir = "/usr/local/backup/db_data";
#What to class the database backup file.
my $db_backup_filename = "dumpall.sql";

#What user to dump the database as.
my $db_user = "postgres";

#Path to pg_dumpall
my $pg_dumpall = "/usr/bin/pg_dumpall";

#Folders to be backed up.
my @backup_folders = ('/var/www', '/etc', "$db_backup_dir");

#Set force to 1 to actualy delete files found to be outdated or orphaned. 
my $force = 0;

#Check or delete files older then: (See duplicity's man page for acceptable values)
my $older_than = "2M";   

#Path to duplicity
my $duplicity = "/usr/bin/duplicity";

#What Amazon S3 Bucket to use.
my $s3_bucketname = "SetMe";

#What GPG key to use.
my $gpg_key = "SetMe";

#Your GPG passphrase
$ENV{'PASSPHRASE'} = 'SetMe';

#Where to send notification emails.
my @to_email = ('ben@domain.com', 'ron@domain.com');

#SMTP settings for sending mail.
my $smtp_user = 'system@domain.com';
my $smtp_passwd = "SetMe";
my $smtp_server = "localhost";

#Your AWS login information.
$ENV{'AWS_ACCESS_KEY_ID'} = 'SetMe';
$ENV{'AWS_SECRET_ACCESS_KEY'} = 'SetMe';

### Shouldn't need to edit anything below this line ###

my $backupCommand = "$duplicity --encrypt-key=$gpg_key --sign-key=$gpg_key ";
foreach my $dir (@backup_folders) {
    $backupCommand .= "--include $dir ";
}
$backupCommand .= "--exclude '**' / s3+http://$s3_bucketname";

my $cleanupCommand1;
my $cleanupCommand2;
if ($force) {
    $cleanupCommand1 = "$duplicity --force --remove-older-than $older_than s3+http://$s3_bucketname";
    $cleanupCommand2 = "$duplicity --force --cleanup s3+http://$s3_bucketname";
}
else {
    $cleanupCommand1 = "$duplicity --remove-older-than $older_than s3+http://$s3_bucketname";
    $cleanupCommand2 = "$duplicity --cleanup s3+http://$s3_bucketname";
}

#Dump database
system( "$pg_dumpall -U $db_user > $db_backup_dir/$db_backup_filename" );
if ( !$? ) {
    $backupOut .= "Dump of PostgreSQL Succeeded!\n\n";
} 
else {
    $subject .= "Error - ";
    $backupOut .= "Dump of PostgreSQL Failed!\n\n";
}
#End of Dump database

#Run Duplicity
open(BACKUPCMD, "$backupCommand 2>&1 |") or die "can't fork: $!";

#Grab output of Duplicity and check for errors
while (<BACKUPCMD>) {
    $backupOut .= $_, "\n";
    print $_;
    if ( $_ =~ m/^Error/ ) {
        my $line = $_;
        chomp($line);
        my @result = split(/ /, $line);

        if ( $result[1] != 0 ) {
            $subject .= "Error - ";
        }
    }
    elsif ( $_ =~ m/Error/ ) {
        $subject .= "Error - ";
    }
    elsif ( $_ =~ m/Traceback/ ) {
        $subject .= "Error - ";
    }
}
close(BACKUPCMD) or die "Can't close backup command: $! $?";

#Check for old files based on $older_than. Delete older files if $force is set to non-zero.
open(CLEANUPCMD1, "$cleanupCommand1 2>&1 |");
my $cleanupOut = "\nChecking for old files:\n";
while (<CLEANUPCMD1>) {
    $cleanupOut .= $_, "\n";
}
close(CLEANUPCMD1);

#Check for orphaned files. Delete orphaned files if $force is set to non-zero.
$cleanupOut .= "\nChecking for files that can be deleted:\n";
open(CLEANUPCMD2, "$cleanupCommand2 2>&1 |");
while (<CLEANUPCMD2>) {
    $cleanupOut .= $_, "\n";
}
close(CLEANUPCMD2);

#Email body
my $message = $backupOut . "\n" . $cleanupOut;

#Unset enviroment variables
$ENV{'AWS_ACCESS_KEY_ID'} = "";
$ENV{'AWS_SECRET_ACCESS_KEY'} = "";
$ENV{'PASSPHRASE'}= "";

#Create and send email
my $dateTime = `date`;
chomp($dateTime);
$subject .= "Backup to Amazon S3 at $dateTime";

my $msg = MIME::Lite->new(  
    From => $smtp_user,  
    To => join(',', @to_email),  
    Subject => $subject,  
    Data => $message  
);
MIME::Lite->send(
    'smtp', 
    $smtp_server, 
    AuthUser=>$smtp_user,
    AuthPass=>$smtp_passwd, 
    Timeout => 60
);
$msg->send;

Resources

• Duplicity
• Duplicity + Amazon S3 = incremental encrypted remote backup
• GPG Quick Start