Create an automated, incremental, remote, encrypted, backup system

The project:
Create an automated, incremental, remote, encrypted, backup system for use with an “untrusted” remote data store server.

The tools:
RHEL/CentOS/Scientific Linux based local data store
Remote, untrusted data store with SSH access
Rsync (http://rsync.samba.org/)
Encfs (http://www.arg0.net/encfs)

The pieces:
Encfs:
From the website: EncFS provides an encrypted filesystem in user-space. It runs without any special permissions and uses the FUSE library and Linux kernel module to provide the filesystem interface. EncFS is open source software, licensed under the GPL.
Normally, encfs accepts two arguments: a root directory and a mount point. The root directory is the encrypted data store, and the mount point is an unencrypted window into the root directory. We, will be using the --reverse flag for encfs. When using the reverse flag, encfs accepts the same two arguments. However, roofdir is now a data store of unencrypted data and the mount point is an encrypted window into the root directory.
The basic idea is for us to mount the encrypted window into our data store, and copy the encrypted data over the network to our untrusted data store.

Rsync:
Rsync powers the copying/updating aspects of the project. There are a lot of words already written about the power and majesty of Rsync so I won’t linger long. The main point to make here is this: Encfs operates in such a way that Rsync loses little efficiency. If you take a directory of 50 files, and encrypt those using Encfs, you’ll have a directory containing 50 encrypted files. This allows Rsync to update only the files that change. Using “encrypted blob” systems, the cypher data is presented to the base filesystem as a single file… and rsync would have to update the entire file.
There is some inefficiently though: when an unencrypted file changes, only the portion of the file that actually changed needs to be updated by Rsync. Encfs using cipher block chaining. Let’s examine a file that undergoes a small change in the middle of the file. From the beginning of the file to the point of change, the enciphered file will be identical. However due to CBC, from the point of change to the end of the file will all be different and require update.

Let’s get going:
First, install rsync and encfs:
yum install rsync fuse-encfs

fuse-encfs is packaged and available from the EPEL repository (http://fedoraproject.org/wiki/EPEL). I assume you either already have that added to your Yum repositories, or know how to do that.

Create the mount point for the crypt:
mkdir /mnt/crypt

Mount the crypt for the first time:
encfs –reverse /path/to/data /mnt/crypt

This will get you started on the interactive process to setup the crypt. If you know what you’re doing and want to fiddle with the knobs and switches, feel free to enter expert mode. The defaults are fine for most people, so you’ll likely just want to hit enter. Encfs is now prompting you for the encryption passphrase. This is the key to the kingdom! You’ll want to use a strong password, but also don’t lose it!

What makes a good passphrase? Here’s a good resource: https://www.grc.com/haystack.htm
Confirm the passphrase and you’re done.

Your very own crypt is now mounted at /mnt/crypt. Feel free to poke around.... cool, eh?

Also notice that there is now a new hidden file in your data store directory: .encfs6.xml. This file contains all of the cryptographic details necessary for encfs to mount the encrypted data “normally” to get your plain text files back. This file also contains enough information for a potential attacker to know exactly how the data is encrypted.

I keep a copy of this file on the remote data store in an unencrypted state. That may make some folks go a little pale, but I don’t think twice. If an attacker gets access to your remote data store, they would also get your .encfs6.xml file and therefore know exactly how the data is encrypted. On the surface, this makes the attacker’s job easier. However, this is not nearly as bad as it sounds. Any serious analysis of the weaknesses of a cryptographic system assumes that the attacker knows all of this information. The strength of a cryptographic system depends on the quality of the cypher itself, and the strength of the passphrase. NOT keeping the details of the cypher secret.

So, there are now three items you *need* to be able to recover your data:
- The cypher text
- The .encfs6.xml file
- The passphrase

Now we just need to copy the enciphered data to the remote data store:
rsync -a --delete /mnt/crypt user@remotehost.com:/path/to/remote/data/store

Will do a very nice job. Setting up ssh keys will allow this to work without interactively entering the remote user password.

When you’re done, just unmount the crypt like you would unmounts anything else:
umount /mnt/crypt

Putting it all together:
Here’s a script I wrote that automates the process. Using cron, I run this periodically to update my remote backup.

crypto-backup.pl
#!/bin/perl
use Getopt::Std;

###################################################################
##################################################################
# crypto-backup.pl
# Written by Andrew Hull
# coffeebreath {.} org
# contact me at scripts {at} coffeebreath {.} org
#
# Distributed under the terms of the GPL, v2
# http://www.gnu.org/licenses/gpl-2.0.html
####################################################################
#######################################################################

### Collect command line switches #####################
my %options=();
getopts("hv:r:", \%options);
######################################################

##### These are the path variables ###########
##### Modify these to suit your requirements ######################
$crypt = "/mnt/crypt/";
$orig_data = "/path/to/local/data";
$passphrase_file = "/path/to/secret"; #Create a file, stored here, with just your passphrase
$remote_crypt = "/remote/path/to/data/store";
$remote_control_dir = "/remote/path/to/control/file";
$remote_site = "remoteuser\@storage.foo.bar";
$control_file = ".encfs6.xml";
$crypt_count_min = 5; #Crypt must contain at least this many files to continue
#This protects your remote store from getting blown away if encfs
#does not mount correctly
##############################################################################

### Help #######################################################
if ($options{h}) {
print "Flags: \n";
print " -h (print this message)\n";
print " -v [n] (set the script verbosity)\n";
print " n<2 = minimal output, 2<=n<4 = moderate, n>=4 max output \n";
print " -r [n] (set rsync verbosity)\n";
print " n<1 = no rsync output, n>=1 uses -v flag, n>=2 uses --progress flag \n\n";
print "Examples:\n";
print " perl crypto-backup.pl -v 2 -r 1 (moderate output of script and rsync)\n";
print " perl crypto-backup.pl -r 4 (max rsync output only)\n";
}
else {
#################################################################

### Set the verbosity level #########################################
if ($options{r} >= 2) {
$rsync_args = "-av --delete --progress ";
}
elsif ($options{r} >= 1) {
$rsync_args = "-av --delete ";
}
else {
$rsync_args = "-a --delete ";
}
###################################################################

### Verbose output of the variables #################
if ($options{v} >= 4) {
print "### Variables ####################################\n";
print "crypt: $crypt\n";
print "orig_data: $orig_data\n";
print "passphrase_file: $passphrase_file\n";
print "remote_crypt: $remote_crypt\n";
print "remote_control_dir: $remote_control_dir\n";
print "remote_site: $remote_site\n";
print "rsync_args: $rsync_args\n";
print "control_file: $control_file\n";
print "crypt_count_min: $crypt_count_min\n";
print "###################################################\n\n";
}
#################################################################

#### Mount the crypt ###############################################
system("encfs -S --reverse $orig_data $crypt < $passphrase_file");
my $status = $? >> 8;

if ($options{v} >= 2) {
print "encfs mount exited with status: $status \n";
}
#######################################################################

#### Saftey check ############################################################
#### Count the number of files and directories in the crypt, recursively #####
my $crypt_count = `ls -1R $crypt | wc -l`;
chomp $crypt_count;
##############################################################################

### If the count is below the saftey, report and bail ##########################
if ($crypt_count < $crypt_count_min) {
print "The number of files and directories in $crypt is $crypt_count.\n";
print "This is lower than the saftey minimum of $crypt_count_min. \n";
print "ABORT, ABORT, ABORT! All hands, abort and stand fast!\n";
die "Count of $crypt is $crypt_count and below saftey num of $crypt_count_min";
}
################################################################################

#### Do the deed ##################################################################
print "syncing .encfs file...\n";
if ($options{v} >= 2) {
print "rsync $rsync_args $orig_data/$control_file $remote_site:$remote_control_dir\n";
}
system("rsync $rsync_args $orig_data/$control_file $remote_site:$remote_control_dir");
print "done\n\n\n";

print "syncing payload...\n";
if ($options{v} >= 2) {
print "rsync $rsync_args $crypt $remote_site:$remote_crypt \n";
}
system("rsync $rsync_args $crypt $remote_site:$remote_crypt");
print "done!\n";
####################################################################################

#### Umount the crypt ###############################
system("umount $crypt");
my $status = $? >> 8;

if ($options{v} >= 2) {
print "umount exited with status: $status \n";
}
#######################################################
}

Now that you've got a nice encrypted remote backup, there is one *HUGE* question left to answer: How do I recover from a disaster?

I'll explore the details in a later entry, but the basic idea is this: using the encrypted data, the .encfs6.xml file, and your passphrase, you'll be able to use encfs in "normal" mode to mount the unencrypted data. Then you can just copy the important things back to your local machine.

I hope anyone reading this finds it helpful. I would love to hear some feedback.
- Andy