Difference between revisions of "Snapshot backups using rsync"

Revision as of 18:03, 3 November 2007

Intro

The scripts on this page are designed to allow the backup of a set of directories to a defined location, with the backups preserving the state of those directories from a number of different times in the past. They require a linux box with a free disk partition to act as the backup server, and the instructions below are designed for Ubuntu, so other distros may require slightly different steps. In summary, they will give you a snapshot directory with contents would look something like (as of 26th Sept 2006):

Contents of /mnt/snapshot:
drwxr-xr-x 7 root root  4096 2006-09-26 00:01 daily.0
drwxr-xr-x 7 root root  4096 2006-09-25 00:00 daily.1
drwxr-xr-x 7 root root  4096 2006-09-24 00:00 daily.2
drwxr-xr-x 7 root root  4096 2006-09-23 00:00 daily.3
drwxr-xr-x 7 root root  4096 2006-09-22 00:00 weekly.0
drwxr-xr-x 5 root root  4096 2006-09-15 00:00 weekly.1
drwxr-xr-x 2 root root  4096 2006-09-08 00:00 weekly.2
drwxr-xr-x 2 root root  4096 2006-09-01 00:00 monthly.0
drwxr-xr-x 2 root root  4096 2006-08-01 00:00 monthly.1
drwxr-xr-x 2 root root  4096 2006-01-01 00:00 annual.0

Note the timestamps of these directories - each one captures the state of the location(s) to be backed up at the time shown. The actual backup process uses rsync and hard links to ensure that if a file doesn't change between two snapshots, then it doesn't take up any additional space (well, virtually no more space). The contents of this page are based on Mike Rubel's instructions at: http://www.mikerubel.org/computers/rsync_snapshots/ - if you find difficulty understanding/using the scripts here, then you may wish to look at his page for additional help.

Prerequisites

• Install rsync and nfs-kernel-server with apt-get (eg. sudo apt-get install rsync)
• http://bretts.org/files/snapshot.tar.bz2

Instructions

• Unzip the contents of snapshot.tar.bz2 to a temporary directory.
• As root, copy the files and directories from the temporary directory to the implied filesystem location (eg. snapshot/etc/cron.d/snapshot -> /etc/cron.d/snapshot, snapshot/var/log/snapshot -> /var/log/snapshot etc.)
• As root, create the directories /root/.snapshot and /mnt/snapshot
• Add the following lines to /etc/fstab, substituting your free disk partition, and it's filesystem type, for /dev/hdb1 and ext3:

/dev/hdb1                   /root/.snapshot   ext3          ro              0       2
localhost:/root/.snapshot   /mnt/snapshot     nfs           ro              0       2

(This mounts your backup partition as /root/.snapshot, away from prying eyes, and mounts the NFS exported version of it as /mnt/snapshot. We mount it twice so that non-root users NEVER see a writable version of the backup directory - see Mike Rubels page for more details.)

• Add the following line to /etc/exports (or create /etc/exports with this line if it doesn't already exist):

/root/.snapshot 127.0.0.1(secure,sync,ro,no_root_squash)

(This defines the NFS share that we will mount as /mnt/snapshot)

• Restart nfs-kernel-server (using sudo /etc/init.d/nfs-kernel-server restart).

Defining areas to be backed up

• For each directory tree to be backed up, create a directory in /etc/snapshot with a logical name for the backup (the name doesn't matter, as long as it's unique).
• Within the newly created directory, created a file called location (eg. /etc/snapshot/my-backup/location), the contents of which should be a single line defining the head of the directory tree to be backed up. This line may be in one of 3 formats:
  • For a local directory, simply the fully qualified path to the location to back up (eg. /home/fred/docs)
  • For a remote directory accessed via an rsync daemon running on the remote machine, the location in the format machinename::rsync_module/subdirectory (eg. xp-box::c/Documents and Settings)
  • For a remote directory accessed via an ssh server running on the remote machine, the location in the format machinename:path (eg. other-host:/usr/local/src)

For remote hosts, an rsync daemon is generally preferable, both for performance reasons, and for ease of use. If ssh is to be used, ssh keys will need to be defined to stop the scripts prompting for a password whenever they're run. However, if the data is being transferred between hosts across an insecure network, ssh may be needed for reasons of security (rsync transmits data between hosts unencrypted, and has a simplistic security model).

• Also within the directory, create a file called filters (eg. /etc/snapshot/my-backup/filters), which may contain multiple lines prefixed with a - to specify locations to exclude from the backup. This file is simply in rsync filters file format (see the rsync man page for more details).

Each one of the locations specified within /etc/snapshot will end up as a subdirectory of /mnt/snapshot (eg. this morning's backup of /home/fred will end up in /mnt/snapshot/daily.0/home/fred).

Example 1: Backup of /home

/etc/snapshot/home/location:

/home

/etc/snapshot/home/filters:

- /andrew/Download/
- /andrew/svn/

Example 2: Backup of /etc

/etc/snapshot/etc/location:

/etc

/etc/snapshot/etc/filters:

Note that filters file is empty, since no subdirectories of /etc are excluded from the backup.

Example 3: Backup of C:\Documents and Settings on alaska

Note that alaska is an XP box running the cygwin rsync daemon.
/etc/snapshot/alaska-home/location:

alaska::c/Documents and Settings

/etc/snapshot/alaska-home/filters:

- /*/Local Settings/Temp
- /*/Local Settings/Temporary Internet Files
- /Andrew/My Documents/Downloads

Further Examples

These and other examples can be found in http://bretts.org/files/snapshot-examples.tar.bz2

Notes

• The partition used for /root/.snapshot should ideally be on a different disk to the files being backed up, to reduce the risk of a disk failure destroying both the data and the backups.
• The scripts can be used without defining /root/.snapshot as a separate partition, subject to the caveat about separate disks directly above. To do this, simply:
  • Don't add the first of the two lines above to /etc/fstab
  • In each of the snapshot scripts within /usr/local/sbin remove the calls near the beginning and end of the file to remount the /root/.snapshot partition (the beginning of each script remounts this partition as read/write, and the end remounts it as read).

Setting up a machine with an rsync daemon

• Edit /etc/xinetd.d/rsync and add:

service rsync
{
        disable         = no
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/bin/rsync
        server_args     = --daemon
        log_on_failure  += USERID
}

• Edit /etc/rsyncd.conf and add (for example):

use chroot = false
strict modes = false
hosts allow = 192.168.1.0/24

[root]
    path = /
    comment = Root
    read only = true

• Restart rsync

Future Development

• It would be nice to have the latest backup burnt to a DVD-RW which could be left in a dedicated DVD-RW drive attached to the backup server. This would help both because DVDs are more robust than hard drives in certain situations (eg. mechanical shock) in case of a problem (eg. power surge) which for some reason destroyed multiple drives on the machine. In addition, the DVD disk could be swapped out periodically to provide an off-site backup copy.
• Some companies provide free or low cost disk space online - this could be used to store a copy of the latest backup for even greater robustness. In addition, since the process uses rsync, unless large numbers of backed up files were changing on a daily basis, there would be relatively little network traffic once the initial backup was rsynced across to the remote site.