New "borgmatic" command to support Borg backup software, a fork of Attic.

This commit is contained in:
Dan Helfman 2015-07-18 18:35:29 -07:00
parent 52d5240fa0
commit 7097ed67a6
13 changed files with 175 additions and 58 deletions

4
NEWS
View file

@ -1,3 +1,7 @@
0.1.0
* New "borgmatic" command to support Borg backup software, a fork of Attic.
0.0.7 0.0.7
* Flag for multiple levels of verbosity: some, and lots. * Flag for multiple levels of verbosity: some, and lots.

View file

@ -4,12 +4,13 @@ save_as: atticmatic/index.html
## Overview ## Overview
atticmatic is a simple Python wrapper script for the [Attic backup atticmatic is a simple Python wrapper script for the
software](https://attic-backup.org/) that initiates a backup, prunes any old [Attic](https://attic-backup.org/) and
backups according to a retention policy, and validates backups for [Borg](https://borgbackup.github.io/borgbackup/) backup software that
consistency. The script supports specifying your settings in a declarative initiates a backup, prunes any old backups according to a retention policy,
configuration file rather than having to put them all on the command-line, and and validates backups for consistency. The script supports specifying your
handles common errors. settings in a declarative configuration file rather than having to put them
all on the command-line, and handles common errors.
Here's an example config file: Here's an example config file:
@ -17,7 +18,7 @@ Here's an example config file:
# Space-separated list of source directories to backup. # Space-separated list of source directories to backup.
source_directories: /home /etc source_directories: /home /etc
# Path to local or remote Attic repository. # Path to local or remote backup repository.
repository: user@backupserver:sourcehostname.attic repository: user@backupserver:sourcehostname.attic
[retention] [retention]
@ -41,14 +42,14 @@ available](https://torsion.org/hg/atticmatic). It's also mirrored on
## Setup ## Setup
To get up and running with Attic, follow the [Attic Quick To get up and running, follow the [Attic Quick
Start](https://attic-backup.org/quickstart.html) guide to create an Attic Start](https://attic-backup.org/quickstart.html) or the [Borg Quick
Start](https://borgbackup.github.io/borgbackup/quickstart.html) to create a
repository on a local or remote host. Note that if you plan to run atticmatic repository on a local or remote host. Note that if you plan to run atticmatic
on a schedule with cron, and you encrypt your attic repository with a on a schedule with cron, and you encrypt your attic repository with a
passphrase instead of a key file, you'll need to set the `ATTIC_PASSPHRASE` passphrase instead of a key file, you'll need to set the `ATTIC_PASSPHRASE`
environment variable. See [attic's repository encryption environment variable. See the repository encryption section of the Quick Start
documentation](https://attic-backup.org/quickstart.html#encrypted-repos) for for more info.
more info.
If the repository is on a remote host, make sure that your local root user has If the repository is on a remote host, make sure that your local root user has
key-based ssh access to the desired user account on the remote host. key-based ssh access to the desired user account on the remote host.
@ -57,13 +58,19 @@ To install atticmatic, run the following command to download and install it:
sudo pip install --upgrade hg+https://torsion.org/hg/atticmatic sudo pip install --upgrade hg+https://torsion.org/hg/atticmatic
Then copy the following configuration files: If you are using Attic, copy the following configuration files:
sudo cp sample/atticmatic.cron /etc/cron.d/atticmatic sudo cp sample/atticmatic.cron /etc/cron.d/atticmatic
sudo mkdir /etc/atticmatic/ sudo mkdir /etc/atticmatic/
sudo cp sample/config sample/excludes /etc/atticmatic/ sudo cp sample/config sample/excludes /etc/atticmatic/
Lastly, modify those files with your desired configuration. If you are using Borg, copy the files like this instead:
sudo cp sample/atticmatic.cron /etc/cron.d/borgmatic
sudo mkdir /etc/borgmatic/
sudo cp sample/config sample/excludes /etc/borgmatic/
Lastly, modify the /etc files with your desired configuration.
## Usage ## Usage
@ -73,6 +80,11 @@ arguments:
atticmatic atticmatic
Or, if you're using Borg, use this command instead to make use of the Borg
backend:
borgmatic
This will also prune any old backups as per the configured retention policy, This will also prune any old backups as per the configured retention policy,
and check backups for consistency problems due to things like file damage. and check backups for consistency problems due to things like file damage.

View file

View file

@ -0,0 +1,12 @@
from functools import partial
from atticmatic.backends import shared
# An atticmatic backend that supports Attic for actually handling backups.
COMMAND = 'attic'
create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

View file

@ -0,0 +1,13 @@
from functools import partial
from atticmatic.backends import shared
# An atticmatic backend that supports Borg for actually handling backups.
COMMAND = 'borg'
create_archive = partial(shared.create_archive, command=COMMAND)
prune_archives = partial(shared.prune_archives, command=COMMAND)
check_archives = partial(shared.check_archives, command=COMMAND)

View file

@ -6,10 +6,16 @@ import subprocess
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
def create_archive(excludes_filename, verbosity, source_directories, repository): # Common backend functionality shared by Attic and Borg. As the two backup
# commands diverge, these shared functions will likely need to be replaced
# with non-shared functions within atticmatic.backends.attic and
# atticmatic.backends.borg.
def create_archive(excludes_filename, verbosity, source_directories, repository, command):
''' '''
Given an excludes filename, a vebosity flag, a space-separated list of source directories, and Given an excludes filename, a vebosity flag, a space-separated list of source directories, a
a local or remote repository path, create an attic archive. local or remote repository path, and a command to run, create an attic archive.
''' '''
sources = tuple(source_directories.split(' ')) sources = tuple(source_directories.split(' '))
verbosity_flags = { verbosity_flags = {
@ -17,8 +23,8 @@ def create_archive(excludes_filename, verbosity, source_directories, repository)
VERBOSITY_LOTS: ('--verbose', '--stats'), VERBOSITY_LOTS: ('--verbose', '--stats'),
}.get(verbosity, ()) }.get(verbosity, ())
command = ( full_command = (
'attic', 'create', command, 'create',
'--exclude-from', excludes_filename, '--exclude-from', excludes_filename,
'{repo}::{hostname}-{timestamp}'.format( '{repo}::{hostname}-{timestamp}'.format(
repo=repository, repo=repository,
@ -27,7 +33,7 @@ def create_archive(excludes_filename, verbosity, source_directories, repository)
), ),
) + sources + verbosity_flags ) + sources + verbosity_flags
subprocess.check_call(command) subprocess.check_call(full_command)
def _make_prune_flags(retention_config): def _make_prune_flags(retention_config):
@ -52,18 +58,19 @@ def _make_prune_flags(retention_config):
) )
def prune_archives(verbosity, repository, retention_config): def prune_archives(verbosity, repository, retention_config, command):
''' '''
Given a verbosity flag, a local or remote repository path, and a retention config dict, prune Given a verbosity flag, a local or remote repository path, a retention config dict, and a
attic archives according the the retention policy specified in that configuration. command to run, prune attic archives according the the retention policy specified in that
configuration.
''' '''
verbosity_flags = { verbosity_flags = {
VERBOSITY_SOME: ('--stats',), VERBOSITY_SOME: ('--stats',),
VERBOSITY_LOTS: ('--verbose', '--stats'), VERBOSITY_LOTS: ('--verbose', '--stats'),
}.get(verbosity, ()) }.get(verbosity, ())
command = ( full_command = (
'attic', 'prune', command, 'prune',
repository, repository,
) + tuple( ) + tuple(
element element
@ -71,7 +78,7 @@ def prune_archives(verbosity, repository, retention_config):
for element in pair for element in pair
) + verbosity_flags ) + verbosity_flags
subprocess.check_call(command) subprocess.check_call(full_command)
DEFAULT_CHECKS = ('repository', 'archives') DEFAULT_CHECKS = ('repository', 'archives')
@ -123,10 +130,10 @@ def _make_check_flags(checks):
) )
def check_archives(verbosity, repository, consistency_config): def check_archives(verbosity, repository, consistency_config, command):
''' '''
Given a verbosity flag, a local or remote repository path, and a consistency config dict, check Given a verbosity flag, a local or remote repository path, a consistency config dict, and a
the contained attic archives for consistency. command to run, check the contained attic archives for consistency.
If there are no consistency checks to run, skip running them. If there are no consistency checks to run, skip running them.
''' '''
@ -139,12 +146,12 @@ def check_archives(verbosity, repository, consistency_config):
VERBOSITY_LOTS: ('--verbose',), VERBOSITY_LOTS: ('--verbose',),
}.get(verbosity, ()) }.get(verbosity, ())
command = ( full_command = (
'attic', 'check', command, 'check',
repository, repository,
) + _make_check_flags(checks) + verbosity_flags ) + _make_check_flags(checks) + verbosity_flags
# Attic's check command spews to stdout even without the verbose flag. Suppress it. # The check command spews to stdout even without the verbose flag. Suppress it.
stdout = None if verbosity_flags else open(os.devnull, 'w') stdout = None if verbosity_flags else open(os.devnull, 'w')
subprocess.check_call(command, stdout=stdout) subprocess.check_call(full_command, stdout=stdout)

View file

@ -1,31 +1,34 @@
from __future__ import print_function from __future__ import print_function
from argparse import ArgumentParser from argparse import ArgumentParser
from importlib import import_module
import os
from subprocess import CalledProcessError from subprocess import CalledProcessError
import sys import sys
from atticmatic.attic import check_archives, create_archive, prune_archives
from atticmatic.config import parse_configuration from atticmatic.config import parse_configuration
DEFAULT_CONFIG_FILENAME = '/etc/atticmatic/config' DEFAULT_CONFIG_FILENAME_PATTERN = '/etc/{}/config'
DEFAULT_EXCLUDES_FILENAME = '/etc/atticmatic/excludes' DEFAULT_EXCLUDES_FILENAME_PATTERN = '/etc/{}/excludes'
def parse_arguments(*arguments): def parse_arguments(command_name, *arguments):
''' '''
Parse the given command-line arguments and return them as an ArgumentParser instance. Given the name of the command with which this script was invoked and command-line arguments,
parse the arguments and return them as an ArgumentParser instance. Use the command name to
determine the default configuration and excludes paths.
''' '''
parser = ArgumentParser() parser = ArgumentParser()
parser.add_argument( parser.add_argument(
'-c', '--config', '-c', '--config',
dest='config_filename', dest='config_filename',
default=DEFAULT_CONFIG_FILENAME, default=DEFAULT_CONFIG_FILENAME_PATTERN.format(command_name),
help='Configuration filename', help='Configuration filename',
) )
parser.add_argument( parser.add_argument(
'--excludes', '--excludes',
dest='excludes_filename', dest='excludes_filename',
default=DEFAULT_EXCLUDES_FILENAME, default=DEFAULT_EXCLUDES_FILENAME_PATTERN.format(command_name),
help='Excludes filename', help='Excludes filename',
) )
parser.add_argument( parser.add_argument(
@ -37,15 +40,30 @@ def parse_arguments(*arguments):
return parser.parse_args(arguments) return parser.parse_args(arguments)
def load_backend(command_name):
'''
Given the name of the command with which this script was invoked, return the corresponding
backend module responsible for actually dealing with backups.
'''
backend_name = {
'atticmatic': 'attic',
'borgmatic': 'borg',
}.get(command_name, 'attic')
return import_module('atticmatic.backends.{}'.format(backend_name))
def main(): def main():
try: try:
args = parse_arguments(*sys.argv[1:]) command_name = os.path.basename(sys.argv[0])
args = parse_arguments(command_name, *sys.argv[1:])
config = parse_configuration(args.config_filename) config = parse_configuration(args.config_filename)
repository = config.location['repository'] repository = config.location['repository']
backend = load_backend(command_name)
create_archive(args.excludes_filename, args.verbosity, **config.location) backend.create_archive(args.excludes_filename, args.verbosity, **config.location)
prune_archives(args.verbosity, repository, config.retention) backend.prune_archives(args.verbosity, repository, config.retention)
check_archives(args.verbosity, repository, config.consistency) backend.check_archives(args.verbosity, repository, config.consistency)
except (ValueError, IOError, CalledProcessError) as error: except (ValueError, IOError, CalledProcessError) as error:
print(error, file=sys.stderr) print(error, file=sys.stderr)
sys.exit(1) sys.exit(1)

View file

@ -5,16 +5,19 @@ from nose.tools import assert_raises
from atticmatic import command as module from atticmatic import command as module
def test_parse_arguments_with_no_arguments_uses_defaults(): COMMAND_NAME = 'foomatic'
parser = module.parse_arguments()
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME def test_parse_arguments_with_no_arguments_uses_defaults():
parser = module.parse_arguments(COMMAND_NAME)
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.verbosity == None assert parser.verbosity == None
def test_parse_arguments_with_filename_arguments_overrides_defaults(): def test_parse_arguments_with_filename_arguments_overrides_defaults():
parser = module.parse_arguments('--config', 'myconfig', '--excludes', 'myexcludes') parser = module.parse_arguments(COMMAND_NAME, '--config', 'myconfig', '--excludes', 'myexcludes')
assert parser.config_filename == 'myconfig' assert parser.config_filename == 'myconfig'
assert parser.excludes_filename == 'myexcludes' assert parser.excludes_filename == 'myexcludes'
@ -22,10 +25,10 @@ def test_parse_arguments_with_filename_arguments_overrides_defaults():
def test_parse_arguments_with_verbosity_flag_overrides_default(): def test_parse_arguments_with_verbosity_flag_overrides_default():
parser = module.parse_arguments('--verbosity', '1') parser = module.parse_arguments(COMMAND_NAME, '--verbosity', '1')
assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME assert parser.config_filename == module.DEFAULT_CONFIG_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME assert parser.excludes_filename == module.DEFAULT_EXCLUDES_FILENAME_PATTERN.format(COMMAND_NAME)
assert parser.verbosity == 1 assert parser.verbosity == 1
@ -35,6 +38,6 @@ def test_parse_arguments_with_invalid_arguments_exits():
try: try:
with assert_raises(SystemExit): with assert_raises(SystemExit):
module.parse_arguments('--posix-me-harder') module.parse_arguments(COMMAND_NAME, '--posix-me-harder')
finally: finally:
sys.stderr = original_stderr sys.stderr = original_stderr

View file

@ -2,7 +2,7 @@ from collections import OrderedDict
from flexmock import flexmock from flexmock import flexmock
from atticmatic import attic as module from atticmatic.backends import shared as module
from atticmatic.tests.builtins import builtins_mock from atticmatic.tests.builtins import builtins_mock
from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS from atticmatic.verbosity import VERBOSITY_SOME, VERBOSITY_LOTS
@ -42,6 +42,7 @@ def test_create_archive_should_call_attic_with_parameters():
verbosity=None, verbosity=None,
source_directories='foo bar', source_directories='foo bar',
repository='repo', repository='repo',
command='attic',
) )
@ -55,6 +56,7 @@ def test_create_archive_with_verbosity_some_should_call_attic_with_stats_paramet
verbosity=VERBOSITY_SOME, verbosity=VERBOSITY_SOME,
source_directories='foo bar', source_directories='foo bar',
repository='repo', repository='repo',
command='attic',
) )
@ -68,6 +70,7 @@ def test_create_archive_with_verbosity_lots_should_call_attic_with_verbose_param
verbosity=VERBOSITY_LOTS, verbosity=VERBOSITY_LOTS,
source_directories='foo bar', source_directories='foo bar',
repository='repo', repository='repo',
command='attic',
) )
@ -108,6 +111,7 @@ def test_prune_archives_should_call_attic_with_parameters():
verbosity=None, verbosity=None,
repository='repo', repository='repo',
retention_config=retention_config, retention_config=retention_config,
command='attic',
) )
@ -122,6 +126,7 @@ def test_prune_archives_with_verbosity_some_should_call_attic_with_stats_paramet
repository='repo', repository='repo',
verbosity=VERBOSITY_SOME, verbosity=VERBOSITY_SOME,
retention_config=retention_config, retention_config=retention_config,
command='attic',
) )
@ -136,6 +141,7 @@ def test_prune_archives_with_verbosity_lots_should_call_attic_with_verbose_param
repository='repo', repository='repo',
verbosity=VERBOSITY_LOTS, verbosity=VERBOSITY_LOTS,
retention_config=retention_config, retention_config=retention_config,
command='attic',
) )
@ -193,6 +199,7 @@ def test_check_archives_should_call_attic_with_parameters():
verbosity=None, verbosity=None,
repository='repo', repository='repo',
consistency_config=consistency_config, consistency_config=consistency_config,
command='attic',
) )
@ -211,6 +218,7 @@ def test_check_archives_with_verbosity_some_should_call_attic_with_verbose_param
verbosity=VERBOSITY_SOME, verbosity=VERBOSITY_SOME,
repository='repo', repository='repo',
consistency_config=consistency_config, consistency_config=consistency_config,
command='attic',
) )
@ -229,6 +237,7 @@ def test_check_archives_with_verbosity_lots_should_call_attic_with_verbose_param
verbosity=VERBOSITY_LOTS, verbosity=VERBOSITY_LOTS,
repository='repo', repository='repo',
consistency_config=consistency_config, consistency_config=consistency_config,
command='attic',
) )
@ -241,4 +250,5 @@ def test_check_archives_without_any_checks_should_bail():
verbosity=None, verbosity=None,
repository='repo', repository='repo',
consistency_config=consistency_config, consistency_config=consistency_config,
command='attic',
) )

View file

@ -0,0 +1,33 @@
from flexmock import flexmock
from atticmatic import command as module
def test_load_backend_with_atticmatic_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)
assert module.load_backend('atticmatic') == backend
def test_load_backend_with_unknown_command_should_return_attic_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.attic')
.and_return(backend).once()
)
assert module.load_backend('unknownmatic') == backend
def test_load_backend_with_borgmatic_command_should_return_borg_backend():
backend = flexmock()
(
flexmock(module).should_receive('import_module').with_args('atticmatic.backends.borg')
.and_return(backend).once()
)
assert module.load_backend('borgmatic') == backend

View file

@ -2,7 +2,7 @@
# Space-separated list of source directories to backup. # Space-separated list of source directories to backup.
source_directories: /home /etc source_directories: /home /etc
# Path to local or remote Attic repository. # Path to local or remote repository.
repository: user@backupserver:sourcehostname.attic repository: user@backupserver:sourcehostname.attic
[retention] [retention]

View file

@ -2,12 +2,17 @@ from setuptools import setup, find_packages
setup( setup(
name='atticmatic', name='atticmatic',
version='0.0.7', version='0.1.0',
description='A wrapper script for Attic backup software that creates and prunes backups', description='A wrapper script for Attic/Borg backup software that creates and prunes backups',
author='Dan Helfman', author='Dan Helfman',
author_email='witten@torsion.org', author_email='witten@torsion.org',
packages=find_packages(), packages=find_packages(),
entry_points={'console_scripts': ['atticmatic = atticmatic.command:main']}, entry_points={
'console_scripts': [
'atticmatic = atticmatic.command:main',
'borgmatic = atticmatic.command:main',
]
},
tests_require=( tests_require=(
'flexmock', 'flexmock',
'nose', 'nose',