Have an idea on how to make this documentation even better? Use our issue tracker to send your +href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues">issue tracker to send your feedback!
- - - - diff --git a/docs/how-to/develop-on-borgmatic.md b/docs/how-to/develop-on-borgmatic.md index dc0327b..865f0d4 100644 --- a/docs/how-to/develop-on-borgmatic.md +++ b/docs/how-to/develop-on-borgmatic.md @@ -10,17 +10,17 @@ eleventyNavigation: To get set up to hack on borgmatic, first clone master via HTTPS or SSH: ```bash -git clone https://projects.torsion.org/witten/borgmatic.git +git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git ``` Or: ```bash -git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git +git clone ssh://git@projects.torsion.org:3022/borgmatic-collective/borgmatic.git ``` Then, install borgmatic -"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)" +"[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)" so that you can run borgmatic commands while you're hacking on them to make sure your changes work. @@ -66,8 +66,6 @@ following: tox -e black ``` -Note that Black requires at minimum Python 3.6. - And if you get a complaint from the [isort](https://github.com/timothycrosley/isort) Python import orderer, you can ask isort to order your imports for you: @@ -118,7 +116,7 @@ See the Black, Flake8, and isort documentation for more information. Each pull request triggers a continuous integration build which runs the test suite. You can view these builds on -[build.torsion.org](https://build.torsion.org/witten/borgmatic), and they're +[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're also linked from the commits list on each pull request. ## Documentation development diff --git a/docs/how-to/set-up-backups.md b/docs/how-to/set-up-backups.md index 0db1e68..b8b860e 100644 --- a/docs/how-to/set-up-backups.md +++ b/docs/how-to/set-up-backups.md @@ -28,7 +28,7 @@ sudo pip3 install --user --upgrade borgmatic This installs borgmatic and its commands at the `/root/.local/bin` path. Your pip binary may have a different name than "pip3". Make sure you're using -Python 3, as borgmatic does not support Python 2. +Python 3.6+, as borgmatic does not support Python 2. The next step is to ensure that borgmatic's commands available are on your system `PATH`, so that you can run borgmatic: @@ -77,7 +77,7 @@ on a relatively dedicated system, then a global install can work out fine. Besides the approaches described above, there are several other options for installing borgmatic: - * [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) + * [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files) * [Docker base image](https://hub.docker.com/r/monachus/borgmatic/) * [Debian](https://tracker.debian.org/pkg/borgmatic) * [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic) @@ -250,7 +250,7 @@ that, you can configure a separate job runner to invoke it periodically. ### cron If you're using cron, download the [sample cron -file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic). +file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic). Then, from the directory where you downloaded it: ```bash @@ -258,7 +258,10 @@ sudo mv borgmatic /etc/cron.d/borgmatic sudo chmod +x /etc/cron.d/borgmatic ``` -You can modify the cron file if you'd like to run borgmatic more or less frequently. +If borgmatic is installed at a different location than +`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You +can also modify the cron file if you'd like to run borgmatic more or less +frequently. ### systemd @@ -271,9 +274,9 @@ you may already have borgmatic systemd service and timer files. If so, you may be able to skip some of the steps below.) First, download the [sample systemd service -file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service) +file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service) and the [sample systemd timer -file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer). +file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer). Then, from the directory where you downloaded them: @@ -294,7 +297,7 @@ borgmatic to run. If you run borgmatic in macOS with launchd, you may encounter permissions issues when reading files to backup. If that happens to you, you may be interested in an [unofficial work-around for Full Disk -Access](https://projects.torsion.org/witten/borgmatic/issues/293). +Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293). ## Colored output diff --git a/sample/systemd/borgmatic.service b/sample/systemd/borgmatic.service index 4fc8895..d025785 100644 --- a/sample/systemd/borgmatic.service +++ b/sample/systemd/borgmatic.service @@ -32,13 +32,16 @@ RestrictSUIDSGID=yes SystemCallArchitectures=native SystemCallFilter=@system-service SystemCallErrorNumber=EPERM -# Restrict write access -# Change to 'ProtectSystem=strict' and uncomment 'ProtectHome' to make the whole file -# system read-only be default and uncomment 'ReadWritePaths' for the required write access. -# Add local repositroy paths to the list of 'ReadWritePaths' like '-/mnt/my_backup_drive'. +# To restrict write access further, change "ProtectSystem" to "strict" and uncomment +# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository +# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This +# leaves most of the filesystem read-only to borgmatic. ProtectSystem=full -# ProtectHome=read-only -# ReadWritePaths=-/root/.config/borg -/root/.cache/borg -/root/.borgmatic +# ReadWritePaths=-/mnt/my_backup_drive +# ReadOnlyPaths=-/var/lib/my_backup_source +# This will mount a tmpfs on top of /root and pass through needed paths +# ProtectHome=tmpfs +# BindPaths=-/root/.cache/borg -/root/.cache/borg -/root/.borgmatic CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW diff --git a/scripts/release b/scripts/release index f728390..b901047 100755 --- a/scripts/release +++ b/scripts/release @@ -38,7 +38,7 @@ twine upload -r pypi dist/borgmatic-*-py3-none-any.whl dist/borgmatic-*-py3-none release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')" escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')" curl --silent --request POST \ - "https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" \ + "https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases" \ --header "Authorization: token $projects_token" \ --header "Accept: application/json" \ --header "Content-Type: application/json" \ diff --git a/scripts/run-full-tests b/scripts/run-full-tests index ca2f5a7..47e4f5f 100755 --- a/scripts/run-full-tests +++ b/scripts/run-full-tests @@ -13,8 +13,8 @@ set -e apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client # If certain dependencies of black are available in this version of Alpine, install them. apk add --no-cache py3-typed-ast py3-regex || true -python3 -m pip install --upgrade pip==20.2.4 setuptools==50.3.2 -pip3 install tox==3.20.1 +python3 -m pip install --upgrade pip==21.3.1 setuptools==58.2.0 +pip3 install tox==3.24.4 export COVERAGE_FILE=/tmp/.coverage tox --workdir /tmp/.tox --sitepackages tox --workdir /tmp/.tox --sitepackages -e end-to-end diff --git a/setup.py b/setup.py index ebc47fd..11a83f1 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,6 @@ from setuptools import find_packages, setup -VERSION = '1.5.16.dev0' +VERSION = '1.5.21.dev0' setup( diff --git a/tests/integration/test_execute.py b/tests/integration/test_execute.py index 490e3ab..51aa5f4 100644 --- a/tests/integration/test_execute.py +++ b/tests/integration/test_execute.py @@ -1,5 +1,6 @@ import logging import subprocess +import sys import pytest from flexmock import flexmock @@ -98,7 +99,7 @@ def test_log_outputs_kills_other_processes_when_one_errors(): process, 2, 'borg' ).and_return(True) other_process = subprocess.Popen( - ['watch', 'true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT ) flexmock(module).should_receive('exit_code_indicates_error').with_args( other_process, None, 'borg' @@ -123,6 +124,75 @@ def test_log_outputs_kills_other_processes_when_one_errors(): assert error.value.output +def test_log_outputs_vents_other_processes_when_one_exits(): + ''' + Execute a command to generate a longish random string and pipe it into another command that + exits quickly. The test is basically to ensure we don't hang forever waiting for the exited + process to read the pipe, and that the string-generating process eventually gets vented and + exits. + ''' + flexmock(module.logger).should_receive('log') + flexmock(module).should_receive('command_for_process').and_return('grep') + + process = subprocess.Popen( + [ + sys.executable, + '-c', + "import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))", + ], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + other_process = subprocess.Popen( + ['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + flexmock(module).should_receive('output_buffer_for_process').with_args( + process, (process.stdout,) + ).and_return(process.stderr) + flexmock(module).should_receive('output_buffer_for_process').with_args( + other_process, (process.stdout,) + ).and_return(other_process.stdout) + flexmock(process.stdout).should_call('readline').at_least().once() + + module.log_outputs( + (process, other_process), + exclude_stdouts=(process.stdout,), + output_log_level=logging.INFO, + borg_local_path='borg', + ) + + +def test_log_outputs_does_not_error_when_one_process_exits(): + flexmock(module.logger).should_receive('log') + flexmock(module).should_receive('command_for_process').and_return('grep') + + process = subprocess.Popen( + [ + sys.executable, + '-c', + "import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))", + ], + stdout=None, # Specifically test the case of a process without stdout captured. + stderr=None, + ) + other_process = subprocess.Popen( + ['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + flexmock(module).should_receive('output_buffer_for_process').with_args( + process, (process.stdout,) + ).and_return(process.stderr) + flexmock(module).should_receive('output_buffer_for_process').with_args( + other_process, (process.stdout,) + ).and_return(other_process.stdout) + + module.log_outputs( + (process, other_process), + exclude_stdouts=(process.stdout,), + output_log_level=logging.INFO, + borg_local_path='borg', + ) + + def test_log_outputs_truncates_long_error_output(): flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0 flexmock(module.logger).should_receive('log') diff --git a/tests/unit/borg/test_create.py b/tests/unit/borg/test_create.py index cc56881..99f2910 100644 --- a/tests/unit/borg/test_create.py +++ b/tests/unit/borg/test_create.py @@ -60,6 +60,30 @@ def test_expand_home_directories_considers_none_as_no_directories(): assert paths == () +def test_map_directories_to_devices_gives_device_id_per_path(): + flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55)) + flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66)) + + device_map = module.map_directories_to_devices(('/foo', '/bar')) + + assert device_map == { + '/foo': 55, + '/bar': 66, + } + + +def test_map_directories_to_devices_with_missing_path_does_not_error(): + flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55)) + flexmock(module.os).should_receive('stat').with_args('/bar').and_raise(FileNotFoundError) + + device_map = module.map_directories_to_devices(('/foo', '/bar')) + + assert device_map == { + '/foo': 55, + '/bar': None, + } + + @pytest.mark.parametrize( 'directories,expected_directories', ( @@ -72,6 +96,7 @@ def test_expand_home_directories_considers_none_as_no_directories(): ({'/root': 1, '/root/foo/': 1}, ('/root',)), ({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')), ({'/root/foo': 1, '/root': 1}, ('/root',)), + ({'/root': None, '/root/foo': None}, ('/root', '/root/foo')), ({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')), ({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)), ({'/dup': 1, '/dup': 1}, ('/dup',)),