psw/borgmatic
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'master' into master

Browse source
This commit is contained in:
cadamswaite 2021-11-14 18:24:17 +00:00
commit 6b182c9d2d
17 changed files with 224 additions and 68 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
.drone.yml
View file

@ -14,6 +14,9 @@ services:
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
clone:
skip_verify: true
steps:
- name: build
image: alpine:3.9
@ -36,6 +39,9 @@ services:
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
clone:
skip_verify: true
steps:
- name: build
image: alpine:3.10
@ -58,6 +64,9 @@ services:
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
clone:
skip_verify: true
steps:
- name: build
image: alpine:3.13
@ -68,9 +77,14 @@ steps:
kind: pipeline
name: documentation
clone:
skip_verify: true
steps:
- name: build
image: plugins/docker
#image: plugins/docker
# Temporary work-around for https://github.com/drone-plugins/drone-docker/pull/327
image: techknowlogick/drone-docker
settings:
username:
from_secret: docker_username
@ -80,5 +94,7 @@ steps:
dockerfile: docs/Dockerfile
trigger:
repo:
- borgmatic-collective/borgmatic
branch:
- master

28
NEWS
View file

@ -1,6 +1,30 @@
1.5.16.dev0
1.5.21.dev0
* Add support for old version (2.x) of jsonschema library.
1.5.20
* Re-release with correct version without dev0 tag.
1.5.19
* #387: Fix error when configured source directories are not present on the filesystem at the time
of backup. Now, Borg will complain, but the backup will still continue.
* #455: Mention changing borgmatic path in cron documentation.
* Update sample systemd service file with more granular read-only filesystem settings.
* Move Gitea and GitHub hosting from a personal namespace to an organization for better
collaboration with related projects.
* 1k ★s on GitHub!
1.5.18
* #389: Fix "message too long" error when logging to rsyslog.
* #440: Fix traceback that can occur when dumping a database.
1.5.17
* #437: Fix error when configuration file contains "umask" option.
* Remove test dependency on vim and /dev/urandom.
1.5.16
* #379: Suppress console output in sample crontab and systemd service files.
* #407: Fix syslog logging on FreeBSD.
* #430: Fix hang when restoring a PostgreSQL "tar" format database dump.
* Better error messages! Switch the library used for validating configuration files (from pykwalify
to jsonschema).
* Link borgmatic Ansible role from installation documentation:
@ -559,7 +583,7 @@
* #49: Support for Borg experimental --patterns-from and --patterns options for specifying mixed
includes/excludes.
* Moved issue tracker from Taiga to integrated Gitea tracker at
https://projects.torsion.org/witten/borgmatic/issues
https://projects.torsion.org/borgmatic-collective/borgmatic/issues
1.1.12
* #46: Declare dependency on pykwalify 1.6 or above, as older versions yield "Unknown key: version"

12
README.md
View file

@ -106,7 +106,7 @@ development or hosting.
### Issues
You've got issues? Or an idea for a feature enhancement? We've got an [issue
tracker](https://projects.torsion.org/witten/borgmatic/issues). In order to
tracker](https://projects.torsion.org/borgmatic-collective/borgmatic/issues). In order to
create a new issue or comment on an issue, you'll need to [login
first](https://projects.torsion.org/user/login). Note that you can login with
an existing GitHub account if you prefer.
@ -129,15 +129,15 @@ Other questions or comments? Contact
### Contributing
borgmatic [source code is
available](https://projects.torsion.org/witten/borgmatic) and is also mirrored
on [GitHub](https://github.com/witten/borgmatic) for convenience.
available](https://projects.torsion.org/borgmatic-collective/borgmatic) and is also mirrored
on [GitHub](https://github.com/borgmatic-collective/borgmatic) for convenience.
borgmatic is licensed under the GNU General Public License version 3 or any
later version.
If you'd like to contribute to borgmatic development, please feel free to
submit a [Pull Request](https://projects.torsion.org/witten/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/witten/borgmatic/issues) first
submit a [Pull Request](https://projects.torsion.org/borgmatic-collective/borgmatic/pulls)
or open an [issue](https://projects.torsion.org/borgmatic-collective/borgmatic/issues) first
to discuss your idea. We also accept Pull Requests on GitHub, if that's more
your thing. In general, contributions are very welcome. We don't bite!
@ -145,5 +145,5 @@ Also, please check out the [borgmatic development
how-to](https://torsion.org/borgmatic/docs/how-to/develop-on-borgmatic/) for
info on cloning source code, running tests, etc.
<a href="https://build.torsion.org/witten/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/witten/borgmatic/status.svg?ref=refs/heads/master)</a>
<a href="https://build.torsion.org/borgmatic-collective/borgmatic" alt="build status">![Build Status](https://build.torsion.org/api/badges/borgmatic-collective/borgmatic/status.svg?ref=refs/heads/master)</a>

9
SECURITY.md
View file

@ -6,14 +6,13 @@ permalink: security-policy/index.html
## Supported versions
While we want to hear about security vulnerabilities in all versions of
borgmatic, security fixes will only be made to the most recently released
version. It's not practical for our small volunteer effort to maintain
multiple different release branches and put out separate security patches for
each.
borgmatic, security fixes are only made to the most recently released version.
It's simply not practical for our small volunteer effort to maintain multiple
release branches and put out separate security patches for each.
## Reporting a vulnerability
If you find a security vulnerability, please [file a
ticket](https://torsion.org/borgmatic/#issues) or [send email
directly](mailto:witten@torsion.org) as appropriate. You should expect to hear
back within a few days at most, and generally sooner.
back within a few days at most and generally sooner.

14
borgmatic/borg/create.py
View file

@ -44,13 +44,18 @@ def _expand_home_directories(directories):
return tuple(os.path.expanduser(directory) for directory in directories)
def map_directories_to_devices(directories): # pragma: no cover
def map_directories_to_devices(directories):
'''
Given a sequence of directories, return a map from directory to an identifier for the device on
which that directory resides. This is handy for determining whether two different directories
are on the same filesystem (have the same device identifier).
which that directory resides or None if the path doesn't exist.
This is handy for determining whether two different directories are on the same filesystem (have
the same device identifier).
'''
return {directory: os.stat(directory).st_dev for directory in directories}
return {
directory: os.stat(directory).st_dev if os.path.exists(directory) else None
for directory in directories
}
def deduplicate_directories(directory_devices):
@ -82,6 +87,7 @@ def deduplicate_directories(directory_devices):
for parent in parents:
if (
pathlib.PurePath(other_directory) == parent
and directory_devices[directory] is not None
and directory_devices[other_directory] == directory_devices[directory]
):
if directory in deduplicated:

14
borgmatic/config/schema.yaml
View file

@ -135,12 +135,14 @@ properties:
type: string
description: |
Any paths matching these patterns are excluded from backups.
Globs and tildes are expanded. Do not backslash spaces in
path names. See the output of "borg help patterns" for more
details.
Globs and tildes are expanded. (Note however that a glob
pattern must either start with a glob or be an absolute
path.) Do not backslash spaces in path names. See the output
of "borg help patterns" for more details.
example:
- '*.pyc'
- /home/*/.cache
- '*/.vim*.tmp'
- /etc/ssl
- /home/user/path with spaces
exclude_from:
@ -298,7 +300,7 @@ properties:
$borg_base_directory/.config/borg/keys
example: /path/to/base/config/keys
umask:
type: string
type: integer
description: Umask to be used for borg create. Defaults to 0077.
example: 0077
lock_wait:
@ -639,7 +641,7 @@ properties:
Password with which to connect to the database.
Omitting a password will only work if PostgreSQL
is configured to trust the configured username
without a password, or you create a ~/.pgpass
without a password or you create a ~/.pgpass
file.
example: trustsome1
format:
@ -793,7 +795,7 @@ properties:
example:
https://cronhub.io/start/1f5e3410-254c-11e8-b61d-55875966d01
umask:
type: scalar
type: integer
description: |
Umask used when executing hooks. Defaults to the umask that
borgmatic is run with.

5
borgmatic/config/validate.py
View file

@ -110,7 +110,10 @@ def parse_configuration(config_filename, schema_filename, overrides=None):
override.apply_overrides(config, overrides)
normalize.normalize(config)
validator = jsonschema.Draft7Validator(schema)
try:
validator = jsonschema.Draft7Validator(schema)
except AttributeError:
validator = jsonschema.Draft4Validator(schema)
validation_errors = tuple(validator.iter_errors(config))
if validation_errors:

31
borgmatic/execute.py
View file

@ -59,11 +59,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
'''
# Map from output buffer to sequence of last lines.
buffer_last_lines = collections.defaultdict(list)
output_buffers = [
output_buffer_for_process(process, exclude_stdouts)
process_for_output_buffer = {
output_buffer_for_process(process, exclude_stdouts): process
for process in processes
if process.stdout or process.stderr
]
}
output_buffers = list(process_for_output_buffer.keys())
# Log output for each process until they all exit.
while True:
@ -71,8 +72,23 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
(ready_buffers, _, _) = select.select(output_buffers, [], [])
for ready_buffer in ready_buffers:
ready_process = process_for_output_buffer.get(ready_buffer)
# The "ready" process has exited, but it might be a pipe destination with other
# processes (pipe sources) waiting to be read from. So as a measure to prevent
# hangs, vent all processes when one exits.
if ready_process and ready_process.poll() is not None:
for other_process in processes:
if (
other_process.poll() is None
and other_process.stdout
and other_process.stdout not in output_buffers
):
# Add the process's output to output_buffers to ensure it'll get read.
output_buffers.append(other_process.stdout)
line = ready_buffer.readline().rstrip().decode()
if not line:
if not line or not ready_process:
continue
# Keep the last few lines of output in case the process errors, and we need the output for
@ -123,9 +139,12 @@ def log_outputs(processes, exclude_stdouts, output_log_level, borg_local_path):
if not output_buffer:
continue
remaining_output = output_buffer.read().rstrip().decode()
while True: # pragma: no cover
remaining_output = output_buffer.readline().rstrip().decode()
if not remaining_output:
break
if remaining_output: # pragma: no cover
logger.log(output_log_level, remaining_output)

14
docs/_includes/components/suggestion-link.html
View file

@ -1,17 +1,5 @@
<h2>Improve this documentation</h2>
<p>Have an idea on how to make this documentation even better? Use our <a
href="https://projects.torsion.org/witten/borgmatic/issues">issue tracker</a> to send your
href="https://projects.torsion.org/borgmatic-collective/borgmatic/issues">issue tracker</a> to send your
feedback!</p>
<script>
document.getElementById('_page').value = window.location.href;
window.sk=window.sk||function(){(sk.q=sk.q||[]).push(arguments)};
sk('form', 'init', {
id: '1d536680ab96',
element: '#suggestion-form'
});
</script>
<script defer src="https://js.statickit.com/statickit.js"></script>

10
docs/how-to/develop-on-borgmatic.md
View file

@ -10,17 +10,17 @@ eleventyNavigation:
To get set up to hack on borgmatic, first clone master via HTTPS or SSH:
```bash
git clone https://projects.torsion.org/witten/borgmatic.git
git clone https://projects.torsion.org/borgmatic-collective/borgmatic.git
```
Or:
```bash
git clone ssh://git@projects.torsion.org:3022/witten/borgmatic.git
git clone ssh://git@projects.torsion.org:3022/borgmatic-collective/borgmatic.git
```
Then, install borgmatic
"[editable](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs)"
"[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs)"
so that you can run borgmatic commands while you're hacking on them to
make sure your changes work.
@ -66,8 +66,6 @@ following:
tox -e black
```
Note that Black requires at minimum Python 3.6.
And if you get a complaint from the
[isort](https://github.com/timothycrosley/isort) Python import orderer, you
can ask isort to order your imports for you:
@ -118,7 +116,7 @@ See the Black, Flake8, and isort documentation for more information.
Each pull request triggers a continuous integration build which runs the test
suite. You can view these builds on
[build.torsion.org](https://build.torsion.org/witten/borgmatic), and they're
[build.torsion.org](https://build.torsion.org/borgmatic-collective/borgmatic), and they're
also linked from the commits list on each pull request.
## Documentation development

17
docs/how-to/set-up-backups.md
View file

@ -28,7 +28,7 @@ sudo pip3 install --user --upgrade borgmatic
This installs borgmatic and its commands at the `/root/.local/bin` path.
Your pip binary may have a different name than "pip3". Make sure you're using
Python 3, as borgmatic does not support Python 2.
Python 3.6+, as borgmatic does not support Python 2.
The next step is to ensure that borgmatic's commands available are on your
system `PATH`, so that you can run borgmatic:
@ -77,7 +77,7 @@ on a relatively dedicated system, then a global install can work out fine.
Besides the approaches described above, there are several other options for
installing borgmatic:
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/)
* [Docker image with scheduled backups](https://hub.docker.com/r/b3vis/borgmatic/) (+ Docker Compose files)
* [Docker base image](https://hub.docker.com/r/monachus/borgmatic/)
* [Debian](https://tracker.debian.org/pkg/borgmatic)
* [Ubuntu](https://launchpad.net/ubuntu/+source/borgmatic)
@ -250,7 +250,7 @@ that, you can configure a separate job runner to invoke it periodically.
### cron
If you're using cron, download the [sample cron
file](https://projects.torsion.org/witten/borgmatic/src/master/sample/cron/borgmatic).
file](https://projects.torsion.org/borgmatic-collective/borgmatic/src/master/sample/cron/borgmatic).
Then, from the directory where you downloaded it:
```bash
@ -258,7 +258,10 @@ sudo mv borgmatic /etc/cron.d/borgmatic
sudo chmod +x /etc/cron.d/borgmatic
```
You can modify the cron file if you'd like to run borgmatic more or less frequently.
If borgmatic is installed at a different location than
`/root/.local/bin/borgmatic`, edit the cron file with the correct path. You
can also modify the cron file if you'd like to run borgmatic more or less
frequently.
### systemd
@ -271,9 +274,9 @@ you may already have borgmatic systemd service and timer files. If so, you may
be able to skip some of the steps below.)
First, download the [sample systemd service
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.service)
and the [sample systemd timer
file](https://projects.torsion.org/witten/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
file](https://projects.torsion.org/borgmatic-collective/borgmatic/raw/branch/master/sample/systemd/borgmatic.timer).
Then, from the directory where you downloaded them:
@ -294,7 +297,7 @@ borgmatic to run.
If you run borgmatic in macOS with launchd, you may encounter permissions
issues when reading files to backup. If that happens to you, you may be
interested in an [unofficial work-around for Full Disk
Access](https://projects.torsion.org/witten/borgmatic/issues/293).
Access](https://projects.torsion.org/borgmatic-collective/borgmatic/issues/293).
## Colored output

15
sample/systemd/borgmatic.service
View file

@ -32,13 +32,16 @@ RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
# Restrict write access
# Change to 'ProtectSystem=strict' and uncomment 'ProtectHome' to make the whole file
# system read-only be default and uncomment 'ReadWritePaths' for the required write access.
# Add local repositroy paths to the list of 'ReadWritePaths' like '-/mnt/my_backup_drive'.
# To restrict write access further, change "ProtectSystem" to "strict" and uncomment
# "ReadWritePaths", "ReadOnlyPaths", "ProtectHome", and "BindPaths". Then add any local repository
# paths to the list of "ReadWritePaths" and local backup source paths to "ReadOnlyPaths". This
# leaves most of the filesystem read-only to borgmatic.
ProtectSystem=full
# ProtectHome=read-only
# ReadWritePaths=-/root/.config/borg -/root/.cache/borg -/root/.borgmatic
# ReadWritePaths=-/mnt/my_backup_drive
# ReadOnlyPaths=-/var/lib/my_backup_source
# This will mount a tmpfs on top of /root and pass through needed paths
# ProtectHome=tmpfs
# BindPaths=-/root/.cache/borg -/root/.cache/borg -/root/.borgmatic
CapabilityBoundingSet=CAP_DAC_READ_SEARCH CAP_NET_RAW

2
scripts/release
View file

@ -38,7 +38,7 @@ twine upload -r pypi dist/borgmatic-*-py3-none-any.whl dist/borgmatic-*-py3-none
release_changelog="$(cat NEWS | sed '/^$/q' | grep -v '^\S')"
escaped_release_changelog="$(echo "$release_changelog" | sed -z 's/\n/\\n/g' | sed -z 's/\"/\\"/g')"
curl --silent --request POST \
"https://projects.torsion.org/api/v1/repos/witten/borgmatic/releases" \
"https://projects.torsion.org/api/v1/repos/borgmatic-collective/borgmatic/releases" \
--header "Authorization: token $projects_token" \
--header "Accept: application/json" \
--header "Content-Type: application/json" \

4
scripts/run-full-tests
View file

@ -13,8 +13,8 @@ set -e
apk add --no-cache python3 py3-pip borgbackup postgresql-client mariadb-client
# If certain dependencies of black are available in this version of Alpine, install them.
apk add --no-cache py3-typed-ast py3-regex || true
python3 -m pip install --upgrade pip==20.2.4 setuptools==50.3.2
pip3 install tox==3.20.1
python3 -m pip install --upgrade pip==21.3.1 setuptools==58.2.0
pip3 install tox==3.24.4
export COVERAGE_FILE=/tmp/.coverage
tox --workdir /tmp/.tox --sitepackages
tox --workdir /tmp/.tox --sitepackages -e end-to-end

2
setup.py
View file

@ -1,6 +1,6 @@
from setuptools import find_packages, setup
VERSION = '1.5.16.dev0'
VERSION = '1.5.21.dev0'
setup(

72
tests/integration/test_execute.py
View file

@ -1,5 +1,6 @@
import logging
import subprocess
import sys
import pytest
from flexmock import flexmock
@ -98,7 +99,7 @@ def test_log_outputs_kills_other_processes_when_one_errors():
process, 2, 'borg'
).and_return(True)
other_process = subprocess.Popen(
['watch', 'true'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
['sleep', '2'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('exit_code_indicates_error').with_args(
other_process, None, 'borg'
@ -123,6 +124,75 @@ def test_log_outputs_kills_other_processes_when_one_errors():
assert error.value.output
def test_log_outputs_vents_other_processes_when_one_exits():
'''
Execute a command to generate a longish random string and pipe it into another command that
exits quickly. The test is basically to ensure we don't hang forever waiting for the exited
process to read the pipe, and that the string-generating process eventually gets vented and
exits.
'''
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
[
sys.executable,
'-c',
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
other_process = subprocess.Popen(
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
process, (process.stdout,)
).and_return(process.stderr)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, (process.stdout,)
).and_return(other_process.stdout)
flexmock(process.stdout).should_call('readline').at_least().once()
module.log_outputs(
(process, other_process),
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
)
def test_log_outputs_does_not_error_when_one_process_exits():
flexmock(module.logger).should_receive('log')
flexmock(module).should_receive('command_for_process').and_return('grep')
process = subprocess.Popen(
[
sys.executable,
'-c',
"import random, string; print(''.join(random.choice(string.ascii_letters) for _ in range(40000)))",
],
stdout=None, # Specifically test the case of a process without stdout captured.
stderr=None,
)
other_process = subprocess.Popen(
['true'], stdin=process.stdout, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
flexmock(module).should_receive('output_buffer_for_process').with_args(
process, (process.stdout,)
).and_return(process.stderr)
flexmock(module).should_receive('output_buffer_for_process').with_args(
other_process, (process.stdout,)
).and_return(other_process.stdout)
module.log_outputs(
(process, other_process),
exclude_stdouts=(process.stdout,),
output_log_level=logging.INFO,
borg_local_path='borg',
)
def test_log_outputs_truncates_long_error_output():
flexmock(module).ERROR_OUTPUT_MAX_LINE_COUNT = 0
flexmock(module.logger).should_receive('log')

25
tests/unit/borg/test_create.py
View file

@ -60,6 +60,30 @@ def test_expand_home_directories_considers_none_as_no_directories():
assert paths == ()
def test_map_directories_to_devices_gives_device_id_per_path():
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
flexmock(module.os).should_receive('stat').with_args('/bar').and_return(flexmock(st_dev=66))
device_map = module.map_directories_to_devices(('/foo', '/bar'))
assert device_map == {
'/foo': 55,
'/bar': 66,
}
def test_map_directories_to_devices_with_missing_path_does_not_error():
flexmock(module.os).should_receive('stat').with_args('/foo').and_return(flexmock(st_dev=55))
flexmock(module.os).should_receive('stat').with_args('/bar').and_raise(FileNotFoundError)
device_map = module.map_directories_to_devices(('/foo', '/bar'))
assert device_map == {
'/foo': 55,
'/bar': None,
}
@pytest.mark.parametrize(
'directories,expected_directories',
(
@ -72,6 +96,7 @@ def test_expand_home_directories_considers_none_as_no_directories():
({'/root': 1, '/root/foo/': 1}, ('/root',)),
({'/root': 1, '/root/foo': 2}, ('/root', '/root/foo')),
({'/root/foo': 1, '/root': 1}, ('/root',)),
({'/root': None, '/root/foo': None}, ('/root', '/root/foo')),
({'/root': 1, '/etc': 1, '/root/foo/bar': 1}, ('/etc', '/root')),
({'/root': 1, '/root/foo': 1, '/root/foo/bar': 1}, ('/root',)),
({'/dup': 1, '/dup': 1}, ('/dup',)),