Update README.md

This is better than trying to initialize it every time

.gitignore

@@ -21,9 +21,10 @@ restic_data/
 restic_cache/
 alerts.env
 
-# docs
+# build
 build/
 docs/_build
+dist
 
 # tests
 .tox

.travis.yml

@@ -0,0 +1,17 @@
+language: python
+
+sudo: false
+
+matrix:
+  include:
+      python: 3.7
+      dist: bionic
+      sudo: true
+
+install: 
+  - pip install -U setuptools pip wheel
+  - pip install -r src/tests/requirements.txt
+  - pip install ./src
+
+script:
+  - tox

README.md

@@ -3,7 +3,8 @@
 
 ![docs](https://readthedocs.org/projects/restic-compose-backup/badge/?version=latest)
 
-Backup using https://restic.net/ for a docker-compose setup.
+Backup using [restic] for a docker-compose setup.
+Currently tested with docker-ce 17, 18 and 19.
 
 * [restic-compose-backup Documentation](https://restic-compose-backup.readthedocs.io)
 * [restic-compose-backup on Github](https://github.com/ZettaIO/restic-compose-backup)
@@ -11,175 +12,145 @@ Backup using https://restic.net/ for a docker-compose setup.
 
 Features:
 
-* Back up docker volumes or host binds
+* Backs up docker volumes or host binds
-* Back up mariadb postgres
+* Backs up postgres, mariadb and mysql databases
-* Back up mariadb databases
+* Notifications over mail/smtp or Discord webhooks
-* Back up mysql databases
-* Notifications over mail/smtp
-* Notifications to Discord through webhooks
 
 Please report issus on [github](https://github.com/ZettaIO/restic-compose-backup/issues).
 
-Automatically detects and backs up volumes, mysql, mariadb and postgres databases in a docker-compose setup.
-
-* Each service in the compose setup is configured with a label
-  to enable backup of volumes or databases
-* When backup starts a new instance of the container is created
-  mapping in all the needed volumes. It will copy networks etc
-  to ensure databases can be reached
-* Volumes are mounted to `/volumes/<service_name>/<path>`
-  in the backup process container. `/volumes` is pushed into restic
-* Databases are backed up from stdin / dumps into restic using path `/databases/<service_name>/dump.sql`
-* Cron triggers backup at 2AM every day
-
 ## Install
 
 ```bash
 docker pull zettaio/restic-compose-backup
 ```
 
-## Configuration
+## Configuration (env vars)
 
-Required env variables for restic:
+Minimum configuration
 
 ```bash
 RESTIC_REPOSITORY
 RESTIC_PASSWORD
 ```
 
-Backend specific env vars : https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
+More config options can be found in the [documentation].
 
-Additional env vars:
+Restic backend specific env vars : https://restic.readthedocs.io/en/stable/040_backup.html#environment-variables
+
+## Compose Example
+
+We simply control what should be backed up by adding
+labels to our containers. More details are covered
+in the [documentation].
+
+restic-backup.env
 
 ```bash
-# Prune rules
+RESTIC_REPOSITORY=<whatever backend restic supports>
+RESTIC_PASSWORD=hopefullyasecturepw
+# snapshot prune rules
 RESTIC_KEEP_DAILY=7
 RESTIC_KEEP_WEEKLY=4
 RESTIC_KEEP_MONTHLY=12
 RESTIC_KEEP_YEARLY=3
-
+# Cron schedule. Run every day at 1am
-# Logging level (debug,info,warning,error)
+CRON_SCHEDULE="0 1 * * *"
-LOG_LEVEL=info
-
-# SMTP alerts
-EMAIL_HOST=my.mail.host
-EMAIL_PORT=465
-EMAIL_HOST_USER=johndoe
-EMAIL_HOST_PASSWORD=s3cr3tpassw0rd
-EMAIL_SEND_TO=johndoe@gmail.com
-
-# Discord webhook
-DISCORD_WEBHOOK=https://discordapp.com/api/webhooks/...
 ```
 
-### Volumes
+docker-compose.yaml
 
 ```yaml
 version: '3'
 services:
   # The backup service
   backup:
-    image: zettaio/restic-compose-backup
+    image: zettaio/restic-compose-backup:<version>
-    environment:
-      - RESTIC_REPOSITORY=<whatever restic supports>
-      - RESTIC_PASSWORD=hopefullyasecturepw
-      - RESTIC_KEEP_DAILY=7
-      - RESTIC_KEEP_WEEKLY=4
-      - RESTIC_KEEP_MONTHLY=12
-      - RESTIC_KEEP_YEARLY=3
     env_file:
-      - some_other_vars.env
+      - restic-backup.env
     volumes:
+      # We need to communicate with docker
       - /var/run/docker.sock:/tmp/docker.sock:ro
-
+      # Persistent storage of restic cache (greatly speeds up all restic operations)
-  example:
+      - cache:/cache
+  web:
     image: some_image
-    # Enable volume backup with label
     labels:
+      # Enables backup of the volumes below
       restic-compose-backup.volumes: true
-    # These volumes will be backed up
     volumes:
-      # Docker volume
       - media:/srv/media
-      # Host map
       - /srv/files:/srv/files
-
-volumes:
-  media:
-```
-
-A simple `include` and `exclude` filter is also available.
-
-```yaml
-  example:
-    image: some_image
-    labels:
-      restic-compose-backup.volumes: true
-      restic-compose-backup.volumes.include: "files,data"
-    volumes:
-      # Source don't match include filter. No backup.
-      - media:/srv/media
-      # Matches include filter
-      - files:/srv/files
-      - /srv/data:/srv/data
-
-volumes:
-  media:
-  files:
-
-```
-
-Exclude
-
-```yaml
-  example:
-    image: some_image
-    labels:
-      restic-compose-backup.volumes: true
-      restic-compose-backup.volumes.exclude: "media"
-    volumes:
-      # Excluded by filter
-      - media:/srv/media
-      # Backed up
-      - files:/srv/files
-      - /srv/data:/srv/data
-
-volumes:
-  media:
-  files:
-```
-
-### Databases
-
-Will dump databases directly into restic through stdin.
-They will appear in restic as a separate snapshot with
-path `/databases/<service_name>/dump.sql` or similar.
-
-```yaml
   mariadb:
     image: mariadb:10
     labels:
+      # Enables backup of this database
       restic-compose-backup.mariadb: true
-```
+    env_file:
-
+      mariadb-credentials.env
-```yaml
+    volumes:
+      - mysqldata:/var/lib/mysql
   mysql:
     image: mysql:5
     labels:
+      # Enables backup of this database
       restic-compose-backup.mysql: true
-```
+    env_file:
+      mysql-credentials.env
+    volumes:
+      - mysqldata:/var/lib/mysql
 
-```yaml
   postgres:
     image: postgres
     labels:
+      # Enables backup of this database
       restic-compose-backup.postgres: true
+    env_file:
+      postgres-credentials.env
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+
+volumes:
+  media:
+  mysqldata:
+  mariadbdata:
+  pgdata:
+  cache:
 ```
 
+## The `rcb` command
+
+Everything is controlled using the `rcb` command.
+After configuring backup with labels and restarted
+the affected services we can quickly view the
+result using the `status` subcommand.
+
+```bash
+$ docker-compose run --rm backup rcb status
+INFO: Status for compose project 'myproject'
+INFO: Repository: '<restic repository>'
+INFO: Backup currently running?: False
+INFO: --------------- Detected Config ---------------
+INFO: service: mysql
+INFO:  - mysql (is_ready=True)
+INFO: service: mariadb
+INFO:  - mariadb (is_ready=True)
+INFO: service: postgres
+INFO:  - postgres (is_ready=True)
+INFO: service: web
+INFO:  - volume: media
+INFO:  - volume: /srv/files
+```
+
+The `status` subcommand lists what will be backed up and
+even pings the database services checking their availability.
+The `restic` command can also be used directly in the container.
+
+More `rcb` commands can be found in the [documentation].
+
 ## Running Tests
 
 ```bash
-pip install -e src/
+pip install -e ./src/
 pip install -r src/tests/requirements.txt
 tox
 ```
@@ -194,3 +165,6 @@ python src/setup.py build_sphinx
 ## Contributing
 
 Contributions are welcome regardless of experience level. Don't hesitate submitting issues, opening partial or completed pull requests.
+
+[restic]: https://restic.net/
+[documentation]: https://restic-compose-backup.readthedocs.io

docker-compose.yaml

@@ -8,8 +8,9 @@ services:
     volumes:
       # Map in docker socket
       - /var/run/docker.sock:/tmp/docker.sock:ro
-      # Map backup database locally
+      # Map local restic repository for dev
       - ./restic_data:/restic_data
+      # Map restic cache
       - ./restic_cache:/cache
       # Map in project source in dev
       - ./src:/restic-compose-backup

docs/conf.py

@@ -22,8 +22,7 @@ copyright = '2019, Zetta.IO Technology AS'
 author = 'Zetta.IO Technology AS'
 
 # The full version, including alpha/beta/rc tags
-release = '0.3.0'
+release = '0.4.0'
-
 
 # -- General configuration ---------------------------------------------------
 

extras/release.md

@@ -2,6 +2,7 @@
 
 - Update version in `setup.py`
 - Update version in `docs/conf.py`
+- Update version in `restic_compose_backup/__init__.py`
 - Build and tag image
 - push: `docker push zettaio/restic-compose-backup:<version>`
 - Ensure RTD has new docs published
@@ -12,9 +13,9 @@ When releasing a bugfix version we need to update the
 main image as well.
 
 ```bash
-docker build . --tag zettaio/restic-compose-backup:0.3
+docker build src --tag zettaio/restic-compose-backup:0.3
-docker build . --tag zettaio/restic-compose-backup:0.3.1
+docker build src --tag zettaio/restic-compose-backup:0.3.3
 
 docker push zettaio/restic-compose-backup:0.3
-docker push zettaio/restic-compose-backup:0.3.1
+docker push zettaio/restic-compose-backup:0.3.3
 ```

restic_compose_backup.env

@@ -10,6 +10,7 @@ RESTIC_KEEP_MONTHLY=12
 RESTIC_KEEP_YEARLY=3
 
 LOG_LEVEL=info
+CRON_SCHEDULE=10 2 * * *
 
 # EMAIL_HOST=
 # EMAIL_PORT=

src/.dockerignore

@@ -1,19 +1,3 @@
-.venv/
-.vscode/
-extras/
-restic_cache/
-restic_data/
 tests/
-.gitignore
-*.env
-*.log
-docker-compose.yaml
-*.ini
-*.egg-info
 __pycache__
 .DS_Store
-.git
-.pytest_cache
-.dockerignore
-build/
-docs/

src/crontab

@@ -1 +1,2 @@
-0 2 * * * source /env.sh && rcb backup > /proc/1/fd/1
+10 2 * * * source /env.sh && rcb backup > /proc/1/fd/1
+

src/entrypoint.sh

@@ -3,6 +3,9 @@
 # Dump all env vars so we can source them in cron jobs
 printenv | sed 's/^\(.*\)$/export \1/g' > /env.sh
 
+# Write crontab
+rcb crontab > crontab
+
 # start cron in the foreground
 crontab crontab
 crond -f

src/restic_compose_backup/__init__.py

@@ -0,0 +1 @@
+__version__ = '0.4.0'

src/restic_compose_backup/alerts/discord.py

@@ -30,11 +30,14 @@ class DiscordWebhookAlert(BaseAlert):
     def send(self, subject: str = None, body: str = None, alert_type: str = None):
         """Send basic webhook request. Max embed size is 6000"""
         logger.info("Triggering discord webhook")
+        # NOTE: The title size is 2048
+        #       The max description size is 2048
+        #       Total embed size limit is 6000 characters (per embed)
         data = {
             'embeds': [
                 {
-                    'title': subject,
+                    'title': subject[-256:],
-                    'description': body[:5000],
+                    'description': body[-2048:] if body else "",
                 },
             ]
         }

src/restic_compose_backup/backup_runner.py

@@ -1,8 +1,7 @@
 import logging
 import os
-import docker
 
-from restic_compose_backup.config import Config
+from restic_compose_backup import utils
 
 logger = logging.getLogger(__name__)
 
@@ -10,8 +9,7 @@ logger = logging.getLogger(__name__)
 def run(image: str = None, command: str = None, volumes: dict = None,
         environment: dict = None, labels: dict = None, source_container_id: str = None):
     logger.info("Starting backup container")
-    config = Config()
+    client = utils.docker_client()
-    client = docker.DockerClient(base_url=config.docker_base_url)
 
     container = client.containers.run(
         image,
@@ -35,7 +33,13 @@ def run(image: str = None, command: str = None, volumes: dict = None,
             line = ""
             while True:
                 try:
-                    line += next(stream).decode()
+                    # Make log streaming work for docker ce 17 and 18.
+                    # For some reason strings are returned instead if bytes.
+                    data = next(stream)
+                    if isinstance(data, bytes):
+                        line += data.decode()
+                    elif isinstance(data, str):
+                        line += data
                     if line.endswith('\n'):
                         break
                 except StopIteration:
@@ -51,9 +55,9 @@ def run(image: str = None, command: str = None, volumes: dict = None,
             fd.write('\n')
             logger.info(line)
 
+    container.wait()
     container.reload()
     logger.debug("Container ExitCode %s", container.attrs['State']['ExitCode'])
-    container.stop()
     container.remove()
 
     return container.attrs['State']['ExitCode']

src/restic_compose_backup/cli.py

@@ -1,4 +1,5 @@
 import argparse
+import os
 import logging
 
 from restic_compose_backup import (
@@ -9,6 +10,7 @@ from restic_compose_backup import (
 )
 from restic_compose_backup.config import Config
 from restic_compose_backup.containers import RunningContainers
+from restic_compose_backup import cron, utils
 
 logger = logging.getLogger(__name__)
 
@@ -42,17 +44,35 @@ def main():
     elif args.action == 'alert':
         alert(config, containers)
 
+    elif args.action == 'version':
+        import restic_compose_backup
+        print(restic_compose_backup.__version__)
+
+    elif args.action == "crontab":
+        crontab(config)
+
 
 def status(config, containers):
     """Outputs the backup config for the compose setup"""
     logger.info("Status for compose project '%s'", containers.project_name)
     logger.info("Repository: '%s'", config.repository)
     logger.info("Backup currently running?: %s", containers.backup_process_running)
+
+    if containers.stale_backup_process_containers:
+        utils.remove_containers(containers.stale_backup_process_containers)
+
+    # Check if repository is initialized with restic snapshots
+    if not restic.is_initialized(config.repository):
+        logger.info("Could not get repository info. Attempting to initialize it.")
+        result = restic.init_repo(config.repository)
+        if result == 0:
+            logger.info("Successfully initialized repository: %s", config.repository)
+        else:
+            logger.error("Failed to initialize repository")
+
     logger.info("%s Detected Config %s", "-" * 25, "-" * 25)
 
-    logger.info("Initializing repository (may fail if already initalized)")
+    # Start making snapshots
-    restic.init_repo(config.repository)
-
     backup_containers = containers.containers_for_backup()
     for container in backup_containers:
         logger.info('service: %s', container.service_name)
@@ -79,7 +99,16 @@ def backup(config, containers):
     """Request a backup to start"""
     # Make sure we don't spawn multiple backup processes
     if containers.backup_process_running:
-        raise ValueError("Backup process already running")
+        alerts.send(
+            subject="Backup process container already running",
+            body=(
+                "A backup process container is already running. \n"
+                f"Id: {containers.backup_process_container.id}\n"
+                f"Name: {containers.backup_process_container.name}\n"
+            ),
+            alert_type='ERROR',
+        )
+        raise RuntimeError("Backup process already running")
 
     # Map all volumes from the backup container into the backup process container
     volumes = containers.this_container.volumes
@@ -96,7 +125,7 @@ def backup(config, containers):
             environment=containers.this_container.environment,
             source_container_id=containers.this_container.id,
             labels={
-                "restic-compose-backup.backup_process": 'True',
+                containers.backup_process_label: 'True',
                 "com.docker.compose.project": containers.project_name,
             },
         )
@@ -128,24 +157,38 @@ def start_backup_process(config, containers):
             "Cannot run backup process in this container. Use backup command instead. "
             "This will spawn a new container with the necessary mounts."
         )
-        return
+        exit(1)
 
     status(config, containers)
     errors = False
 
-    # Back up volumes
+    # Did we actually get any volumes mounted?
     try:
-        logger.info('Backing up volumes')
+        has_volumes = os.stat('/volumes') is not None
-        vol_result = restic.backup_files(config.repository, source='/volumes')
+    except FileNotFoundError:
-        logger.debug('Volume backup exit code: %s', vol_result)
+        logger.warning("Found no volumes to back up")
-        if vol_result != 0:
+        has_volumes = False
-            logger.error('Backup command exited with non-zero code: %s', vol_result)
+
+    # Warn if there is nothing to do
+    if len(containers.containers_for_backup()) == 0 and not has_volumes:
+        logger.error("No containers for backup found")
+        exit(1)
+
+    if has_volumes:
+        try:
+            logger.info('Backing up volumes')
+            vol_result = restic.backup_files(config.repository, source='/volumes')
+            logger.debug('Volume backup exit code: %s', vol_result)
+            if vol_result != 0:
+                logger.error('Volume backup exited with non-zero code: %s', vol_result)
+                errors = True
+        except Exception as ex:
+            logger.error('Exception raised during volume backup')
+            logger.exception(ex)
             errors = True
-    except Exception as ex:
-        logger.exception(ex)
-        errors = True
 
     # back up databases
+    logger.info('Backing up databases')
     for container in containers.containers_for_backup():
         if container.database_backup_enabled:
             try:
@@ -161,14 +204,25 @@ def start_backup_process(config, containers):
                 errors = True
 
     if errors:
+        logger.error('Exit code: %s', errors)
         exit(1)
 
     # Only run cleanup if backup was successful
     result = cleanup(config, container)
-    logger.debug('cleanup exit code: %s', errors)
+    logger.debug('cleanup exit code: %s', result)
     if result != 0:
+        logger.error('cleanup exit code: %s', result)
         exit(1)
 
+    # Test the repository for errors
+    logger.info("Checking the repository for errors")
+    result = restic.check(config.repository)
+    if result != 0:
+        logger.error('Check exit code: %s', result)
+        exit(1)
+
+    logger.info('Backup completed')
+
 
 def cleanup(config, containers):
     """Run forget / prune to minimize storage space"""
@@ -201,11 +255,25 @@ def alert(config, containers):
     )
 
 
+def crontab(config):
+    """Generate the crontab"""
+    print(cron.generate_crontab(config))
+
+
 def parse_args():
     parser = argparse.ArgumentParser(prog='restic_compose_backup')
     parser.add_argument(
         'action',
-        choices=['status', 'snapshots', 'backup', 'start-backup-process', 'alert', 'cleanup'],
+        choices=[
+            'status',
+            'snapshots',
+            'backup',
+            'start-backup-process',
+            'alert',
+            'cleanup',
+            'version',
+            'crontab',
+        ],
     )
     parser.add_argument(
         '--log-level',

src/restic_compose_backup/commands.py

@@ -53,14 +53,12 @@ def run(cmd: List[str]) -> int:
     child = Popen(cmd, stdout=PIPE, stderr=PIPE)
     stdoutdata, stderrdata = child.communicate()
 
-    if stdoutdata:
+    if stdoutdata.strip():
-        logger.debug(stdoutdata.decode().strip())
+        log_std('stdout', stdoutdata.decode(),
-        logger.debug('-' * 28)
+                logging.DEBUG if child.returncode == 0 else logging.ERROR)
 
-    if stderrdata:
+    if stderrdata.strip():
-        logger.error('%s STDERR %s', '-' * 10, '-' * 10)
+        log_std('stderr', stderrdata.decode(), logging.ERROR)
-        logger.error(stderrdata.decode().strip())
-        logger.error('-' * 28)
 
     logger.debug("returncode %s", child.returncode)
     return child.returncode
@@ -71,3 +69,23 @@ def run_capture_std(cmd: List[str]) -> Tuple[str, str]:
     logger.debug('cmd: %s', ' '.join(cmd))
     child = Popen(cmd, stdout=PIPE, stderr=PIPE)
     return child.communicate()
+
+
+def log_std(source: str, data: str, level: int):
+    if isinstance(data, bytes):
+        data = data.decode()
+
+    if not data.strip():
+        return
+
+    log_func = logger.debug if level == logging.DEBUG else logger.error
+    log_func('%s %s %s', '-' * 10, source, '-' * 10)
+
+    lines = data.split('\n')
+    if lines[-1] == '':
+        lines.pop()
+
+    for line in lines:
+        log_func(line)
+
+    log_func('-' * 28)

src/restic_compose_backup/config.py

@@ -2,12 +2,17 @@ import os
 
 
 class Config:
+    default_backup_command = "source /env.sh && rcb backup > /proc/1/fd/1"
+    default_crontab_schedule = "0 2 * * *"
+
     """Bag for config values"""
     def __init__(self, check=True):
         # Mandatory values
         self.repository = os.environ.get('RESTIC_REPOSITORY')
         self.password = os.environ.get('RESTIC_REPOSITORY')
         self.docker_base_url = os.environ.get('DOCKER_BASE_URL') or "unix://tmp/docker.sock"
+        self.cron_schedule = os.environ.get('CRON_SCHEDULE') or self.default_crontab_schedule
+        self.cron_command = os.environ.get('CRON_COMMAND') or self.default_backup_command
 
         # Log
         self.log_level = os.environ.get('LOG_LEVEL')

src/restic_compose_backup/containers.py

@@ -2,7 +2,8 @@ import os
 from pathlib import Path
 from typing import List
 
-from restic_compose_backup import utils
+from restic_compose_backup import enums, utils
+
 
 VOLUME_TYPE_BIND = "bind"
 VOLUME_TYPE_VOLUME = "volume"
@@ -27,8 +28,8 @@ class Container:
         if self._labels is None:
             raise ValueError('Container meta missing Config->Labels')
 
-        self._include = self._parse_pattern(self.get_label('restic-compose-backup.volumes.include'))
+        self._include = self._parse_pattern(self.get_label(enums.LABEL_VOLUMES_INCLUDE))
-        self._exclude = self._parse_pattern(self.get_label('restic-compose-backup.volumes.exclude'))
+        self._exclude = self._parse_pattern(self.get_label(enums.LABEL_VOLUMES_EXCLUDE))
 
     @property
     def instance(self) -> 'Container':
@@ -65,6 +66,9 @@ class Container:
         """All configured env vars for the container as a list"""
         return self.get_config('Env')
 
+    def remove(self):
+        self._data.remove()
+
     def get_config_env(self, name) -> str:
         """Get a config environment variable by name"""
         # convert to dict and fetch env var by name
@@ -107,7 +111,8 @@ class Container:
 
     @property
     def volume_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.volumes'))
+        """bool: If the ``restic-compose-backup.volumes`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_VOLUMES_ENABLED))
 
     @property
     def database_backup_enabled(self) -> bool:
@@ -120,24 +125,27 @@ class Container:
 
     @property
     def mysql_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.mysql'))
+        """bool: If the ``restic-compose-backup.mysql`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_MYSQL_ENABLED))
 
     @property
     def mariadb_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.mariadb'))
+        """bool: If the ``restic-compose-backup.mariadb`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_MARIADB_ENABLED))
 
     @property
     def postgresql_backup_enabled(self) -> bool:
-        return utils.is_true(self.get_label('restic-compose-backup.postgres'))
+        """bool: If the ``restic-compose-backup.postgres`` label is set"""
+        return utils.is_true(self.get_label(enums.LABEL_POSTGRES_ENABLED))
 
     @property
     def is_backup_process_container(self) -> bool:
         """Is this container the running backup process?"""
-        return self.get_label('restic-compose-backup.backup_process') == 'True'
+        return self.get_label(self.backup_process_label) == 'True'
 
     @property
     def is_running(self) -> bool:
-        """Is the container running?"""
+        """bool: Is the container running?"""
         return self._state.get('Running', False)
 
     @property
@@ -150,6 +158,11 @@ class Container:
         """Name of the container/service"""
         return self.get_label('com.docker.compose.service', default='')
 
+    @property
+    def backup_process_label(self) -> str:
+        """str: The unique backup process label for this project"""
+        return f"{enums.LABEL_BACKUP_PROCESS}-{self.project_name}"
+
     @property
     def project_name(self) -> str:
         """Name of the compose setup"""
@@ -310,6 +323,7 @@ class RunningContainers:
         self.containers = []
         self.this_container = None
         self.backup_process_container = None
+        self.stale_backup_process_containers = []
 
         # Find the container we are running in.
         # If we don't have this information we cannot continue
@@ -320,10 +334,20 @@ class RunningContainers:
         if not self.this_container:
             raise ValueError("Cannot find metadata for backup container")
 
-        # Gather all containers in the current compose setup
+        # Gather all running containers in the current compose setup
         for container_data in all_containers:
             container = Container(container_data)
 
+            # Gather stale backup process containers
+            if (self.this_container.image == container.image
+                    and not container.is_running
+                    and container.is_backup_process_container):
+                self.stale_backup_process_containers.append(container)
+
+            # We only care about running containers after this point
+            if not container.is_running:
+                continue
+
             # Detect running backup process container
             if container.is_backup_process_container:
                 self.backup_process_container = container
@@ -339,6 +363,11 @@ class RunningContainers:
         """str: Name of the compose project"""
         return self.this_container.project_name
 
+    @property
+    def backup_process_label(self) -> str:
+        """str: The backup process label for this project"""
+        return self.this_container.backup_process_label
+
     @property
     def backup_process_running(self) -> bool:
         """Is the backup process container running?"""
@@ -358,6 +387,7 @@ class RunningContainers:
         return mounts
 
     def get_service(self, name) -> Container:
+        """Container: Get a service by name"""
         for container in self.containers:
             if container.service_name == name:
                 return container

src/restic_compose_backup/cron.py

@@ -0,0 +1,69 @@
+"""
+# ┌───────────── minute (0 - 59)
+# │ ┌───────────── hour (0 - 23)
+# │ │ ┌───────────── day of the month (1 - 31)
+# │ │ │ ┌───────────── month (1 - 12)
+# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
+# │ │ │ │ │                                   7 is also Sunday on some systems)
+# │ │ │ │ │
+# │ │ │ │ │
+# * * * * * command to execute
+"""
+QUOTE_CHARS = ['"', "'"]
+
+
+def generate_crontab(config):
+    """Generate a crontab entry for running backup job"""
+    command = config.cron_command.strip()
+    schedule = config.cron_schedule
+
+    if schedule:
+        schedule = schedule.strip()
+        schedule = strip_quotes(schedule)
+        if not validate_schedule(schedule):
+            schedule = config.default_crontab_schedule
+    else:
+        schedule = config.default_crontab_schedule
+
+    return f'{schedule} {command}\n'
+
+
+def validate_schedule(schedule: str):
+    """Validate crontab format"""
+    parts = schedule.split()
+    if len(parts) != 5:
+        return False
+
+    for p in parts:
+        if p != '*' and not p.isdigit():
+            return False
+
+    minute, hour, day, month, weekday = parts
+    try:
+        validate_field(minute, 0, 59)
+        validate_field(hour, 0, 23)
+        validate_field(day, 1, 31)
+        validate_field(month, 1, 12)
+        validate_field(weekday, 0, 6)
+    except ValueError:
+        return False
+
+    return True
+
+
+def validate_field(value, min, max):
+    if value == '*':
+        return
+
+    i = int(value)
+    return min <= i <= max
+
+
+def strip_quotes(value: str):
+    """Strip enclosing single or double quotes if present"""
+    if value[0] in QUOTE_CHARS:
+        value = value[1:]
+    if value[-1] in QUOTE_CHARS:
+        value = value[:-1]
+
+    return value

src/restic_compose_backup/enums.py

@@ -0,0 +1,11 @@
+
+# Labels
+LABEL_VOLUMES_ENABLED = 'restic-compose-backup.volumes'
+LABEL_VOLUMES_INCLUDE = 'restic-compose-backup.volumes.include'
+LABEL_VOLUMES_EXCLUDE = 'restic-compose-backup.volumes.exclude'
+
+LABEL_MYSQL_ENABLED = 'restic-compose-backup.mysql'
+LABEL_POSTGRES_ENABLED = 'restic-compose-backup.postgres'
+LABEL_MARIADB_ENABLED = 'restic-compose-backup.mariadb'
+
+LABEL_BACKUP_PROCESS = 'restic-compose-backup.process'

src/restic_compose_backup/restic.py

@@ -40,33 +40,41 @@ def backup_from_stdin(repository: str, filename: str, source_command: List[str])
     ])
 
     # pipe source command into dest command
-    # NOTE: Using the default buffer size: io.DEFAULT_BUFFER_SIZE = 8192
+    source_process = Popen(source_command, stdout=PIPE, bufsize=65536)
-    #       We might want to tweak that to speed up large dumps.
+    dest_process = Popen(dest_command, stdin=source_process.stdout, stdout=PIPE, stderr=PIPE, bufsize=65536)
-    #       Actual tests tests must be done.
-    source_process = Popen(source_command, stdout=PIPE)
-    dest_process = Popen(dest_command, stdin=source_process.stdout, stdout=PIPE, stderr=PIPE)
     stdout, stderr = dest_process.communicate()
 
-    if stdout:
-        for line in stdout.decode().split('\n'):
-            logger.debug(line)
-
-    if stderr:
-        for line in stderr.decode().split('\n'):
-            logger.error(line)
-
     # Ensure both processes exited with code 0
     source_exit, dest_exit = source_process.poll(), dest_process.poll()
-    return 0 if (source_exit == 0 and dest_exit == 0) else 1
+    exit_code = 0 if (source_exit == 0 and dest_exit == 0) else 1
+
+    if stdout:
+        commands.log_std('stdout', stdout, logging.DEBUG if exit_code == 0 else logging.ERROR)
+
+    if stderr:
+        commands.log_std('stderr', stderr, logging.ERROR)
+
+    return exit_code
 
 
 def snapshots(repository: str, last=True) -> Tuple[str, str]:
+    """Returns the stdout and stderr info"""
     args = ["snapshots"]
     if last:
         args.append('--last')
     return commands.run_capture_std(restic(repository, args))
 
 
+def is_initialized(repository: str) -> bool:
+    """
+    Checks if a repository is initialized using snapshots command.
+    Note that this cannot separate between uninitalized repo
+    and other errors, but this method is reccomended by the restic
+    community.
+    """
+    return commands.run(restic(repository, ["snapshots", '--last'])) == 0
+
+
 def forget(repository: str, daily: str, weekly: str, monthly: str, yearly: str):
     return commands.run(restic(repository, [
         'forget',
@@ -90,6 +98,7 @@ def prune(repository: str):
 def check(repository: str):
     return commands.run(restic(repository, [
         "check",
+        # "--with-cache",
     ]))
 
 

src/restic_compose_backup/utils.py

@@ -1,26 +1,45 @@
 import os
+import logging
+from typing import List
 from contextlib import contextmanager
 import docker
-
 from restic_compose_backup.config import Config
 
+logger = logging.getLogger(__name__)
+
 TRUE_VALUES = ['1', 'true', 'True', True, 1]
 
 
-def list_containers():
+def docker_client():
+    config = Config()
+    return docker.DockerClient(base_url=config.docker_base_url)
+
+
+def list_containers() -> List[dict]:
     """
     List all containers.
 
     Returns:
         List of raw container json data from the api
     """
-    config = Config()
+    client = docker_client()
-    client = docker.DockerClient(base_url=config.docker_base_url)
+    all_containers = client.containers.list(all=True)
-    all_containers = client.containers.list()
     client.close()
     return [c.attrs for c in all_containers]
 
 
+def remove_containers(containers: List['Container']):
+    client = docker_client()
+    logger.info('Attempting to delete stale backup process containers')
+    for container in containers:
+        logger.info(' -> deleting %s', container.name)
+        try:
+            c = client.containers.get(container.name)
+            c.remove()
+        except Exception as ex:
+            logger.exception(ex)
+
+
 def is_true(value):
     """
     Evaluates the truthfullness of a bool value in container labels

src/setup.py

@@ -3,7 +3,7 @@ from setuptools import setup, find_namespace_packages
 setup(
     name="restic-compose-backup",
     url="https://github.com/ZettaIO/restic-compose-backup",
-    version="0.3.0",
+    version="0.4.0",
     author="Einar Forselv",
     author_email="eforselv@gmail.com",
     packages=find_namespace_packages(include=['restic_compose_backup']),

src/tests/tests.py

@@ -191,7 +191,7 @@ class ResticBackupTests(unittest.TestCase):
             {
                 'service': 'backup_runner',
                 'labels': {
-                    'restic-compose-backup.backup_process': 'True',
+                    'restic-compose-backup.process-default': 'True',
                 },
             },
         ]

tox.ini

@@ -14,7 +14,7 @@ basepython =
     py37: python3.7
 
 deps = 
-    -r{toxinidir}/tests/requirements.txt
+    -r{toxinidir}/src//tests/requirements.txt
 commands =
     ; coverage run --source=restic_compose_backup -m pytest tests/
     ; coverage report