Remove break-lock, add checking, and fix healthcheck

Dockerfile

@@ -15,7 +15,7 @@ COPY src /src
 COPY config/ssh_config /etc/ssh/
 COPY config/exclude.conf /exclude.conf
 
-# Add healthcheck to verify backup completed within allowed time
+# Check for backup failures and a success within the allowed time.
 ENV MAX_BACKUP_AGE_SECONDS=86400
 HEALTHCHECK --interval=10s --timeout=10s CMD /src/healthcheck.sh
 

README.md

@@ -18,18 +18,18 @@ Over the past 2 years, this backup setup has enabled me to successfully restore
   > I self-host multiple databases and this is the most feasible way of avoiding data corruption.
 - **Scheduled Backups**: Automates backups according to a defined schedule.
 - **Log Rotation**: Maintains weekly logs of all backup activities.
-- **Multi-Repository Backups**: Allows backups to multiple BorgBackup repositories simultaneously.
-- **Healthcheck**: The healthcheck is based on the time of the last successful backup.
+- **Multi-Repository Backups**: Backs up to multiple BorgBackup repositories sequentially.
+- **Healthcheck**: Reports failed backups and checks the age of the last successful backup.
 
 ### Multi-target backups
 
 Set the required `ARCHIVE_PREFIX` environment variable to a non-empty prefix shared by all backup targets. Include any separator in the value: `ARCHIVE_PREFIX=my-host-` creates archives named `my-host-{now:%Y-%m-%dT%H:%M:%S}`. Pruning uses the glob `${ARCHIVE_PREFIX}*`, so retention rules only apply to matching archives. To continue pruning existing archives, set the prefix to their previous hostname followed by `-`.
 
-To adhere to the [3-2-1 backup rule](https://en.wikipedia.org/wiki/Backup) without disk-level redundancy, you can configure backups to multiple destinations. For example, backups can be sent simultaneously to [rsync.net](rsync.net) and a local HDD.
+To adhere to the [3-2-1 backup rule](https://en.wikipedia.org/wiki/Backup) without disk-level redundancy, you can configure backups to multiple destinations. For example, backups can be sent to [rsync.net](rsync.net) and a local HDD.
 
 The [`docker-compose.yml`](docker-compose.yml) file demonstrates how to set up multiple backup targets using environment variables such as `BORG_REPO_0`, `BORG_REPO_1`, `BORG_PASSPHRASE_0`, `BORG_PASSPHRASE_1`, and so forth. The backup script sequentially handles each repository defined by the environment variables, ensuring your source volume is backed up across all specified targets.
 
-The backup script first takes `BORG_REPO_0` and the corresponding env vars and sets up the [`BORG_REPO`](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls), `BORG_REMOTE_PATH`, and `BORG_PASSPHRASE` environment variables for `borg`. Once the backup finished (successfully or otherwise), the script checks whether `BORG_REPO_1` exists, if so, it sets `BORG_REPO` and the other env vars to their expected values and backs up again. The script keeps going to `BORG_REPO_2`, `BORG_REPO_3` and so on as long as these are set. Otherwise, it unsets the previous `BORG_REPO` and corresponding env vars and goes to sleep.
+The backup script first takes `BORG_REPO_0` and the corresponding env vars and sets up the [`BORG_REPO`](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls), `BORG_REMOTE_PATH`, and `BORG_PASSPHRASE` environment variables for `borg`. Once the backup finished (successfully or otherwise), the script checks whether `BORG_REPO_1` exists, if so, it sets `BORG_REPO` and the other env vars to their expected values and backs up again. The script keeps going to `BORG_REPO_2`, `BORG_REPO_3` and so on as long as these are set. It then checks each repository with the corresponding environment before starting the next backup cycle.
 
 Thus, the following sets of environment variables are valid for multi-target backups:
 
@@ -72,17 +72,30 @@ Thus, the following sets of environment variables are valid for multi-target bac
 
   > This first back up to a remote repository, then to a local one
 
+### Sharing repositories
+
+Multiple containers can back up different sources to the same repository, using distinct, non-overlapping `ARCHIVE_PREFIX` values and separate cache and snapshot directories. Borg 1.4 serializes writes with repository locks. Each Borg command waits indefinitely for a repository or cache lock, with no maximum wait setting. Because Borg 1.4's CLI requires a finite wait per attempt, the wrapper retries lock timeouts every 60 seconds until the lock becomes available. Other failures stop that target and are reflected in container health.
+
+### Checks between backups
+
+After attempting every backup target, the wrapper runs `borg check --repository-only --max-duration=SECONDS --info` on each configured repository, including targets whose backup failed. Checks use the same credentials, remote executable and indefinite lock waiting as backups. They do not initialize, repair, prune or compact repositories. Any check warning or error immediately stops the cycle and the scheduler with Borg's nonzero exit status; later targets are not checked in that cycle. Lock contention continues waiting indefinitely.
+
+Borg resumes partial checks from the last segment checked. These checks inspect segment checksums; they do not verify repository indexes, archive metadata or cryptographic data integrity, and do not replace periodic full checks. See [Borg's partial-check documentation](https://borgbackup.readthedocs.io/en/1.4.0/usage/check.html).
+
 ### Healthcheck
 
-The container is healthy when its last successful backup is less than `MAX_BACKUP_AGE_SECONDS` old (default: `86400`, or one day). All configured targets must succeed. Set this to a positive integer that allows enough time for backups and `SLEEP_TIME`.
+The container is healthy when no backup or repository-check failure is recorded and its last successful cycle is less than `MAX_BACKUP_AGE_SECONDS` old (default: `86400`, or one day). All configured targets must succeed. Set this to a positive integer that allows enough time for backups, checks and lock waits.
 
-Before the first successful backup, the same limit provides a startup grace period. Failed backups leave the last success time unchanged, and restarting does not hide an overdue backup.
+Before the first successful backup, the same limit provides a startup grace period, which ends immediately on a failed target or invalid configuration. A failure makes health checks fail even if a previous success or the startup timestamp is recent. Docker marks the container unhealthy after its configured healthcheck retries. The backup failure marker persists in `/health` across restarts and retries and is cleared only after all backups and scheduled checks succeed. Only then is the completion timestamp updated.
+
+A failed repository check also creates `/health/check_failed`. While it exists, the wrapper refuses to start any backups or checks, including after Docker restarts the container. This prevents a later partial check from skipping a damaged segment and hiding the failure. Investigate the failure and complete full checks of the configured repositories before manually removing this marker from that container's health volume. Scheduled checks never clear it automatically.
 
 ## Repository layout
 
 - src
   - [backup.sh](src/backup.sh): Creates a new BorgBackup repository if none exists, takes a snapshot of the BTRFS volume, performs the backup, and prunes old backups.
-  - [backup-wrapper.sh](src/backup-wrapper.sh): Executes backup.sh for each repository configured via environment variables.
+  - [backup-wrapper.sh](src/backup-wrapper.sh): Backs up all configured targets, then performs time-limited repository checks.
+  - [borg-common.sh](src/borg-common.sh): Shares Borg SSH settings and indefinite lock waiting between backups and checks.
   - [schedule.sh](src/schedule.sh): Manages and logs the operation of backup-wrapper.sh and runs it in a continuous loop.
 - config
   - [exclude.conf](config/exclude.conf): Exclude list for `borg`. Files matching these patterns won't be backed up.

docker-compose.yml

@@ -5,7 +5,7 @@ services:
     init: true
     environment:
       - TZ=Europe/London
-      - SLEEP_TIME=1h # we will wait an hour between backups
+      - SLEEP_TIME=1h # total repository-check budget between backup rounds
       - ARCHIVE_PREFIX=my-host- # required; include any separator before the timestamp
 
       # first backup goes to rsync.net

src/backup-wrapper.sh

@@ -2,15 +2,56 @@
 
 echo "Starting backup wrapper script at $(date)"
 
+mkdir -p /health || exit 1
+
+# Called indirectly by the EXIT trap, including configuration failures.
+# shellcheck disable=SC2317
+finish() {
+    local status=$?
+    if (( status != 0 )); then
+        touch /health/backup_failed || echo "Cannot record backup failure" >&2
+    fi
+    rm -f /health/backup_completion_time.log.$$
+    exit "$status"
+}
+trap 'finish' EXIT
+
+if [ -e /health/check_failed ]; then
+    echo "A previous repository check failed. Investigate and run full checks before clearing /health/check_failed." >&2
+    exit 1
+fi
+
+# shellcheck source=src/borg-common.sh
+source /src/borg-common.sh || exit 1
+
 execute_script() {
-    echo "Executing script with:"
+    echo "Executing $operation with:"
     if [ -n "$BORG_PASSPHRASE" ]; then
         echo "BORG_PASSPHRASE=<redacted>"
     fi
     echo "BORG_REMOTE_PATH='${BORG_REMOTE_PATH}'"
     echo "BORG_REPO='${BORG_REPO}'"
 
-    /src/backup.sh
+    local status=0
+    if [[ $operation == backup ]]; then
+        /src/backup.sh || status=$?
+    else
+        echo "Checking repository for up to $check_duration seconds"
+        borg_wait check --repository-only --max-duration="$check_duration" --info || status=$?
+        if (( status != 0 )); then
+            # Later partial checks may skip the damaged segment. Only clear
+            # this marker manually after investigating and fully checking it.
+            touch /health/check_failed || status=1
+            echo "Repository check failed (Borg exit status $status). Stopping backup cycle." >&2
+            exit "$status"
+        fi
+    fi
+    if (( status != 0 )); then
+        # Report failure while later targets are still running, and keep it
+        # across retries and container restarts until every target succeeds.
+        touch /health/backup_failed || return 1
+    fi
+    return "$status"
 }
 
 configure_environment() {
@@ -25,55 +66,103 @@ configure_environment() {
     fi
 }
 
+duration_seconds() {
+    local duration=$1 whole fraction multiplier seconds
+    local index product carry=0 remainder=0
+    local range_error='SLEEP_TIME must represent between 1 and 2147483647 seconds.'
+
+    if [[ ! $duration =~ ^([0-9]+([.][0-9]*)?|[.][0-9]+)([smhd]?)$ ]]; then
+        echo 'SLEEP_TIME must be a positive duration in seconds, minutes, hours or days.' >&2
+        return 1
+    fi
+    duration=${BASH_REMATCH[1]}
+    case ${BASH_REMATCH[3]} in
+        ''|s) multiplier=1 ;;
+        m) multiplier=60 ;;
+        h) multiplier=3600 ;;
+        d) multiplier=86400 ;;
+    esac
+
+    whole=${duration%%.*}
+    fraction=${duration#"$whole"}
+    fraction=${fraction#.}
+    # Strip leading zeros and bound the length before using Bash arithmetic.
+    whole=${whole#"${whole%%[!0]*}"}
+    if (( ${#whole} > 10 )); then
+        echo "$range_error" >&2
+        return 1
+    fi
+    seconds=$((10#${whole:-0} * multiplier))
+
+    # Multiply fractional digits from right to left, retaining any remainder
+    # so values above the upper limit are rejected before truncating seconds.
+    for ((index = ${#fraction} - 1; index >= 0; index--)); do
+        product=$((10#${fraction:index:1} * multiplier + carry))
+        carry=$((product / 10))
+        remainder=$((remainder || product % 10))
+    done
+    seconds=$((seconds + carry))
+    if (( seconds < 1 || seconds > 2147483647 || (seconds == 2147483647 && remainder) )); then
+        echo "$range_error" >&2
+        return 1
+    fi
+    echo "$seconds"
+}
+
 main() {
-    if [ -n "$BORG_REPO" ]; then
-        # fallback case if multi-target backup isn't needed
-        if ! execute_script; then
-            echo "Skipping completion log due to backup failure"
+    local check_budget check_duration operation
+    local index target_count=1 any_failed=false completed_indices indexed_var_name
+    # Preserve the existing SLEEP_TIME setting, including fractional durations.
+    check_budget=$(duration_seconds "${SLEEP_TIME:-1h}") || return 1
+
+    # Finish every backup before spending the former sleep interval checking.
+    for operation in backup check; do
+        check_duration=$((check_budget / target_count))
+        if (( check_duration == 0 )); then
+            echo "SLEEP_TIME must allow at least one second per repository."
             return 1
         fi
-    else
-        local index=0
-        local any_failed=false
-        local completed_indices=" "
-
-        while configure_environment "$index"; do
+        if [ -n "$BORG_REPO" ]; then
             execute_script || any_failed=true
-            completed_indices+="$index "
-            unset BORG_PASSPHRASE BORG_REMOTE_PATH BORG_REPO
-            ((index++))
-        done
+        else
+            index=0
+            completed_indices=" "
+            while configure_environment "$index"; do
+                execute_script || any_failed=true
+                completed_indices+="$index "
+                unset BORG_PASSPHRASE BORG_REMOTE_PATH BORG_REPO
+                ((index++))
+            done
+            target_count=$index
 
-        echo "Finished backup script at $(date)"
-        if (( index == 0 )); then
-            echo "No valid configuration found. Please ensure environment variables are set properly."
-            return 1
+            if (( index == 0 )); then
+                echo "No valid configuration found. Please ensure environment variables are set properly."
+                return 1
+            fi
+
+            # A missing or incomplete target must not silently hide later targets.
+            for indexed_var_name in ${!BORG_REPO_@} ${!BORG_PASSPHRASE_@} ${!BORG_REMOTE_PATH_@}; do
+                [[ -n ${!indexed_var_name} ]] || continue
+                case "$completed_indices" in
+                    *" ${indexed_var_name##*_} "*) ;;
+                    *)
+                        echo "Invalid or non-contiguous backup configuration: $indexed_var_name. Skipping completion log."
+                        return 1
+                        ;;
+                esac
+            done
         fi
+    done
 
-        # A missing or incomplete target must not silently hide later targets.
-        local indexed_var_name
-        for indexed_var_name in ${!BORG_REPO_@} ${!BORG_PASSPHRASE_@} ${!BORG_REMOTE_PATH_@}; do
-            [[ -n ${!indexed_var_name} ]] || continue
-            case "$completed_indices" in
-                *" ${indexed_var_name##*_} "*) ;;
-                *)
-                    echo "Invalid or non-contiguous backup configuration: $indexed_var_name. Skipping completion log."
-                    return 1
-                    ;;
-            esac
-        done
-
-        if [[ $any_failed == true ]]; then
-            echo "Skipping completion log due to backup failure(s)"
-            return 1
-        fi
+    if [[ $any_failed == true ]]; then
+        echo "Skipping completion log due to backup or check failure(s)"
+        return 1
     fi
 
-    # Rename only a complete timestamp, leaving the previous success intact on failure.
-    mkdir -p /health || return 1
-    trap 'rm -f /health/backup_completion_time.log.$$' EXIT
+    # Clear the failure only after publishing a complete, successful timestamp.
     date -u '+%Y-%m-%dT%H:%M:%SZ' > /health/backup_completion_time.log.$$ &&
-        mv -fT /health/backup_completion_time.log.$$ /health/backup_completion_time.log
+        mv -fT /health/backup_completion_time.log.$$ /health/backup_completion_time.log &&
+        rm -f /health/backup_failed
 }
 
 main

src/backup.sh

@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -e
+
 : "${ARCHIVE_PREFIX:?Set ARCHIVE_PREFIX to a non-empty archive prefix (e.g. my-host-)}"
 
 KEEP_DAILY=${KEEP_DAILY:-6}
@@ -9,20 +11,30 @@ KEEP_YEARLY=${KEEP_YEARLY:-10}
 
 echo "Starting backup script at $(date)"
 
-export BORG_RSH='ssh -oBatchMode=yes' # https://borgbackup.readthedocs.io/en/stable/usage/notes.html#ssh-batch-mode
+# shellcheck source=src/borg-common.sh
+source /src/borg-common.sh
 
-# break any stale locks in case the script was interrupted
-borg break-lock
-
-if ! borg info; then
-    echo "Borg info returned a non-zero status. Initializing Borg..."
-    borg init --encryption=repokey
-fi
-
-# The above command will fail if the repo hasn't been already initialized,
-# so we can ignore the return status. However, if any of the commands below fail,
-# we want to stop the script immediately.
-set -e
+# Only a missing repository (modern exit code 13) triggers initialization.
+info_status=0
+borg_wait info || info_status=$?
+case "$info_status" in
+    0) ;;
+    13)
+        echo "Repository does not exist. Initializing Borg..."
+        init_status=0
+        borg_wait init --encryption=repokey || init_status=$?
+        case "$init_status" in
+            0) ;;
+            # Another client may have initialized the repository in between.
+            10) borg_wait info ;;
+            *) exit "$init_status" ;;
+        esac
+        ;;
+    *)
+        echo "Cannot access repository (Borg exit status $info_status). Skipping backup." >&2
+        exit "$info_status"
+        ;;
+esac
 
 cleanup() {
     if [ -n "${GIT_EXCLUDE_FILE:-}" ] && [ -f "$GIT_EXCLUDE_FILE" ]; then
@@ -68,7 +80,7 @@ if [ "${IGNORE_GIT_UNTRACKED:-false}" = "true" ]; then
     EXCLUDE_ARGS+=(--exclude-from "$GIT_EXCLUDE_FILE")
 fi
 
-borg create --stats \
+borg_wait create --stats \
     --list \
     --filter=AMCE \
     --files-cache=ctime,size,inode \
@@ -77,11 +89,11 @@ borg create --stats \
 
 cd -
 
-borg prune --list --stats \
+borg_wait prune --list --stats \
     --glob-archives="${ARCHIVE_PREFIX}*" \
     --keep-daily="$KEEP_DAILY" \
     --keep-weekly="$KEEP_WEEKLY" \
     --keep-monthly="$KEEP_MONTHLY" \
     --keep-yearly="$KEEP_YEARLY"
 
-borg compact --threshold=5 --cleanup-commits --verbose --progress
+borg_wait compact --threshold=5 --cleanup-commits --verbose --progress

src/borg-common.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+export BORG_RSH='ssh -oBatchMode=yes'
+
+# Never break another client's repository/cache lock. Borg 1.4 only accepts
+# finite waits, so retry lock timeouts forever; other errors must propagate.
+borg_wait() {
+    local status
+    while true; do
+        if BORG_EXIT_CODES=modern borg "$@" --lock-wait=60; then
+            return 0
+        else
+            status=$?
+        fi
+        if (( status != 73 )); then
+            return "$status"
+        fi
+        echo "Repository or cache is locked; continuing to wait..." >&2
+    done
+}

src/healthcheck.sh

@@ -13,6 +13,9 @@ if ! [ "$MAX_BACKUP_AGE_SECONDS" -gt 0 ] 2>/dev/null; then
     unhealthy "MAX_BACKUP_AGE_SECONDS must be a positive integer."
 fi
 
+[ ! -e /health/check_failed ] || unhealthy "A repository check failed. Investigate and run full checks before clearing /health/check_failed."
+[ ! -e /health/backup_failed ] || unhealthy "A backup failed. Check the backup logs; all targets must succeed to recover."
+
 timestamp_file=/health/backup_completion_time.log
 [ -f "$timestamp_file" ] || timestamp_file=/health/container_start_time.log
 

src/schedule.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 
-SLEEP_TIME=${SLEEP_TIME:-1h}
+# Preserve the wrapper's failure through the logging pipeline and stop the loop.
+set -eo pipefail
 
 log_message() {
     stdbuf -o0 tee -a "$(get_log_file_name)"
@@ -17,9 +18,7 @@ date -u '+%Y-%m-%dT%H:%M:%SZ' > /health/container_start_time.log.$$ &&
     mv -fT /health/container_start_time.log.$$ /health/container_start_time.log || exit 1
 
 while true; do
+    # The wrapper checks every repository after backups, using SLEEP_TIME as
+    # its total check budget. Start the next cycle as soon as checks finish.
     /src/backup-wrapper.sh 2>&1 | log_message
-    echo "Sleeping for $SLEEP_TIME" | log_message
-
-    # Using a simple sleep loop to schedule backups instead of cron to avoid concurrency issues
-    sleep "$SLEEP_TIME"
 done