Handle nested subvolumes & remove broken git exclusion

Dockerfile

@@ -8,7 +8,7 @@ RUN apk --no-cache add \
     openssh \
     bash \
     coreutils \
-    git \
+    tzdata \
     borgbackup=1.4.0-r0
 
 COPY src /src

README.md

@@ -6,7 +6,7 @@ Create a snapshot of a [BTRFS](https://docs.kernel.org/filesystems/btrfs.html) v
 
 1. Review and modify the [docker-compose.yml](docker-compose.yml) file to set your environment variables as needed
 2. Customise the [exclude.conf](config/exclude.conf) according to your requirements
-3. Execute the command `docker compose --env-file ./config/default.env up` to spin up the container
+3. Build the updated image with `docker build -t ghcr.io/schmelczer/backup-container:latest .`, then run `docker compose up -d`
 
 ## Background
 
@@ -14,7 +14,7 @@ Over the past 2 years, this backup setup has enabled me to successfully restore
 
 ## Features
 
-- **Snapshotting**: Takes a snapshot of a BTRFS volume to ensure file consistency during backups.
+- **Snapshotting**: Takes snapshots of a BTRFS subvolume and its nested subvolumes to preserve each subvolume's file state during backups.
   > I self-host multiple databases and this is the most feasible way of avoiding data corruption.
 - **Scheduled Backups**: Automates backups according to a defined schedule.
 - **Log Rotation**: Maintains weekly logs of all backup activities.
@@ -23,7 +23,7 @@ Over the past 2 years, this backup setup has enabled me to successfully restore
 
 ### Multi-target backups
 
-Set the required `ARCHIVE_PREFIX` environment variable to a non-empty prefix shared by all backup targets. Include any separator in the value: `ARCHIVE_PREFIX=my-host-` creates archives named `my-host-{now:%Y-%m-%dT%H:%M:%S}`. Pruning uses the glob `${ARCHIVE_PREFIX}*`, so retention rules only apply to matching archives. To continue pruning existing archives, set the prefix to their previous hostname followed by `-`.
+Set the required `ARCHIVE_PREFIX` environment variable to a non-empty prefix shared by all backup targets. Include any separator in the value: `ARCHIVE_PREFIX=my-host-` creates archives named `my-host-{now:%Y-%m-%dT%H:%M:%S}`. The prefix is literal: glob characters and braces are escaped before passing it to Borg. Retention rules only apply to names beginning with that literal prefix. To continue pruning existing archives, set the prefix to their previous hostname followed by `-`.
 
 To adhere to the [3-2-1 backup rule](https://en.wikipedia.org/wiki/Backup) without disk-level redundancy, you can configure backups to multiple destinations. For example, backups can be sent to [rsync.net](rsync.net) and a local HDD.
 
@@ -78,13 +78,23 @@ Multiple containers can back up different sources to the same repository, using
 
 ### Checks between backups
 
+`SLEEP_TIME` (default `1h`) is the minimum interval from the end of a backup round to the start of the next one. Repository checks share this time budget equally. If checks finish early, the wrapper sleeps for the remaining interval; checks and lock waits that exceed the budget add no further sleep. Durations may use `s`, `m`, `h`, or `d`, including fractions such as `1.5h`. The duration must be between 1 and 2147483647 seconds and allow at least one whole second per repository. The image includes timezone data, so `TZ` controls archive timestamps and Borg's retention-day boundaries.
+
 After attempting every backup target, the wrapper runs `borg check --repository-only --max-duration=SECONDS --info` on each configured repository, including targets whose backup failed. Checks use the same credentials, remote executable and indefinite lock waiting as backups. They do not initialize, repair, prune or compact repositories. Any check warning or error immediately stops the cycle and the scheduler with Borg's nonzero exit status; later targets are not checked in that cycle. Lock contention continues waiting indefinitely.
 
 Borg resumes partial checks from the last segment checked. These checks inspect segment checksums; they do not verify repository indexes, archive metadata or cryptographic data integrity, and do not replace periodic full checks. See [Borg's partial-check documentation](https://borgbackup.readthedocs.io/en/1.4.0/usage/check.html).
 
+### Snapshot coverage and exclusions
+
+Mount a Btrfs subvolume at `/btrfs-root`, and a writable directory on the same Btrfs filesystem at `/snapshot`. The backup assembles snapshots of the root and every nested subvolume, including Btrfs subvolumes mounted below the source. Child snapshots are taken sequentially: each is consistent individually, but there is no single atomic snapshot across subvolumes. Applications spanning subvolumes may need to be paused to obtain a consistent backup.
+
+Failure to snapshot a child fails the backup. A mounted child on a different Btrfs filesystem cannot be snapshotted into the same destination; configure a separate backup container for it. Non-Btrfs mounts are outside snapshot coverage. A mount point hiding nonempty underlying directories is rejected rather than discarding that hidden data. The temporary snapshot itself is omitted, and cleanup deletes child snapshots before their parents.
+
+Set `BACKUP_RELATIVE_PATH=/path/within/source` to select a directory within the assembled snapshot. The default exclusions match `.env`, `.dev.env`, `node_modules`, `.venv`, and `__pycache__` at both the backup root and deeper levels. Edit [config/exclude.conf](config/exclude.conf) and rebuild the image to change these exclusions.
+
 ### Healthcheck
 
-The container is healthy when no backup or repository-check failure is recorded and its last successful cycle is less than `MAX_BACKUP_AGE_SECONDS` old (default: `86400`, or one day). All configured targets must succeed. Set this to a positive integer that allows enough time for backups, checks and lock waits.
+The container is healthy when no backup or repository-check failure is recorded and its last successful cycle is less than `MAX_BACKUP_AGE_SECONDS` old (default: `86400`, or one day). All configured targets must succeed. Set this to a positive integer that allows enough time for backups, checks, lock waits and the configured sleep interval.
 
 Before the first successful backup, the same limit provides a startup grace period, which ends immediately on a failed target or invalid configuration. A failure makes health checks fail even if a previous success or the startup timestamp is recent. Docker marks the container unhealthy after its configured healthcheck retries. The backup failure marker persists in `/health` across restarts and retries and is cleared only after all backups and scheduled checks succeed. Only then is the completion timestamp updated.
 
@@ -96,6 +106,8 @@ A failed repository check also creates `/health/check_failed`. While it exists,
   - [backup.sh](src/backup.sh): Creates a new BorgBackup repository if none exists, takes a snapshot of the BTRFS volume, performs the backup, and prunes old backups.
   - [backup-wrapper.sh](src/backup-wrapper.sh): Backs up all configured targets, then performs time-limited repository checks.
   - [borg-common.sh](src/borg-common.sh): Shares Borg SSH settings and indefinite lock waiting between backups and checks.
+  - [snapshot.sh](src/snapshot.sh): Assembles nested snapshots and deletes them in reverse order.
+  - [interval.sh](src/interval.sh): Validates durations and waits for the unused check interval using system uptime, with 10 ms resolution.
   - [schedule.sh](src/schedule.sh): Manages and logs the operation of backup-wrapper.sh and runs it in a continuous loop.
 - config
   - [exclude.conf](config/exclude.conf): Exclude list for `borg`. Files matching these patterns won't be backed up.

config/exclude.conf

@@ -1,9 +1,9 @@
 # https://borgbackup.readthedocs.io/en/stable/usage/create.html#description
 # https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-patterns
 
-*/node_modules
-*/.env
-*/.dev.env
-*/__pycache__
-*/.venv
+sh:**/node_modules
+sh:**/.env
+sh:**/.dev.env
+sh:**/__pycache__
+sh:**/.venv
 /snapshot

docker-compose.yml

@@ -5,7 +5,7 @@ services:
     init: true
     environment:
       - TZ=Europe/London
-      - SLEEP_TIME=1h # total repository-check budget between backup rounds
+      - SLEEP_TIME=1h # minimum interval after backups; checks run during this time
       - ARCHIVE_PREFIX=my-host- # required; include any separator before the timestamp
 
       # first backup goes to rsync.net

src/backup-wrapper.sh

@@ -66,57 +66,17 @@ configure_environment() {
     fi
 }
 
-duration_seconds() {
-    local duration=$1 whole fraction multiplier seconds
-    local index product carry=0 remainder=0
-    local range_error='SLEEP_TIME must represent between 1 and 2147483647 seconds.'
-
-    if [[ ! $duration =~ ^([0-9]+([.][0-9]*)?|[.][0-9]+)([smhd]?)$ ]]; then
-        echo 'SLEEP_TIME must be a positive duration in seconds, minutes, hours or days.' >&2
-        return 1
-    fi
-    duration=${BASH_REMATCH[1]}
-    case ${BASH_REMATCH[3]} in
-        ''|s) multiplier=1 ;;
-        m) multiplier=60 ;;
-        h) multiplier=3600 ;;
-        d) multiplier=86400 ;;
-    esac
-
-    whole=${duration%%.*}
-    fraction=${duration#"$whole"}
-    fraction=${fraction#.}
-    # Strip leading zeros and bound the length before using Bash arithmetic.
-    whole=${whole#"${whole%%[!0]*}"}
-    if (( ${#whole} > 10 )); then
-        echo "$range_error" >&2
-        return 1
-    fi
-    seconds=$((10#${whole:-0} * multiplier))
-
-    # Multiply fractional digits from right to left, retaining any remainder
-    # so values above the upper limit are rejected before truncating seconds.
-    for ((index = ${#fraction} - 1; index >= 0; index--)); do
-        product=$((10#${fraction:index:1} * multiplier + carry))
-        carry=$((product / 10))
-        remainder=$((remainder || product % 10))
-    done
-    seconds=$((seconds + carry))
-    if (( seconds < 1 || seconds > 2147483647 || (seconds == 2147483647 && remainder) )); then
-        echo "$range_error" >&2
-        return 1
-    fi
-    echo "$seconds"
-}
-
 main() {
-    local check_budget check_duration operation
+    local check_budget check_duration check_deadline operation
     local index target_count=1 any_failed=false completed_indices indexed_var_name
     # Preserve the existing SLEEP_TIME setting, including fractional durations.
-    check_budget=$(duration_seconds "${SLEEP_TIME:-1h}") || return 1
+    check_budget=$(/src/interval.sh budget "${SLEEP_TIME:-1h}") || return 1
 
     # Finish every backup before spending the former sleep interval checking.
     for operation in backup check; do
+        if [[ $operation == check ]]; then
+            check_deadline=$(/src/interval.sh deadline "${SLEEP_TIME:-1h}") || return 1
+        fi
         check_duration=$((check_budget / target_count))
         if (( check_duration == 0 )); then
             echo "SLEEP_TIME must allow at least one second per repository."
@@ -162,7 +122,11 @@ main() {
     # Clear the failure only after publishing a complete, successful timestamp.
     date -u '+%Y-%m-%dT%H:%M:%SZ' > /health/backup_completion_time.log.$$ &&
         mv -fT /health/backup_completion_time.log.$$ /health/backup_completion_time.log &&
-        rm -f /health/backup_failed
+        rm -f /health/backup_failed || return 1
+
+    # A partial check can finish early. Keep the configured interval between
+    # backup rounds, counting checks and lock waits toward that interval.
+    /src/interval.sh wait "$check_deadline"
 }
 
 main

src/backup.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-set -e
+set -eo pipefail
 
 : "${ARCHIVE_PREFIX:?Set ARCHIVE_PREFIX to a non-empty archive prefix (e.g. my-host-)}"
 
@@ -13,6 +13,8 @@ echo "Starting backup script at $(date)"
 
 # shellcheck source=src/borg-common.sh
 source /src/borg-common.sh
+archive_prefix=$(borg_literal_prefix archive "$ARCHIVE_PREFIX")
+archive_glob=$(borg_literal_prefix glob "$ARCHIVE_PREFIX")
 
 # Only a missing repository (modern exit code 13) triggers initialization.
 info_status=0
@@ -37,60 +39,35 @@ case "$info_status" in
 esac
 
 cleanup() {
-    if [ -n "${GIT_EXCLUDE_FILE:-}" ] && [ -f "$GIT_EXCLUDE_FILE" ]; then
-        rm -f "$GIT_EXCLUDE_FILE"
-    fi
+    local status=$?
     if [ -d "/snapshot/btrfs-root" ]; then
         cd /
-        btrfs subvolume delete /snapshot/btrfs-root || true
+        /src/snapshot.sh delete /snapshot/btrfs-root || status=1
     fi
+    exit "$status"
 }
 trap cleanup EXIT
 
 if [ -d "/snapshot/btrfs-root" ]; then
-    btrfs subvolume delete /snapshot/btrfs-root
+    /src/snapshot.sh delete /snapshot/btrfs-root
 fi
 
-btrfs subvolume snapshot /btrfs-root /snapshot
+/src/snapshot.sh create /btrfs-root /snapshot/btrfs-root
 
 cd "/snapshot/btrfs-root${BACKUP_RELATIVE_PATH:-}"
 
-# Generate exclusions for git-untracked files if enabled
-EXCLUDE_ARGS=(--exclude-from /exclude.conf)
-if [ "${IGNORE_GIT_UNTRACKED:-false}" = "true" ]; then
-    echo "Generating exclusions for git-untracked files..."
-    GIT_EXCLUDE_FILE=$(mktemp)
-
-    # Find all git repositories and list their untracked files
-    find . -name .git -type d | while read -r gitdir; do
-        repo_dir=$(dirname "$gitdir")
-        (
-            cd "$repo_dir"
-            # Get untracked files (respecting .gitignore)
-            git ls-files --others --exclude-standard | while read -r file; do
-                # Output path relative to backup root
-                echo "${repo_dir#./}/$file"
-            done
-        )
-    done > "$GIT_EXCLUDE_FILE"
-
-    excluded_count=$(wc -l < "$GIT_EXCLUDE_FILE")
-    echo "Found $excluded_count git-untracked files to exclude"
-
-    EXCLUDE_ARGS+=(--exclude-from "$GIT_EXCLUDE_FILE")
-fi
-
 borg_wait create --stats \
     --list \
     --filter=AMCE \
     --files-cache=ctime,size,inode \
     --compression=zstd,12 \
-    "${EXCLUDE_ARGS[@]}" ::"${ARCHIVE_PREFIX}{now:%Y-%m-%dT%H:%M:%S}" .
+    --exclude-from /exclude.conf \
+    ::"${archive_prefix}{now:%Y-%m-%dT%H:%M:%S}" .
 
 cd -
 
 borg_wait prune --list --stats \
-    --glob-archives="${ARCHIVE_PREFIX}*" \
+    --glob-archives="${archive_glob}*" \
     --keep-daily="$KEEP_DAILY" \
     --keep-weekly="$KEEP_WEEKLY" \
     --keep-monthly="$KEEP_MONTHLY" \

src/borg-common.sh

@@ -2,6 +2,27 @@
 
 export BORG_RSH='ssh -oBatchMode=yes'
 
+# Borg expands placeholders in both names and archive filters. Only the filter
+# needs glob escaping; user-supplied prefixes are literal in both places.
+borg_literal_prefix() {
+    local mode=$1 prefix=$2 character index
+    for ((index = 0; index < ${#prefix}; index++)); do
+        character=${prefix:index:1}
+        case "$character" in
+            '{') printf '{{' ;;
+            '}') printf '}}' ;;
+            '['|'?'|'*')
+                if [[ $mode == glob ]]; then
+                    printf '[%s]' "$character"
+                else
+                    printf '%s' "$character"
+                fi
+                ;;
+            *) printf '%s' "$character" ;;
+        esac
+    done
+}
+
 # Never break another client's repository/cache lock. Borg 1.4 only accepts
 # finite waits, so retry lock timeouts forever; other errors must propagate.
 borg_wait() {

src/interval.sh

@@ -0,0 +1,88 @@
+#!/bin/bash
+
+# Return whole seconds and a fractional part without floating-point rounding.
+# Long multiplication keeps the range checks exact even for tiny fractions.
+interval_duration() {
+    local value=$1 whole fraction multiplier carry=0 product='' digit index
+    if [[ ! $value =~ ^([0-9]+([.][0-9]*)?|[.][0-9]+)([smhd]?)$ ]]; then
+        echo 'SLEEP_TIME must be a positive duration in seconds, minutes, hours or days.' >&2
+        return 1
+    fi
+    case ${BASH_REMATCH[3]} in
+        ''|s) multiplier=1 ;;
+        m) multiplier=60 ;;
+        h) multiplier=3600 ;;
+        d) multiplier=86400 ;;
+    esac
+    value=${BASH_REMATCH[1]}
+    whole=${value%%.*}
+    whole=${whole#"${whole%%[!0]*}"}
+    whole=${whole:-0}
+    if (( ${#whole} > 10 )); then
+        echo 'SLEEP_TIME must represent between 1 and 2147483647 seconds.' >&2
+        return 1
+    fi
+    fraction=''
+    [[ $value != *.* ]] || fraction=${value#*.}
+    for ((index = ${#fraction} - 1; index >= 0; index--)); do
+        digit=$((10#${fraction:index:1} * multiplier + carry))
+        product="$((digit % 10))$product"
+        carry=$((digit / 10))
+    done
+    whole=$((10#$whole * multiplier + carry))
+    product=${product%"${product##*[!0]}"}
+    if (( whole < 1 || whole > 2147483647 )) || { (( whole == 2147483647 )) && [[ -n $product ]]; }; then
+        echo 'SLEEP_TIME must represent between 1 and 2147483647 seconds.' >&2
+        return 1
+    fi
+    printf '%s %s\n' "$whole" "${product:-0}"
+}
+
+# /proc/uptime advances independently of wall-clock adjustments, in 10 ms ticks.
+interval_now() {
+    local uptime _idle whole fraction
+    read -r uptime _idle < /proc/uptime || return 1
+    whole=${uptime%.*}
+    fraction=${uptime#*.}00
+    printf '%s\n' "$((10#$whole * 100 + 10#${fraction:0:2}))"
+}
+
+interval_budget() {
+    local duration
+    duration=$(interval_duration "$1") || return 1
+    printf '%s\n' "${duration%% *}"
+}
+
+interval_deadline() {
+    local duration whole fraction ticks now
+    duration=$(interval_duration "$1") || return 1
+    read -r whole fraction <<< "$duration"
+    fraction+='00'
+    ticks=$((whole * 100 + 10#${fraction:0:2}))
+    # Round up fractions and the current clock tick to preserve the minimum wait.
+    [[ ${fraction:2} != *[1-9]* ]] || ticks=$((ticks + 1))
+    now=$(interval_now) || return 1
+    printf '%s\n' "$((now + ticks + 1))"
+}
+
+interval_wait() {
+    local deadline=$1 now remaining delay
+    [[ $deadline =~ ^[0-9]{1,15}$ ]] || return 1
+    while true; do
+        now=$(interval_now) || return 1
+        remaining=$((10#$deadline - now))
+        (( remaining > 0 )) || return 0
+        printf -v delay '%d.%02d' "$((remaining / 100))" "$((remaining % 100))"
+        sleep "$delay" || return 1
+    done
+}
+
+if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
+    case ${1:-} in
+        budget|deadline|wait)
+            [[ $# == 2 ]] || exit 1
+            "interval_$1" "$2"
+            ;;
+        *) echo 'Expected budget, deadline or wait and a value.' >&2; exit 1 ;;
+    esac
+fi

src/schedule.sh

@@ -18,7 +18,7 @@ date -u '+%Y-%m-%dT%H:%M:%SZ' > /health/container_start_time.log.$$ &&
     mv -fT /health/container_start_time.log.$$ /health/container_start_time.log || exit 1
 
 while true; do
-    # The wrapper checks every repository after backups, using SLEEP_TIME as
-    # its total check budget. Start the next cycle as soon as checks finish.
+    # The wrapper checks every repository after backups, then sleeps for any
+    # unused part of SLEEP_TIME before allowing the next backup round.
     /src/backup-wrapper.sh 2>&1 | log_message
 done

src/snapshot.sh

@@ -0,0 +1,112 @@
+#!/bin/bash
+
+# Read mountinfo from stdin and fill the caller's associative array. Kernel
+# escapes protect whitespace and backslashes in the mount-point field.
+snapshot_mounted_subvolumes() {
+    local source=${1%/} mount remainder _ignored
+    local -n discovered=$2
+    while IFS=' ' read -r _ignored _ignored _ignored _ignored mount remainder; do
+        [[ $remainder == *' - btrfs '* ]] || continue
+        printf -v mount '%b' "$mount"
+        if [[ $mount == "$source/"* && $mount != "$source/" ]]; then
+            # The nameref fills the caller's array.
+            # shellcheck disable=SC2034
+            discovered["${mount#"$source/"}"]=1
+        fi
+    done
+}
+
+snapshot_create() (
+    local source target parent name directory child relative inode original
+    local -a pending
+    local -A mounts=()
+    unset GLOBIGNORE
+    shopt -s dotglob nullglob
+    IFS= read -r -d '' source < <(realpath -ez -- "$1") || return 1
+    target=$2
+    while [[ $target == */ && $target != / ]]; do target=${target%/}; done
+    name=${target##*/}
+    parent=${target%/*}
+    [[ $target == */* ]] || parent=.
+    IFS= read -r -d '' parent < <(realpath -ez -- "${parent:-/}") || return 1
+    target=${parent%/}/$name
+    if [[ -e $target || -L $target ]]; then
+        printf 'Snapshot destination already exists: %s\n' "$target" >&2
+        return 1
+    fi
+    snapshot_mounted_subvolumes "$source" mounts < /proc/self/mountinfo || return 1
+    btrfs subvolume snapshot "$source" "$target" || return 1
+    pending=("$target")
+    while (( ${#pending[@]} )); do
+        directory=${pending[-1]}
+        unset 'pending[-1]'
+        if [[ ! -r $directory || ! -x $directory ]]; then
+            printf 'Cannot traverse snapshot directory: %s\n' "$directory" >&2
+            return 1
+        fi
+        for child in "$directory"/*; do
+            [[ -d $child && ! -L $child ]] || continue
+            relative=${child#"$target/"}
+            inode=$(stat -c %i -- "$child") || return 1
+            # Unsnapshotted children have inode 2. A mounted subvolume may
+            # instead hide an ordinary directory in its parent's snapshot.
+            if [[ $inode == 2 || ${mounts[$relative]:-} == 1 ]]; then
+                original=${source%/}/$relative
+                if [[ -L $original ]]; then
+                    printf 'Subvolume changed into a symlink: %s\n' "$original" >&2
+                    return 1
+                fi
+                # Never recurse into the scratch snapshot, even below a child.
+                if [[ $original -ef $target ]]; then
+                    rmdir -- "$child" || return 1
+                    continue
+                fi
+                # Refuse to discard data hidden beneath a mounted subvolume.
+                rmdir -- "$child" || return 1
+                btrfs subvolume snapshot "$original" "$child" || return 1
+            fi
+            pending+=("$child")
+        done
+    done
+)
+
+snapshot_delete() (
+    local target=$1 directory child inode index
+    local -a pending subvolumes
+    unset GLOBIGNORE
+    shopt -s dotglob nullglob
+    while [[ $target == */ && $target != / ]]; do target=${target%/}; done
+    if [[ -L $target ]]; then
+        printf 'Refusing to delete a symlink as a snapshot: %s\n' "$target" >&2
+        return 1
+    fi
+    btrfs subvolume show "$target" > /dev/null || return 1
+    pending=("$target")
+    subvolumes=("$target")
+    while (( ${#pending[@]} )); do
+        directory=${pending[-1]}
+        unset 'pending[-1]'
+        if [[ ! -r $directory || ! -x $directory ]]; then
+            printf 'Cannot traverse snapshot directory: %s\n' "$directory" >&2
+            return 1
+        fi
+        for child in "$directory"/*; do
+            [[ -d $child && ! -L $child ]] || continue
+            inode=$(stat -c %i -- "$child") || return 1
+            [[ $inode != 256 ]] || subvolumes+=("$child")
+            pending+=("$child")
+        done
+    done
+    # Reverse discovery order puts every child before its parent.
+    for ((index = ${#subvolumes[@]} - 1; index >= 0; index--)); do
+        btrfs subvolume delete "${subvolumes[index]}" || return 1
+    done
+)
+
+if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
+    case ${1:-} in
+        create) [[ $# == 3 ]] && snapshot_create "$2" "$3" ;;
+        delete) [[ $# == 2 ]] && snapshot_delete "$2" ;;
+        *) echo 'Expected create SOURCE TARGET or delete TARGET.' >&2; exit 1 ;;
+    esac
+fi