fix/offsite bounded timeout - #384
Merged
Merged
Conversation
swares
commented
Aug 11, 2026
swares
commented
Owner
- docs: record the four-night backup-nas outage caused by the offsite seed
- backup-offsite: bound TimeoutStartSec; reject cold-sec as a bulk copy source
backup-nas failed outright on 08, 09, 10 and 11 Aug -- not the copy step, the
whole unit. Four consecutive nights with no new backup of /srv/nas,
/mnt/cold-8t/VMs or /mnt/cold-8t/immich to any repository. Recovered by hand on
2026-08-11 03:46.
unable to create lock in backend: repository is already locked by PID 1693818
That PID was the offsite seed. restic -r <primary> copy --repo2 <offsite> holds a
lock on the SOURCE repo for the whole operation, and the seed ran for three days,
so the nightly restic backup could never acquire its own lock.
The seed was given TimeoutStartSec=infinity because it was expected to be slow.
What was not considered is that a multi-day operation against the primary starves
the nightly job it exists to protect. Nightly incrementals take about ten seconds,
so this only appears on bulk work -- and then it appears for as long as the bulk
work runs.
Fix recorded for any future bulk operation: source it from cold-sec rather than
the primary. Same data, deeper retention, nothing else contends for it, and the
01:30 window stays clear. Applies to a re-seed, check --read-data, or a migration.
The alerting worked. Both paths fired and reached the phone -- healthchecks.io
went down because hc-ping is ExecStartPost and is skipped on failure, and
LabBackupUnitFailed fired on node_systemd_unit_state{state="failed"}.
Investigation was deliberately deferred until the seed finished rather than
interrupting it.
That is the first real-world validation of the 08-07 alerting work: a genuine
multi-night backup outage, detected, delivered and acted on. The three failures
before it went unnoticed for days each.
… source TimeoutStartSec was infinity for the initial ~204 GiB seed, which ran three days. restic copy holds a lock on the SOURCE repo for its whole duration, so backup-nas could not acquire its own lock and failed outright on 08, 09, 10 and 11 Aug -- four nights with no backup of /srv/nas, the VMs or the immich library to any repository. Now bounded to 4h. A nightly incremental takes about ten seconds, so that is enormous headroom, and it ends at 06:30, clear of the next backup-nas at 01:30. The distinction that matters: an unbounded copy starves the nightly job silently for days, while a bounded one fails -- and a failed unit skips hc-ping and trips LabBackupUnitFailed. Both fired for this outage and reached the phone. Sourcing future bulk copies from cold-sec was proposed as the fix and is rejected on measurement: /mnt/cold-8t/restic chunker_polynomial 30f6553d487a79 R2 homelab-nas chunker_polynomial 30f6553d487a79 /mnt/cold-sec/restic chunker_polynomial 2d710aa7618093 backup-nas-copy.sh initialised the secondary plainly, without --copy-chunker-params, and says so in its own comments. Copying cold-sec to the offsite repo would therefore re-chunk all 204 GiB: no dedup against what is already there, a full re-upload, and an offsite repo carrying two chunkings of the same data. Records the re-seed procedure instead -- raise the timeout deliberately and stop backup-nas.timer for the duration -- and notes re-initialising cold-sec with matching chunker params as a later option, explicitly gated behind testing a restore first. Rebuilding redundancy you have never verified you can restore from is the wrong order. Also fills in the break-glass envelope contents in BACKLOG, with two corrections: the restic password was rotated on 08-07 so any existing copy is stale, and item 4 still lists a Vault root token that deliberately no longer exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.