fix/offsite snapshot count - #381
Merged
Merged
Conversation
swares
commented
Aug 11, 2026
swares
commented
Owner
- docs: record that lldap's restic mount is hard NFS, and correct a bad finding
- backup-offsite: count snapshots with grep -o, not grep -c
… finding An untracked gitops/workloads/lldap/restic-pv.yaml was spotted on the H4 and recorded in BACKLOG as a live PV that urgently needed committing. It was not. kubectl get pv lldap-restic returns NotFound: the file was an unapplied draft, and it has since vanished from the working tree. The claim came from reading its contents without checking whether the object existed -- the same mistake as trusting a green timer, in miniature. But the draft existed for a real reason, now recorded as 1.8. The lldap backup CronJob mounts the restic repo with an inline nfs: volume, and inline NFS volumes cannot carry mountOptions -- that is a PV-only field. So it mounts hard, and an unreachable NFS export blocks the pod indefinitely instead of erroring. That is the 2026-08-02 incident where lldap-backup hung for 3d15h. It was bounded, not fixed: activeDeadlineSeconds: 1800 kills the job after 30 minutes and LabBackupJobFailed pages when it does, so a backup can no longer stall silently for days. The stall itself remains. Fixing it means a PV plus PVC and switching the volume type, which is a change to a backup path that has run reliably since the deadline was added -- recorded as a decision, not scheduled.
The seed completed 2026-08-11 02:19 UTC -- ~204 GiB over ~3.1 days, retention applied on the first run keeping 32 snapshots across five groups. But the completion line read: offsite copy + retention complete -- 1 snapshots at destination restic snapshots --json emits the entire array on a single line, and grep -c counts matching LINES rather than matches. So the count was always going to be 1 for any non-empty repository. The assertion itself still held -- an empty array contains no "short_id", so a destination with zero snapshots is still caught, which is what the check exists for. But the number it printed was meaningless, and a number that looks like evidence while meaning nothing is worse than no number. Same family as the grep -q/pipefail bug in the immich dump job: a grep flag that answers a subtly different question than the one being asked, and only reveals itself at a scale or shape the author did not picture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.