From 42a31b6a4496203fb52d70ebbf89a3e07dbbbe21 Mon Sep 17 00:00:00 2001 From: poprhythm Date: Tue, 25 Aug 2026 21:56:48 +0000 Subject: [PATCH] Add NFS self-heal automation for immich/calibre/ocis, backed up in git Stale NFS handles inside containers (ESTALE/errno -116) can persist even after the host-side mount looks healthy, requiring a manual remount + container restart. Adds systemd mount-ready hooks (BindsTo) to restart immich_server/calibre/ocis when their NFS mount unit restarts, plus a 5-minute health-check timer that does a real nested read and force-remounts on staleness. Host files backed up under system-config/nfs-self-heal/ so they can be reinstalled after a host rebuild. --- NAS-CONNECTION-STRATEGIES.md | 20 +++++++ system-config/nfs-self-heal/README.md | 55 +++++++++++++++++++ .../nfs-self-heal/calibre-mount-ready.service | 13 +++++ .../nfs-self-heal/immich-mount-ready.service | 13 +++++ .../nfs-self-heal/nfs-mount-heal.service | 6 ++ system-config/nfs-self-heal/nfs-mount-heal.sh | 55 +++++++++++++++++++ .../nfs-self-heal/nfs-mount-heal.timer | 9 +++ .../nfs-self-heal/ocis-mount-ready.service | 13 +++++ 8 files changed, 184 insertions(+) create mode 100644 system-config/nfs-self-heal/README.md create mode 100644 system-config/nfs-self-heal/calibre-mount-ready.service create mode 100644 system-config/nfs-self-heal/immich-mount-ready.service create mode 100644 system-config/nfs-self-heal/nfs-mount-heal.service create mode 100644 system-config/nfs-self-heal/nfs-mount-heal.sh create mode 100644 system-config/nfs-self-heal/nfs-mount-heal.timer create mode 100644 system-config/nfs-self-heal/ocis-mount-ready.service diff --git a/NAS-CONNECTION-STRATEGIES.md b/NAS-CONNECTION-STRATEGIES.md index 02e6af7..035ec7b 100644 --- a/NAS-CONNECTION-STRATEGIES.md +++ b/NAS-CONNECTION-STRATEGIES.md @@ -115,6 +115,26 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168. --- +## Self-Healing (nas_family / nas_books / nas_owncloud) + +As of 2026-08-25, stale-handle recovery for immich, calibre, and oCIS is +automated instead of requiring manual `systemctl restart` + `docker restart`: + +- `-mount-ready.service` (`immich-mount-ready`, `calibre-mount-ready`, + `ocis-mount-ready`) — `BindsTo=mnt-nas_*.mount`, restarts the container + whenever its mount unit restarts. +- `nfs-mount-heal.timer` (every 5 min) → `nfs-mount-heal.sh` — does a nested + file read (not just `ls` the mount root, which can succeed even when nested + handles are stale) against each of the three mounts. On failure it force- + remounts the mount unit, which cascades into the container restart above, + then sends a Telegram alert. + +Host files backed up at `system-config/nfs-self-heal/` in this repo (see its +README for the reinstall procedure — these are systemd units, not +docker-compose, so they don't redeploy via Portainer). + +--- + ## Recommended Migration (future) Plex, qBittorrent, Immich, and Audiobookshelf all use the fragile Docker NFS named volume pattern. The safer pattern is systemd permanent mount + Docker bind mount (as used by calibre-import and oCIS). This would involve: diff --git a/system-config/nfs-self-heal/README.md b/system-config/nfs-self-heal/README.md new file mode 100644 index 0000000..3f30c3a --- /dev/null +++ b/system-config/nfs-self-heal/README.md @@ -0,0 +1,55 @@ +# NFS Self-Heal (host-level systemd units) + +These files are **not** deployed by Docker/Portainer — they're systemd units and a +script that live directly on the host at `/etc/systemd/system/` and +`/usr/local/bin/`. They're checked in here purely as a backup so they can be +restored if the host is rebuilt. Editing a file here does **not** change +running behavior; you have to re-copy it to the host and reload systemd. + +## What this does + +Permanent NFS mounts (`mnt-nas_family.mount`, `mnt-nas_books.mount`, +`mnt-nas_owncloud.mount`) can go stale — the host mount looks fine on a +top-level `ls`, but file handles nested inside it return ESTALE +(errno -116) to processes that opened them earlier, including containers +that bind-mount the path. A container restart is required to pick up a +fresh handle even after the host-side mount is healthy again. See +`../../NAS-CONNECTION-STRATEGIES.md` for the full mount inventory and +`../../immich/` / `../../calibre/MOUNT-HISTORY.md` for history on this +failure mode (first hit immich on 2026-08-25; calibre-import had the same +issue documented earlier but unresolved). + +Two pieces: + +1. **`-mount-ready.service`** (one per affected container) — + `BindsTo=mnt-nas_*.mount`, so restarting the mount unit automatically + restarts the bound container. Currently covers `immich_server`, + `calibre`, `ocis`. (Older instances of this same pattern — + `backrest`, `kiwix`, `pmtiles`, `audiobookshelf`, `romm` — already + exist on the host but aren't backed up here yet.) +2. **`nfs-mount-heal.timer`** → **`nfs-mount-heal.service`** → + **`nfs-mount-heal.sh`** — runs every 5 minutes, does a real nested + read (not just `ls` the mount root) against `nas_family`, `nas_books`, + and `nas_owncloud`. On failure it force-remounts the mount unit + (`systemctl restart`), which cascades into the container restart via + the hooks above, then re-checks and sends a Telegram alert either way. + +## Reinstall after a host rebuild + +```bash +sudo cp *.service *.timer /etc/systemd/system/ +sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh +sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh + +sudo systemctl daemon-reload +sudo systemctl enable --now immich-mount-ready.service +sudo systemctl enable --now calibre-mount-ready.service +sudo systemctl enable --now ocis-mount-ready.service +sudo systemctl enable --now nfs-mount-heal.timer +``` + +Requires the `mnt-nas_family.mount`, `mnt-nas_books.mount`, and +`mnt-nas_owncloud.mount` systemd units to already exist (see +`NAS-CONNECTION-STRATEGIES.md`) and `.credentials` at the repo root to be +present on the host (the heal script sources it for the Telegram bot +token/chat ID). diff --git a/system-config/nfs-self-heal/calibre-mount-ready.service b/system-config/nfs-self-heal/calibre-mount-ready.service new file mode 100644 index 0000000..f960318 --- /dev/null +++ b/system-config/nfs-self-heal/calibre-mount-ready.service @@ -0,0 +1,13 @@ +[Unit] +Description=Restart calibre after NAS books mount is ready +After=mnt-nas_books.mount +Requires=mnt-nas_books.mount +BindsTo=mnt-nas_books.mount + +[Service] +Type=oneshot +ExecStart=/usr/bin/docker restart calibre +RemainAfterExit=yes + +[Install] +WantedBy=mnt-nas_books.mount diff --git a/system-config/nfs-self-heal/immich-mount-ready.service b/system-config/nfs-self-heal/immich-mount-ready.service new file mode 100644 index 0000000..062f21b --- /dev/null +++ b/system-config/nfs-self-heal/immich-mount-ready.service @@ -0,0 +1,13 @@ +[Unit] +Description=Restart immich_server after NAS family mount is ready +After=mnt-nas_family.mount +Requires=mnt-nas_family.mount +BindsTo=mnt-nas_family.mount + +[Service] +Type=oneshot +ExecStart=/usr/bin/docker restart immich_server +RemainAfterExit=yes + +[Install] +WantedBy=mnt-nas_family.mount diff --git a/system-config/nfs-self-heal/nfs-mount-heal.service b/system-config/nfs-self-heal/nfs-mount-heal.service new file mode 100644 index 0000000..4ca41bc --- /dev/null +++ b/system-config/nfs-self-heal/nfs-mount-heal.service @@ -0,0 +1,6 @@ +[Unit] +Description=Deep health check and self-heal for permanent NFS mounts (immich/calibre/ocis) + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/nfs-mount-heal.sh diff --git a/system-config/nfs-self-heal/nfs-mount-heal.sh b/system-config/nfs-self-heal/nfs-mount-heal.sh new file mode 100644 index 0000000..0dd6de3 --- /dev/null +++ b/system-config/nfs-self-heal/nfs-mount-heal.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# Deep health check + self-heal for permanent NFS mounts backing +# immich, calibre, and ocis. +# +# A plain `ls` on a mount root can succeed even when nested file +# handles inside it are stale (ESTALE / errno -116) -- that's the +# failure mode that broke immich on 2026-08-25 and is documented as +# a recurring issue for calibre-import in calibre/MOUNT-HISTORY.md. +# This does an actual nested read; on failure it forces a full +# remount of the NFS mount unit, which triggers the matching +# -mount-ready.service (BindsTo) to restart the container +# automatically. + +set -uo pipefail + +source /home/poprhythm/docker-infrastructure/.credentials + +# mount:unit:testdir:container +CHECKS=( + "/mnt/nas_family:mnt-nas_family.mount:/mnt/nas_family/immich-library/thumbs:immich_server" + "/mnt/nas_books:mnt-nas_books.mount:/mnt/nas_books/calibre-import:calibre" + "/mnt/nas_owncloud:mnt-nas_owncloud.mount:/mnt/nas_owncloud:ocis" +) + +alert() { + curl -fsS -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \ + -d chat_id="${TELEGRAM_CHAT_ID}" -d text="$1" > /dev/null 2>&1 +} + +healthy() { + local testdir="$1" + local f + f=$(timeout 10 find "$testdir" -maxdepth 3 -type f -print -quit 2>/dev/null) || return 1 + [ -n "$f" ] || return 1 + timeout 10 cat "$f" > /dev/null 2>&1 +} + +for entry in "${CHECKS[@]}"; do + IFS=':' read -r mount unit testdir container <<< "$entry" + + healthy "$testdir" && continue + + logger -t nfs-mount-heal "${mount} unhealthy (stale/unreachable), remounting ${unit}" + + if timeout 30 systemctl restart "$unit"; then + sleep 3 + if healthy "$testdir"; then + alert "🔧 NFS self-heal: ${mount} went stale, remounted ${unit} and restarted ${container} automatically. All clear." + else + alert "⚠️ NFS self-heal: ${mount} was stale, remounted ${unit} and restarted ${container}, but it's still failing the health check. Needs a look." + fi + else + alert "❗ NFS self-heal FAILED: ${mount} is unhealthy and systemctl restart ${unit} failed. Manual intervention needed." + fi +done diff --git a/system-config/nfs-self-heal/nfs-mount-heal.timer b/system-config/nfs-self-heal/nfs-mount-heal.timer new file mode 100644 index 0000000..77a7f33 --- /dev/null +++ b/system-config/nfs-self-heal/nfs-mount-heal.timer @@ -0,0 +1,9 @@ +[Unit] +Description=Run NFS mount self-heal check every 5 minutes + +[Timer] +OnBootSec=5min +OnUnitActiveSec=5min + +[Install] +WantedBy=timers.target diff --git a/system-config/nfs-self-heal/ocis-mount-ready.service b/system-config/nfs-self-heal/ocis-mount-ready.service new file mode 100644 index 0000000..737a826 --- /dev/null +++ b/system-config/nfs-self-heal/ocis-mount-ready.service @@ -0,0 +1,13 @@ +[Unit] +Description=Restart ocis after NAS owncloud mount is ready +After=mnt-nas_owncloud.mount +Requires=mnt-nas_owncloud.mount +BindsTo=mnt-nas_owncloud.mount + +[Service] +Type=oneshot +ExecStart=/usr/bin/docker restart ocis +RemainAfterExit=yes + +[Install] +WantedBy=mnt-nas_owncloud.mount