Add NFS self-heal automation for immich/calibre/ocis, backed up in git

Stale NFS handles inside containers (ESTALE/errno -116) can persist even after the host-side mount looks healthy, requiring a manual remount + container restart. Adds systemd mount-ready hooks (BindsTo) to restart immich_server/calibre/ocis when their NFS mount unit restarts, plus a 5-minute health-check timer that does a real nested read and force-remounts on staleness. Host files backed up under system-config/nfs-self-heal/ so they can be reinstalled after a host rebuild.
2026-08-25 21:56:48 +00:00
parent d302060e4b
commit 42a31b6a44
8 changed files with 184 additions and 0 deletions
@@ -115,6 +115,26 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168.

 ---

+## Self-Healing (nas_family / nas_books / nas_owncloud)
+
+As of 2026-08-25, stale-handle recovery for immich, calibre, and oCIS is
+automated instead of requiring manual `systemctl restart` + `docker restart`:
+
+- `<container>-mount-ready.service` (`immich-mount-ready`, `calibre-mount-ready`,
+  `ocis-mount-ready`) — `BindsTo=mnt-nas_*.mount`, restarts the container
+  whenever its mount unit restarts.
+- `nfs-mount-heal.timer` (every 5 min) → `nfs-mount-heal.sh` — does a nested
+  file read (not just `ls` the mount root, which can succeed even when nested
+  handles are stale) against each of the three mounts. On failure it force-
+  remounts the mount unit, which cascades into the container restart above,
+  then sends a Telegram alert.
+
+Host files backed up at `system-config/nfs-self-heal/` in this repo (see its
+README for the reinstall procedure — these are systemd units, not
+docker-compose, so they don't redeploy via Portainer).
+
+---
+
 ## Recommended Migration (future)

 Plex, qBittorrent, Immich, and Audiobookshelf all use the fragile Docker NFS named volume pattern. The safer pattern is systemd permanent mount + Docker bind mount (as used by calibre-import and oCIS). This would involve:
@@ -0,0 +1,55 @@
+# NFS Self-Heal (host-level systemd units)
+
+These files are **not** deployed by Docker/Portainer — they're systemd units and a
+script that live directly on the host at `/etc/systemd/system/` and
+`/usr/local/bin/`. They're checked in here purely as a backup so they can be
+restored if the host is rebuilt. Editing a file here does **not** change
+running behavior; you have to re-copy it to the host and reload systemd.
+
+## What this does
+
+Permanent NFS mounts (`mnt-nas_family.mount`, `mnt-nas_books.mount`,
+`mnt-nas_owncloud.mount`) can go stale — the host mount looks fine on a
+top-level `ls`, but file handles nested inside it return ESTALE
+(errno -116) to processes that opened them earlier, including containers
+that bind-mount the path. A container restart is required to pick up a
+fresh handle even after the host-side mount is healthy again. See
+`../../NAS-CONNECTION-STRATEGIES.md` for the full mount inventory and
+`../../immich/` / `../../calibre/MOUNT-HISTORY.md` for history on this
+failure mode (first hit immich on 2026-08-25; calibre-import had the same
+issue documented earlier but unresolved).
+
+Two pieces:
+
+1. **`<container>-mount-ready.service`** (one per affected container) —
+   `BindsTo=mnt-nas_*.mount`, so restarting the mount unit automatically
+   restarts the bound container. Currently covers `immich_server`,
+   `calibre`, `ocis`. (Older instances of this same pattern —
+   `backrest`, `kiwix`, `pmtiles`, `audiobookshelf`, `romm` — already
+   exist on the host but aren't backed up here yet.)
+2. **`nfs-mount-heal.timer`** → **`nfs-mount-heal.service`** →
+   **`nfs-mount-heal.sh`** — runs every 5 minutes, does a real nested
+   read (not just `ls` the mount root) against `nas_family`, `nas_books`,
+   and `nas_owncloud`. On failure it force-remounts the mount unit
+   (`systemctl restart`), which cascades into the container restart via
+   the hooks above, then re-checks and sends a Telegram alert either way.
+
+## Reinstall after a host rebuild
+
+```bash
+sudo cp *.service *.timer /etc/systemd/system/
+sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
+sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh
+
+sudo systemctl daemon-reload
+sudo systemctl enable --now immich-mount-ready.service
+sudo systemctl enable --now calibre-mount-ready.service
+sudo systemctl enable --now ocis-mount-ready.service
+sudo systemctl enable --now nfs-mount-heal.timer
+```
+
+Requires the `mnt-nas_family.mount`, `mnt-nas_books.mount`, and
+`mnt-nas_owncloud.mount` systemd units to already exist (see
+`NAS-CONNECTION-STRATEGIES.md`) and `.credentials` at the repo root to be
+present on the host (the heal script sources it for the Telegram bot
+token/chat ID).
@@ -0,0 +1,13 @@
+[Unit]
+Description=Restart calibre after NAS books mount is ready
+After=mnt-nas_books.mount
+Requires=mnt-nas_books.mount
+BindsTo=mnt-nas_books.mount
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/docker restart calibre
+RemainAfterExit=yes
+
+[Install]
+WantedBy=mnt-nas_books.mount
@@ -0,0 +1,13 @@
+[Unit]
+Description=Restart immich_server after NAS family mount is ready
+After=mnt-nas_family.mount
+Requires=mnt-nas_family.mount
+BindsTo=mnt-nas_family.mount
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/docker restart immich_server
+RemainAfterExit=yes
+
+[Install]
+WantedBy=mnt-nas_family.mount
@@ -0,0 +1,6 @@
+[Unit]
+Description=Deep health check and self-heal for permanent NFS mounts (immich/calibre/ocis)
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/nfs-mount-heal.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+# Deep health check + self-heal for permanent NFS mounts backing
+# immich, calibre, and ocis.
+#
+# A plain `ls` on a mount root can succeed even when nested file
+# handles inside it are stale (ESTALE / errno -116) -- that's the
+# failure mode that broke immich on 2026-08-25 and is documented as
+# a recurring issue for calibre-import in calibre/MOUNT-HISTORY.md.
+# This does an actual nested read; on failure it forces a full
+# remount of the NFS mount unit, which triggers the matching
+# <container>-mount-ready.service (BindsTo) to restart the container
+# automatically.
+
+set -uo pipefail
+
+source /home/poprhythm/docker-infrastructure/.credentials
+
+# mount:unit:testdir:container
+CHECKS=(
+  "/mnt/nas_family:mnt-nas_family.mount:/mnt/nas_family/immich-library/thumbs:immich_server"
+  "/mnt/nas_books:mnt-nas_books.mount:/mnt/nas_books/calibre-import:calibre"
+  "/mnt/nas_owncloud:mnt-nas_owncloud.mount:/mnt/nas_owncloud:ocis"
+)
+
+alert() {
+  curl -fsS -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
+    -d chat_id="${TELEGRAM_CHAT_ID}" -d text="$1" > /dev/null 2>&1
+}
+
+healthy() {
+  local testdir="$1"
+  local f
+  f=$(timeout 10 find "$testdir" -maxdepth 3 -type f -print -quit 2>/dev/null) || return 1
+  [ -n "$f" ] || return 1
+  timeout 10 cat "$f" > /dev/null 2>&1
+}
+
+for entry in "${CHECKS[@]}"; do
+  IFS=':' read -r mount unit testdir container <<< "$entry"
+
+  healthy "$testdir" && continue
+
+  logger -t nfs-mount-heal "${mount} unhealthy (stale/unreachable), remounting ${unit}"
+
+  if timeout 30 systemctl restart "$unit"; then
+    sleep 3
+    if healthy "$testdir"; then
+      alert "🔧 NFS self-heal: ${mount} went stale, remounted ${unit} and restarted ${container} automatically. All clear."
+    else
+      alert "⚠️ NFS self-heal: ${mount} was stale, remounted ${unit} and restarted ${container}, but it's still failing the health check. Needs a look."
+    fi
+  else
+    alert "❗ NFS self-heal FAILED: ${mount} is unhealthy and systemctl restart ${unit} failed. Manual intervention needed."
+  fi
+done
@@ -0,0 +1,9 @@
+[Unit]
+Description=Run NFS mount self-heal check every 5 minutes
+
+[Timer]
+OnBootSec=5min
+OnUnitActiveSec=5min
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,13 @@
+[Unit]
+Description=Restart ocis after NAS owncloud mount is ready
+After=mnt-nas_owncloud.mount
+Requires=mnt-nas_owncloud.mount
+BindsTo=mnt-nas_owncloud.mount
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/docker restart ocis
+RemainAfterExit=yes
+
+[Install]
+WantedBy=mnt-nas_owncloud.mount