diff --git a/system-config/nfs-self-heal/README.md b/system-config/nfs-self-heal/README.md index 3f30c3a..d161197 100644 --- a/system-config/nfs-self-heal/README.md +++ b/system-config/nfs-self-heal/README.md @@ -9,47 +9,64 @@ running behavior; you have to re-copy it to the host and reload systemd. ## What this does Permanent NFS mounts (`mnt-nas_family.mount`, `mnt-nas_books.mount`, -`mnt-nas_owncloud.mount`) can go stale — the host mount looks fine on a -top-level `ls`, but file handles nested inside it return ESTALE -(errno -116) to processes that opened them earlier, including containers -that bind-mount the path. A container restart is required to pick up a -fresh handle even after the host-side mount is healthy again. See +`mnt-nas_owncloud.mount`, `mnt-nas_audiobooks.mount`) can go stale — the host +mount looks fine on a top-level `ls`, but file handles nested inside it +return ESTALE (errno -116) to processes that opened them earlier, including +containers that bind-mount the path. A container restart is required to pick +up a fresh handle even after the host-side mount is healthy again. See `../../NAS-CONNECTION-STRATEGIES.md` for the full mount inventory and `../../immich/` / `../../calibre/MOUNT-HISTORY.md` for history on this failure mode (first hit immich on 2026-08-25; calibre-import had the same issue documented earlier but unresolved). +**All four `.mount` units must be *permanent* (enabled directly, no +`.automount` wrapper), not just present.** `mnt-nas_audiobooks.mount` was +mistakenly left on an `.automount` unit until 2026-09-17 and flapped ESTALE +for days as a result — autofs can cycle the underlying mount for reasons +other than idle timeout even with `TimeoutIdleSec=0`, which invalidates any +container's bind-mounted reference exactly like a stale handle. The self-heal +below papered over it (auto-remounting every ~5 min) but never fixed it; +switching to a permanent mount did. See NAS-CONNECTION-STRATEGIES.md's +"Docker bind mounts + automount = stale handles" entry. + Two pieces: 1. **`-mount-ready.service`** (one per affected container) — `BindsTo=mnt-nas_*.mount`, so restarting the mount unit automatically restarts the bound container. Currently covers `immich_server`, - `calibre`, `ocis`. (Older instances of this same pattern — - `backrest`, `kiwix`, `pmtiles`, `audiobookshelf`, `romm` — already - exist on the host but aren't backed up here yet.) + `calibre`, `ocis`, `audiobookshelf`. (Older instances of this same + pattern — `backrest`, `kiwix`, `pmtiles`, `romm` — already exist on + the host but aren't backed up here yet.) 2. **`nfs-mount-heal.timer`** → **`nfs-mount-heal.service`** → **`nfs-mount-heal.sh`** — runs every 5 minutes, does a real nested read (not just `ls` the mount root) against `nas_family`, `nas_books`, - and `nas_owncloud`. On failure it force-remounts the mount unit - (`systemctl restart`), which cascades into the container restart via - the hooks above, then re-checks and sends a Telegram alert either way. + `nas_owncloud`, and `nas_audiobooks`. On failure it force-remounts the + mount unit (`systemctl restart`), which cascades into the container + restart via the hooks above, then re-checks and sends a Telegram alert + either way. This is a safety net for genuine transient staleness, not a + fix for a mount that's structurally wrong (e.g. sitting on autofs). ## Reinstall after a host rebuild ```bash -sudo cp *.service *.timer /etc/systemd/system/ +sudo cp *.mount *.service *.timer /etc/systemd/system/ sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh sudo systemctl daemon-reload +sudo systemctl enable --now mnt-nas_family.mount +sudo systemctl enable --now mnt-nas_books.mount +sudo systemctl enable --now mnt-nas_owncloud.mount +sudo systemctl enable --now mnt-nas_audiobooks.mount sudo systemctl enable --now immich-mount-ready.service sudo systemctl enable --now calibre-mount-ready.service sudo systemctl enable --now ocis-mount-ready.service +sudo systemctl enable --now audiobookshelf-mount-ready.service sudo systemctl enable --now nfs-mount-heal.timer ``` -Requires the `mnt-nas_family.mount`, `mnt-nas_books.mount`, and -`mnt-nas_owncloud.mount` systemd units to already exist (see -`NAS-CONNECTION-STRATEGIES.md`) and `.credentials` at the repo root to be -present on the host (the heal script sources it for the Telegram bot -token/chat ID). +**Do not** create `.automount` units for any of these — they must be +enabled directly as permanent mounts (see the autofs warning above). + +Requires `.credentials` at the repo root to be present on the host (the +heal script sources it for the Telegram bot token/chat ID). diff --git a/system-config/nfs-self-heal/audiobookshelf-mount-ready.service b/system-config/nfs-self-heal/audiobookshelf-mount-ready.service new file mode 100644 index 0000000..bff109f --- /dev/null +++ b/system-config/nfs-self-heal/audiobookshelf-mount-ready.service @@ -0,0 +1,13 @@ +[Unit] +Description=Restart audiobookshelf after mnt-nas_audiobooks is ready +After=mnt-nas_audiobooks.mount +Requires=mnt-nas_audiobooks.mount +BindsTo=mnt-nas_audiobooks.mount + +[Service] +Type=oneshot +ExecStart=/usr/bin/docker restart audiobookshelf +RemainAfterExit=yes + +[Install] +WantedBy=mnt-nas_audiobooks.mount diff --git a/system-config/nfs-self-heal/mnt-nas_audiobooks.mount b/system-config/nfs-self-heal/mnt-nas_audiobooks.mount new file mode 100644 index 0000000..32fc4e5 --- /dev/null +++ b/system-config/nfs-self-heal/mnt-nas_audiobooks.mount @@ -0,0 +1,13 @@ +[Unit] +Description=Mount unRAID audiobooks NFS share +After=network-online.target +Wants=network-online.target + +[Mount] +What=192.168.1.192:/mnt/user/media/audiobooks +Where=/mnt/nas_audiobooks +Type=nfs +Options=nfsvers=3,hard,rw,noatime + +[Install] +WantedBy=multi-user.target diff --git a/system-config/nfs-self-heal/mnt-nas_books.mount b/system-config/nfs-self-heal/mnt-nas_books.mount new file mode 100644 index 0000000..335504e --- /dev/null +++ b/system-config/nfs-self-heal/mnt-nas_books.mount @@ -0,0 +1,13 @@ +[Unit] +Description=Mount unRAID books NFS share +After=network-online.target +Wants=network-online.target + +[Mount] +What=192.168.1.192:/mnt/user/media/books +Where=/mnt/nas_books +Type=nfs +Options=nfsvers=3,hard,rw,noatime + +[Install] +WantedBy=multi-user.target diff --git a/system-config/nfs-self-heal/mnt-nas_family.mount b/system-config/nfs-self-heal/mnt-nas_family.mount new file mode 100644 index 0000000..7f21eb5 --- /dev/null +++ b/system-config/nfs-self-heal/mnt-nas_family.mount @@ -0,0 +1,13 @@ +[Unit] +Description=Mount unRAID family NFS share +After=network-online.target +Wants=network-online.target + +[Mount] +What=192.168.1.192:/mnt/user/family +Where=/mnt/nas_family +Type=nfs +Options=nfsvers=3,hard,rw,noatime + +[Install] +WantedBy=multi-user.target diff --git a/system-config/nfs-self-heal/mnt-nas_owncloud.mount b/system-config/nfs-self-heal/mnt-nas_owncloud.mount new file mode 100644 index 0000000..99dfc0c --- /dev/null +++ b/system-config/nfs-self-heal/mnt-nas_owncloud.mount @@ -0,0 +1,13 @@ +[Unit] +Description=Mount unRAID owncloud NFS share +After=network-online.target +Wants=network-online.target + +[Mount] +What=192.168.1.192:/mnt/user/owncloud +Where=/mnt/nas_owncloud +Type=nfs +Options=nfsvers=3,hard,nolock,rw,noatime + +[Install] +WantedBy=multi-user.target diff --git a/system-config/nfs-self-heal/nfs-mount-heal.sh b/system-config/nfs-self-heal/nfs-mount-heal.sh index 037aaa8..cd8c6bf 100644 --- a/system-config/nfs-self-heal/nfs-mount-heal.sh +++ b/system-config/nfs-self-heal/nfs-mount-heal.sh @@ -1,6 +1,6 @@ #!/bin/bash # Deep health check + self-heal for permanent NFS mounts backing -# immich, calibre, and ocis. +# immich, calibre, ocis, and audiobookshelf. # # A plain `ls` on a mount root can succeed even when nested file # handles inside it are stale (ESTALE / errno -116) -- that's the @@ -20,6 +20,7 @@ CHECKS=( "/mnt/nas_family:mnt-nas_family.mount:/mnt/nas_family/immich-library/thumbs:immich_server" "/mnt/nas_books:mnt-nas_books.mount:/mnt/nas_books/calibre-import:calibre" "/mnt/nas_owncloud:mnt-nas_owncloud.mount:/mnt/nas_owncloud:ocis" + "/mnt/nas_audiobooks:mnt-nas_audiobooks.mount:/mnt/nas_audiobooks:audiobookshelf" ) alert() {