poprhythm
c95fdef0e5
Root cause of nas_audiobooks still flapping after the automount fix: the health check finds the first file in each share via `find -quit` and `cat`s the whole thing within a 10s timeout. That's fine for immich's 13-byte marker and owncloud's 0-byte one, but audiobooks has no such marker - find picked a real 2.9GB audiobook file, which can never fully transfer in 10s regardless of mount health. Switched to `head -c 64k`, which only needs to prove the file handle/NFS path is alive. Confirmed: the mount was never actually unhealthy post the automount fix, this test methodology was generating the "stale" verdicts (cat exit=124) itself. Also captures and logs/alerts the actual find/head error text now instead of a generic "stale/unreachable" guess, so any future real failure is diagnosable without SSH archaeology.
NFS Self-Heal (host-level systemd units)
These files are not deployed by Docker/Portainer — they're systemd units and a
script that live directly on the host at /etc/systemd/system/ and
/usr/local/bin/. They're checked in here purely as a backup so they can be
restored if the host is rebuilt. Editing a file here does not change
running behavior; you have to re-copy it to the host and reload systemd.
What this does
Permanent NFS mounts (mnt-nas_family.mount, mnt-nas_books.mount,
mnt-nas_owncloud.mount, mnt-nas_audiobooks.mount) can go stale — the host
mount looks fine on a top-level ls, but file handles nested inside it
return ESTALE (errno -116) to processes that opened them earlier, including
containers that bind-mount the path. A container restart is required to pick
up a fresh handle even after the host-side mount is healthy again. See
../../NAS-CONNECTION-STRATEGIES.md for the full mount inventory and
../../immich/ / ../../calibre/MOUNT-HISTORY.md for history on this
failure mode (first hit immich on 2026-08-25; calibre-import had the same
issue documented earlier but unresolved).
All four .mount units must be permanent (enabled directly, no
.automount wrapper), not just present. mnt-nas_audiobooks.mount was
mistakenly left on an .automount unit until 2026-09-17 and flapped ESTALE
for days as a result — autofs can cycle the underlying mount for reasons
other than idle timeout even with TimeoutIdleSec=0, which invalidates any
container's bind-mounted reference exactly like a stale handle. The self-heal
below papered over it (auto-remounting every ~5 min) but never fixed it;
switching to a permanent mount did. See NAS-CONNECTION-STRATEGIES.md's
"Docker bind mounts + automount = stale handles" entry.
Two pieces:
<container>-mount-ready.service(one per affected container) —BindsTo=mnt-nas_*.mount, so restarting the mount unit automatically restarts the bound container. Currently coversimmich_server,calibre,ocis,audiobookshelf. (Older instances of this same pattern —backrest,kiwix,pmtiles,romm— already exist on the host but aren't backed up here yet.)nfs-mount-heal.timer→nfs-mount-heal.service→nfs-mount-heal.sh— runs every 5 minutes, does a real nested read (not justlsthe mount root) againstnas_family,nas_books,nas_owncloud, andnas_audiobooks. On failure it force-remounts the mount unit (systemctl restart), which cascades into the container restart via the hooks above, then re-checks and sends a Telegram alert either way. This is a safety net for genuine transient staleness, not a fix for a mount that's structurally wrong (e.g. sitting on autofs).
Reinstall after a host rebuild
sudo cp *.mount *.service *.timer /etc/systemd/system/
sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh
sudo systemctl daemon-reload
sudo systemctl enable --now mnt-nas_family.mount
sudo systemctl enable --now mnt-nas_books.mount
sudo systemctl enable --now mnt-nas_owncloud.mount
sudo systemctl enable --now mnt-nas_audiobooks.mount
sudo systemctl enable --now immich-mount-ready.service
sudo systemctl enable --now calibre-mount-ready.service
sudo systemctl enable --now ocis-mount-ready.service
sudo systemctl enable --now audiobookshelf-mount-ready.service
sudo systemctl enable --now nfs-mount-heal.timer
Do not create .automount units for any of these — they must be
enabled directly as permanent mounts (see the autofs warning above).
Requires .credentials at the repo root to be present on the host (the
heal script sources it for the Telegram bot token/chat ID).