poprhythm
42a31b6a44
Stale NFS handles inside containers (ESTALE/errno -116) can persist even after the host-side mount looks healthy, requiring a manual remount + container restart. Adds systemd mount-ready hooks (BindsTo) to restart immich_server/calibre/ocis when their NFS mount unit restarts, plus a 5-minute health-check timer that does a real nested read and force-remounts on staleness. Host files backed up under system-config/nfs-self-heal/ so they can be reinstalled after a host rebuild.
2.6 KiB
NFS Self-Heal (host-level systemd units)
These files are not deployed by Docker/Portainer — they're systemd units and a
script that live directly on the host at /etc/systemd/system/ and
/usr/local/bin/. They're checked in here purely as a backup so they can be
restored if the host is rebuilt. Editing a file here does not change
running behavior; you have to re-copy it to the host and reload systemd.
What this does
Permanent NFS mounts (mnt-nas_family.mount, mnt-nas_books.mount,
mnt-nas_owncloud.mount) can go stale — the host mount looks fine on a
top-level ls, but file handles nested inside it return ESTALE
(errno -116) to processes that opened them earlier, including containers
that bind-mount the path. A container restart is required to pick up a
fresh handle even after the host-side mount is healthy again. See
../../NAS-CONNECTION-STRATEGIES.md for the full mount inventory and
../../immich/ / ../../calibre/MOUNT-HISTORY.md for history on this
failure mode (first hit immich on 2026-08-25; calibre-import had the same
issue documented earlier but unresolved).
Two pieces:
<container>-mount-ready.service(one per affected container) —BindsTo=mnt-nas_*.mount, so restarting the mount unit automatically restarts the bound container. Currently coversimmich_server,calibre,ocis. (Older instances of this same pattern —backrest,kiwix,pmtiles,audiobookshelf,romm— already exist on the host but aren't backed up here yet.)nfs-mount-heal.timer→nfs-mount-heal.service→nfs-mount-heal.sh— runs every 5 minutes, does a real nested read (not justlsthe mount root) againstnas_family,nas_books, andnas_owncloud. On failure it force-remounts the mount unit (systemctl restart), which cascades into the container restart via the hooks above, then re-checks and sends a Telegram alert either way.
Reinstall after a host rebuild
sudo cp *.service *.timer /etc/systemd/system/
sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh
sudo systemctl daemon-reload
sudo systemctl enable --now immich-mount-ready.service
sudo systemctl enable --now calibre-mount-ready.service
sudo systemctl enable --now ocis-mount-ready.service
sudo systemctl enable --now nfs-mount-heal.timer
Requires the mnt-nas_family.mount, mnt-nas_books.mount, and
mnt-nas_owncloud.mount systemd units to already exist (see
NAS-CONNECTION-STRATEGIES.md) and .credentials at the repo root to be
present on the host (the heal script sources it for the Telegram bot
token/chat ID).