docker-infrastructure/system-config/nfs-self-heal/README.md

NFS Self-Heal (host-level systemd units)

These files are not deployed by Docker/Portainer — they're systemd units and a script that live directly on the host at /etc/systemd/system/ and /usr/local/bin/. They're checked in here purely as a backup so they can be restored if the host is rebuilt. Editing a file here does not change running behavior; you have to re-copy it to the host and reload systemd.

What this does

Permanent NFS mounts (mnt-nas_family.mount, mnt-nas_books.mount, mnt-nas_owncloud.mount, mnt-nas_audiobooks.mount) can go stale — the host mount looks fine on a top-level ls, but file handles nested inside it return ESTALE (errno -116) to processes that opened them earlier, including containers that bind-mount the path. A container restart is required to pick up a fresh handle even after the host-side mount is healthy again. See ../../NAS-CONNECTION-STRATEGIES.md for the full mount inventory and ../../immich/ / ../../calibre/MOUNT-HISTORY.md for history on this failure mode (first hit immich on 2026-08-25; calibre-import had the same issue documented earlier but unresolved).

All four .mount units must be permanent (enabled directly, no .automount wrapper), not just present. mnt-nas_audiobooks.mount was mistakenly left on an .automount unit until 2026-09-17 and flapped ESTALE for days as a result — autofs can cycle the underlying mount for reasons other than idle timeout even with TimeoutIdleSec=0, which invalidates any container's bind-mounted reference exactly like a stale handle. The self-heal below papered over it (auto-remounting every ~5 min) but never fixed it; switching to a permanent mount did. See NAS-CONNECTION-STRATEGIES.md's "Docker bind mounts + automount = stale handles" entry.

Two pieces:

1. <container>-mount-ready.service (one per affected container) — BindsTo=mnt-nas_*.mount, so restarting the mount unit automatically restarts the bound container. Currently covers immich_server, calibre, ocis, audiobookshelf. (Older instances of this same pattern — backrest, kiwix, pmtiles, romm — already exist on the host but aren't backed up here yet.)
2. nfs-mount-heal.timer → nfs-mount-heal.service → nfs-mount-heal.sh — runs every 5 minutes, does a real nested read (not just ls the mount root) against nas_family, nas_books, nas_owncloud, and nas_audiobooks. On failure it force-remounts the mount unit (systemctl restart), which cascades into the container restart via the hooks above, then re-checks and sends a Telegram alert either way. This is a safety net for genuine transient staleness, not a fix for a mount that's structurally wrong (e.g. sitting on autofs).

Reinstall after a host rebuild

sudo cp *.mount *.service *.timer /etc/systemd/system/
sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh

sudo systemctl daemon-reload
sudo systemctl enable --now mnt-nas_family.mount
sudo systemctl enable --now mnt-nas_books.mount
sudo systemctl enable --now mnt-nas_owncloud.mount
sudo systemctl enable --now mnt-nas_audiobooks.mount
sudo systemctl enable --now immich-mount-ready.service
sudo systemctl enable --now calibre-mount-ready.service
sudo systemctl enable --now ocis-mount-ready.service
sudo systemctl enable --now audiobookshelf-mount-ready.service
sudo systemctl enable --now nfs-mount-heal.timer

Do not create .automount units for any of these — they must be enabled directly as permanent mounts (see the autofs warning above).

Requires .credentials at the repo root to be present on the host (the heal script sources it for the Telegram bot token/chat ID).