poprhythm
a0921dd387
Adds the 4 permanent .mount units (family/books/owncloud/audiobooks) and audiobookshelf-mount-ready.service to the repo backup, which weren't tracked here yet, and updates the README to document the autofs->permanent mount fix applied to nas_audiobooks today so a host rebuild doesn't recreate the broken automount version.
73 lines
3.7 KiB
Markdown
73 lines
3.7 KiB
Markdown
# NFS Self-Heal (host-level systemd units)
|
|
|
|
These files are **not** deployed by Docker/Portainer — they're systemd units and a
|
|
script that live directly on the host at `/etc/systemd/system/` and
|
|
`/usr/local/bin/`. They're checked in here purely as a backup so they can be
|
|
restored if the host is rebuilt. Editing a file here does **not** change
|
|
running behavior; you have to re-copy it to the host and reload systemd.
|
|
|
|
## What this does
|
|
|
|
Permanent NFS mounts (`mnt-nas_family.mount`, `mnt-nas_books.mount`,
|
|
`mnt-nas_owncloud.mount`, `mnt-nas_audiobooks.mount`) can go stale — the host
|
|
mount looks fine on a top-level `ls`, but file handles nested inside it
|
|
return ESTALE (errno -116) to processes that opened them earlier, including
|
|
containers that bind-mount the path. A container restart is required to pick
|
|
up a fresh handle even after the host-side mount is healthy again. See
|
|
`../../NAS-CONNECTION-STRATEGIES.md` for the full mount inventory and
|
|
`../../immich/` / `../../calibre/MOUNT-HISTORY.md` for history on this
|
|
failure mode (first hit immich on 2026-08-25; calibre-import had the same
|
|
issue documented earlier but unresolved).
|
|
|
|
**All four `.mount` units must be *permanent* (enabled directly, no
|
|
`.automount` wrapper), not just present.** `mnt-nas_audiobooks.mount` was
|
|
mistakenly left on an `.automount` unit until 2026-09-17 and flapped ESTALE
|
|
for days as a result — autofs can cycle the underlying mount for reasons
|
|
other than idle timeout even with `TimeoutIdleSec=0`, which invalidates any
|
|
container's bind-mounted reference exactly like a stale handle. The self-heal
|
|
below papered over it (auto-remounting every ~5 min) but never fixed it;
|
|
switching to a permanent mount did. See NAS-CONNECTION-STRATEGIES.md's
|
|
"Docker bind mounts + automount = stale handles" entry.
|
|
|
|
Two pieces:
|
|
|
|
1. **`<container>-mount-ready.service`** (one per affected container) —
|
|
`BindsTo=mnt-nas_*.mount`, so restarting the mount unit automatically
|
|
restarts the bound container. Currently covers `immich_server`,
|
|
`calibre`, `ocis`, `audiobookshelf`. (Older instances of this same
|
|
pattern — `backrest`, `kiwix`, `pmtiles`, `romm` — already exist on
|
|
the host but aren't backed up here yet.)
|
|
2. **`nfs-mount-heal.timer`** → **`nfs-mount-heal.service`** →
|
|
**`nfs-mount-heal.sh`** — runs every 5 minutes, does a real nested
|
|
read (not just `ls` the mount root) against `nas_family`, `nas_books`,
|
|
`nas_owncloud`, and `nas_audiobooks`. On failure it force-remounts the
|
|
mount unit (`systemctl restart`), which cascades into the container
|
|
restart via the hooks above, then re-checks and sends a Telegram alert
|
|
either way. This is a safety net for genuine transient staleness, not a
|
|
fix for a mount that's structurally wrong (e.g. sitting on autofs).
|
|
|
|
## Reinstall after a host rebuild
|
|
|
|
```bash
|
|
sudo cp *.mount *.service *.timer /etc/systemd/system/
|
|
sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
|
|
sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh
|
|
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now mnt-nas_family.mount
|
|
sudo systemctl enable --now mnt-nas_books.mount
|
|
sudo systemctl enable --now mnt-nas_owncloud.mount
|
|
sudo systemctl enable --now mnt-nas_audiobooks.mount
|
|
sudo systemctl enable --now immich-mount-ready.service
|
|
sudo systemctl enable --now calibre-mount-ready.service
|
|
sudo systemctl enable --now ocis-mount-ready.service
|
|
sudo systemctl enable --now audiobookshelf-mount-ready.service
|
|
sudo systemctl enable --now nfs-mount-heal.timer
|
|
```
|
|
|
|
**Do not** create `.automount` units for any of these — they must be
|
|
enabled directly as permanent mounts (see the autofs warning above).
|
|
|
|
Requires `.credentials` at the repo root to be present on the host (the
|
|
heal script sources it for the Telegram bot token/chat ID).
|