diff --git a/NAS-CONNECTION-STRATEGIES.md b/NAS-CONNECTION-STRATEGIES.md index 124d1c9..f9e40d4 100644 --- a/NAS-CONNECTION-STRATEGIES.md +++ b/NAS-CONNECTION-STRATEGIES.md @@ -8,8 +8,8 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168. | Strategy | Services | Reliability | Notes | |----------|----------|-------------|-------| -| Docker NFS named volume | Plex, qBittorrent, Audiobookshelf | ⚠️ Fragile | Soft mount; stale handles on drive spin-down | -| systemd permanent mount + bind | Calibre (import), oCIS, Immich | ✅ Solid | Hard NFS; permanent; no idle cycling | +| Docker NFS named volume | Plex, qBittorrent | ⚠️ Fragile | Soft mount; stale handles on drive spin-down | +| systemd permanent mount + bind | Calibre (import), oCIS, Immich, Audiobookshelf | ✅ Solid | Hard NFS; permanent; no idle cycling | | systemd automount + bind | Backrest, Filebrowser-Colleen-HD | ✅ Solid | CIFS; TimeoutIdleSec=0; never unmounts once triggered | | Local SSD + lsyncd sync | Calibre (library) | ✅ Best | SQLite never touches NFS; NAS copy is read-only replica | @@ -84,9 +84,11 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168. - **Database**: PostgreSQL on local SSD (`/var/lib/postgresql/data`) — correct, not on NFS ### Audiobookshelf -- **Strategy**: Docker NFS named volume -- **NAS path**: `192.168.1.192:/mnt/user/media/audiobooks` +- **Strategy**: systemd permanent NFS mount + Docker bind mount (fixed 2026-09-17, was previously on `mnt-nas_audiobooks.automount`) +- **Host mount**: `/mnt/nas_audiobooks` (NFS `192.168.1.192:/mnt/user/media/audiobooks`) - **Container path**: `/audiobooks` +- **Mount options**: `nfsvers=3,hard,rw,noatime` +- **Why permanent (not automount)**: see "Docker bind mounts + automount = stale handles" below - this share was on autofs and flapped (ESTALE) on a roughly hourly cadence for days (18 remounts in 82 min on 2026-09-15, 13 in 67 min and then continuous through several more hours on 2026-09-17) before being switched to a permanent mount, exactly matching this repo's own documented gotcha for this pattern. - **Config/metadata**: Local SSD (`/srv/audiobookshelf/`) — correct --- @@ -98,6 +100,7 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168. | `mnt-nas_books.mount` | NFS permanent | `:/mnt/user/media/books` | enabled, always up | | `mnt-nas_owncloud.mount` | NFS permanent | `:/mnt/user/owncloud` | enabled, always up | | `mnt-nas_family.mount` | NFS permanent | `:/mnt/user/family` | enabled, always up | +| `mnt-nas_audiobooks.mount` | NFS permanent | `:/mnt/user/media/audiobooks` | enabled, always up (was `.automount` until 2026-09-17 - see Known Issues) | | `mnt-nas_library.mount` | CIFS automount | `//192.168.1.192/library` | triggered on access, TimeoutIdleSec=0 | | `mnt-nas_library.automount` | CIFS automount | — | enabled | | `mnt-nas_backup.mount` | CIFS automount | `//192.168.1.192/backup` | triggered on access, TimeoutIdleSec=0 | @@ -109,26 +112,30 @@ Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168. - **Docker NFS named volumes**: The `nas_media` volume uses `soft,nolock,timeo=14,vers=3`. This is fragile — short timeout means EIO on spin-up. Plex/qBittorrent survive because they open/close handles per request. If either starts having I/O errors, migrate to the systemd permanent mount pattern. - **Docker's local NFS volume driver requires `host:/path` in `device`, and can't do NFSv4**: it calls `mount(2)` directly rather than shelling out to the `mount.nfs`/`mount.nfs4` userspace helper. Two consequences, both hit in the 2026-09-16 outage below: (1) nfs-utils 2.6.4 (Ubuntu 24.04) rejects the old `device=":/path"` + separate `o=addr=` split form - the host must be in the device string itself now (`device=:/path`); (2) `vers=4`/`nfsvers=4` fails with "protocol not supported" via the raw syscall even though a manual `mount -t nfs -o nfsvers=4 ...` succeeds (that goes through `mount.nfs`, which does version negotiation the raw syscall skips). Stick to `vers=3` for any Docker-managed NFS volume on this host. -- **Docker bind mounts + automount = stale handles**: Docker snapshots the mount reference at container start. If autofs cycles (unmount + remount) between container restarts, the container's bind mount goes stale while the host sees the path fine. Fix: use permanent `.mount` (no `.automount`) for any share that Docker containers bind-mount. +- **Docker bind mounts + automount = stale handles**: Docker snapshots the mount reference at container start. If autofs cycles (unmount + remount) between container restarts, the container's bind mount goes stale while the host sees the path fine. Fix: use permanent `.mount` (no `.automount`) for any share that Docker containers bind-mount. **Hit in practice 2026-09-17**: `mnt-nas_audiobooks.mount` had been left on an `.automount` unit (autofs, `TimeoutIdleSec=0`) instead of the permanent-mount pattern used by every other Docker-bind-mounted NFS share. It flapped ESTALE on a recurring, multi-hour cadence over several days (self-healed every ~5 min by `nfs-mount-heal.sh`, but never actually fixed) even though `TimeoutIdleSec=0` should prevent autofs idle expiry - autofs can still cycle the underlying mount for reasons other than idle timeout, and any such cycle invalidates the container's bind-mounted reference exactly as this gotcha describes. Fix: `systemctl disable --now mnt-nas_audiobooks.automount`, delete the unit file, `systemctl enable --now mnt-nas_audiobooks.mount` directly (matching nas_family/nas_books/nas_owncloud exactly). Confirmed stable afterward. - **SQLite on NFS**: Never store SQLite databases (calibre `metadata.db`/`notes.db`, any app DB) on NFS. Even with `hard` mounts and NLM locking, transient NFS errors cause `SQLITE_IOERR`. Keep SQLite on local SSD; sync non-SQLite files to NAS if sharing is needed. - **CIFS vs NFS for file permissions**: CIFS enforces server-side ACLs based on the SMB user. Files created via NFS by uid=99 with mode 600 are inaccessible via CIFS. Use NFS when container needs uid-mapped access to NFS-created files. - **lsyncd for NAS sync**: inotify-based, syncs within ~15 seconds of any write. Config at `/etc/lsyncd/lsyncd.conf.lua`. Log at `/var/log/lsyncd.log`. --- -## Self-Healing (nas_family / nas_books / nas_owncloud) +## Self-Healing (nas_family / nas_books / nas_owncloud / nas_audiobooks) As of 2026-08-25, stale-handle recovery for immich, calibre, and oCIS is -automated instead of requiring manual `systemctl restart` + `docker restart`: +automated instead of requiring manual `systemctl restart` + `docker restart` +(audiobookshelf added 2026-09-16, and its underlying mount fixed to actually +stop going stale on 2026-09-17 - see Known Issues): - `-mount-ready.service` (`immich-mount-ready`, `calibre-mount-ready`, - `ocis-mount-ready`) — `BindsTo=mnt-nas_*.mount`, restarts the container - whenever its mount unit restarts. + `ocis-mount-ready`, `audiobookshelf-mount-ready`) — `BindsTo=mnt-nas_*.mount`, + restarts the container whenever its mount unit restarts. - `nfs-mount-heal.timer` (every 5 min) → `nfs-mount-heal.sh` — does a nested file read (not just `ls` the mount root, which can succeed even when nested - handles are stale) against each of the three mounts. On failure it force- + handles are stale) against each of the four mounts. On failure it force- remounts the mount unit, which cascades into the container restart above, - then sends a Telegram alert. + then sends a Telegram alert. This is a safety net for genuine transient + staleness - it is not a substitute for fixing a mount that's flapping for + a structural reason (e.g. sitting on autofs when it shouldn't be). Host files backed up at `system-config/nfs-self-heal/` in this repo (see its README for the reinstall procedure — these are systemd units, not @@ -138,7 +145,7 @@ docker-compose, so they don't redeploy via Portainer). ## Recommended Migration (future) -Plex, qBittorrent, Immich, and Audiobookshelf all use the fragile Docker NFS named volume pattern. The safer pattern is systemd permanent mount + Docker bind mount (as used by calibre-import and oCIS). This would involve: +Plex and qBittorrent still use the fragile Docker NFS named volume pattern (Immich and Audiobookshelf have both since migrated off it). The safer pattern is systemd permanent mount + Docker bind mount (as used by calibre-import, oCIS, and now audiobookshelf). This would involve: 1. Create `/mnt/nas_media` systemd permanent mount unit 2. Update compose files to use bind mounts instead of the external `nas_media` volume 3. `docker volume rm nas_media` after redeployment