poprhythm
d397938c18
Root cause of the recurring nas_audiobooks ESTALE flapping (18 remounts in 82min on 9/15, 13+ continuous through several hours on 9/17): the mount sat behind mnt-nas_audiobooks.automount instead of a permanent .mount unit, exactly matching this doc's own pre-existing "Docker bind mounts + automount = stale handles" gotcha. Switched to systemd enable --now on the .mount unit directly (matching nas_family/nas_books/nas_owncloud), removed the automount unit. Confirmed stable after the change - self-heal script no longer needed as anything but a backstop for this share.
152 lines
10 KiB
Markdown
152 lines
10 KiB
Markdown
# NAS Connection Strategies
|
||
|
||
Summary of how each service connects to the unRAID NAS (192.168.1.192 / 192.168.1.4).
|
||
|
||
---
|
||
|
||
## Strategy Overview
|
||
|
||
| Strategy | Services | Reliability | Notes |
|
||
|----------|----------|-------------|-------|
|
||
| Docker NFS named volume | Plex, qBittorrent | ⚠️ Fragile | Soft mount; stale handles on drive spin-down |
|
||
| systemd permanent mount + bind | Calibre (import), oCIS, Immich, Audiobookshelf | ✅ Solid | Hard NFS; permanent; no idle cycling |
|
||
| systemd automount + bind | Backrest, Filebrowser-Colleen-HD | ✅ Solid | CIFS; TimeoutIdleSec=0; never unmounts once triggered |
|
||
| Local SSD + lsyncd sync | Calibre (library) | ✅ Best | SQLite never touches NFS; NAS copy is read-only replica |
|
||
|
||
---
|
||
|
||
## Per-Service Details
|
||
|
||
### Plex
|
||
- **Strategy**: Docker NFS named volume (`nas_media`, external)
|
||
- **NAS path**: `192.168.1.192:/mnt/user/media`
|
||
- **Mount options**: `nfsvers=4,soft,nolock,timeo=14,rsize=8192,wsize=8192`
|
||
- **Container path**: `/data`
|
||
- **Risk**: `soft` + `timeo=14` means EIO after 14 retries × 0.1s = ~1.4s timeout. If unRAID drives spin down, Plex gets I/O errors. Works in practice because Plex opens/closes file handles per-request rather than holding them open.
|
||
|
||
### qBittorrent (all 3 instances: open, vpn, vpn2)
|
||
- **Strategy**: Docker NFS named volume (`nas_media`, same as Plex)
|
||
- **NAS path**: `192.168.1.192:/mnt/user/media`
|
||
- **Container path**: `/data`
|
||
- **Risk**: Same as Plex. Active downloads write continuously — more exposure to spin-down EIO than Plex.
|
||
|
||
### Calibre
|
||
- **Strategy (library)**: Local SSD + lsyncd one-way sync to NAS
|
||
- Local: `/srv/calibre/library` → container `/config/Calibre Library`
|
||
- lsyncd syncs to `/mnt/nas_books/calibre/Calibre Library` within ~15 seconds
|
||
- SQLite (`metadata.db`, `notes.db`) never touches NFS — eliminates `apsw.IOError`
|
||
- **Strategy (import)**: systemd permanent NFS mount + Docker bind mount
|
||
- Host: `/mnt/nas_books/calibre-import` (permanent NFS, no automount)
|
||
- Container: `/calibre-import`
|
||
- NAS path: `192.168.1.192:/mnt/user/media/books`
|
||
- **Mount options**: `nfsvers=3,hard,rw,noatime`
|
||
- **Why permanent (not automount)**: Docker bind mounts snapshot the mount reference at container start. If autofs cycles between restarts, the container gets a stale handle. Permanent mount avoids this entirely.
|
||
- **Manual book add** (when auto-add fails):
|
||
```bash
|
||
docker stop calibre
|
||
docker run --rm \
|
||
-v /srv/calibre/library:/config/Calibre\ Library \
|
||
-v /mnt/nas_books/calibre-import:/calibre-import \
|
||
lscr.io/linuxserver/calibre:latest \
|
||
sh -c "cd /opt/calibre && LD_LIBRARY_PATH=/opt/calibre ./calibredb add \
|
||
--library-path '/config/Calibre Library' '/calibre-import/<book>' 2>&1"
|
||
docker start calibre
|
||
```
|
||
|
||
### Filebrowser-Colleen-HD
|
||
- **Strategy**: systemd CIFS automount + Docker bind mount
|
||
- **Host mount**: `/mnt/nas_backup` (CIFS `//192.168.1.192/backup`)
|
||
- **Container path**: `/folder`
|
||
- **Mount options**: `vers=3.0,uid=0,gid=0,noperm,noserverino,soft,TimeoutIdleSec=0`
|
||
- **Why CIFS**: Container runs as root; CIFS with `uid=0,gid=0` maps all files to root
|
||
- **TimeoutIdleSec=0**: Never auto-unmounts once triggered; safe for persistent container access
|
||
|
||
### Backrest (backup service)
|
||
- **Strategy**: systemd CIFS automount + Docker bind mount (same mount as Filebrowser)
|
||
- **Host mount**: `/mnt/nas_backup/nas-docker-data`
|
||
- **Container path**: `/backup-export`
|
||
- **Same CIFS mount as Filebrowser-Colleen-HD** (`mnt-nas_backup`)
|
||
|
||
### oCIS (ownCloud Infinite Scale)
|
||
- **Strategy**: systemd permanent NFS mount + Docker bind mount
|
||
- **Host mount**: `/mnt/nas_owncloud` (permanent NFS, no automount)
|
||
- **Container path**: `/var/lib/ocis/storage/users`
|
||
- **NAS path**: `192.168.1.192:/mnt/user/owncloud`
|
||
- **Mount options**: `nfsvers=3,hard,rw,noatime`
|
||
- **Local config/index**: `/srv/ocis/data` and `/srv/ocis/config` stay on local SSD
|
||
|
||
### Immich
|
||
- **Strategy**: systemd permanent NFS mount + Docker bind mount
|
||
- **Host mount**: `/mnt/nas_family` (NFS `192.168.1.192:/mnt/user/family`)
|
||
- **Container path**: `/usr/src/app/upload` → `/mnt/nas_family/immich-library`
|
||
- **Mount options**: `nfsvers=3,hard,rw,noatime`
|
||
- **Why permanent (not automount)**: Drive doesn't spin down; permanent mount avoids any autofs cycling risk
|
||
- **Database**: PostgreSQL on local SSD (`/var/lib/postgresql/data`) — correct, not on NFS
|
||
|
||
### Audiobookshelf
|
||
- **Strategy**: systemd permanent NFS mount + Docker bind mount (fixed 2026-09-17, was previously on `mnt-nas_audiobooks.automount`)
|
||
- **Host mount**: `/mnt/nas_audiobooks` (NFS `192.168.1.192:/mnt/user/media/audiobooks`)
|
||
- **Container path**: `/audiobooks`
|
||
- **Mount options**: `nfsvers=3,hard,rw,noatime`
|
||
- **Why permanent (not automount)**: see "Docker bind mounts + automount = stale handles" below - this share was on autofs and flapped (ESTALE) on a roughly hourly cadence for days (18 remounts in 82 min on 2026-09-15, 13 in 67 min and then continuous through several more hours on 2026-09-17) before being switched to a permanent mount, exactly matching this repo's own documented gotcha for this pattern.
|
||
- **Config/metadata**: Local SSD (`/srv/audiobookshelf/`) — correct
|
||
|
||
---
|
||
|
||
## systemd Mount Units
|
||
|
||
| Unit | Type | Share | Status |
|
||
|------|------|-------|--------|
|
||
| `mnt-nas_books.mount` | NFS permanent | `:/mnt/user/media/books` | enabled, always up |
|
||
| `mnt-nas_owncloud.mount` | NFS permanent | `:/mnt/user/owncloud` | enabled, always up |
|
||
| `mnt-nas_family.mount` | NFS permanent | `:/mnt/user/family` | enabled, always up |
|
||
| `mnt-nas_audiobooks.mount` | NFS permanent | `:/mnt/user/media/audiobooks` | enabled, always up (was `.automount` until 2026-09-17 - see Known Issues) |
|
||
| `mnt-nas_library.mount` | CIFS automount | `//192.168.1.192/library` | triggered on access, TimeoutIdleSec=0 |
|
||
| `mnt-nas_library.automount` | CIFS automount | — | enabled |
|
||
| `mnt-nas_backup.mount` | CIFS automount | `//192.168.1.192/backup` | triggered on access, TimeoutIdleSec=0 |
|
||
| `mnt-nas_backup.automount` | CIFS automount | — | enabled |
|
||
|
||
---
|
||
|
||
## Known Issues & Gotchas
|
||
|
||
- **Docker NFS named volumes**: The `nas_media` volume uses `soft,nolock,timeo=14,vers=3`. This is fragile — short timeout means EIO on spin-up. Plex/qBittorrent survive because they open/close handles per request. If either starts having I/O errors, migrate to the systemd permanent mount pattern.
|
||
- **Docker's local NFS volume driver requires `host:/path` in `device`, and can't do NFSv4**: it calls `mount(2)` directly rather than shelling out to the `mount.nfs`/`mount.nfs4` userspace helper. Two consequences, both hit in the 2026-09-16 outage below: (1) nfs-utils 2.6.4 (Ubuntu 24.04) rejects the old `device=":/path"` + separate `o=addr=<ip>` split form - the host must be in the device string itself now (`device=<ip>:/path`); (2) `vers=4`/`nfsvers=4` fails with "protocol not supported" via the raw syscall even though a manual `mount -t nfs -o nfsvers=4 ...` succeeds (that goes through `mount.nfs`, which does version negotiation the raw syscall skips). Stick to `vers=3` for any Docker-managed NFS volume on this host.
|
||
- **Docker bind mounts + automount = stale handles**: Docker snapshots the mount reference at container start. If autofs cycles (unmount + remount) between container restarts, the container's bind mount goes stale while the host sees the path fine. Fix: use permanent `.mount` (no `.automount`) for any share that Docker containers bind-mount. **Hit in practice 2026-09-17**: `mnt-nas_audiobooks.mount` had been left on an `.automount` unit (autofs, `TimeoutIdleSec=0`) instead of the permanent-mount pattern used by every other Docker-bind-mounted NFS share. It flapped ESTALE on a recurring, multi-hour cadence over several days (self-healed every ~5 min by `nfs-mount-heal.sh`, but never actually fixed) even though `TimeoutIdleSec=0` should prevent autofs idle expiry - autofs can still cycle the underlying mount for reasons other than idle timeout, and any such cycle invalidates the container's bind-mounted reference exactly as this gotcha describes. Fix: `systemctl disable --now mnt-nas_audiobooks.automount`, delete the unit file, `systemctl enable --now mnt-nas_audiobooks.mount` directly (matching nas_family/nas_books/nas_owncloud exactly). Confirmed stable afterward.
|
||
- **SQLite on NFS**: Never store SQLite databases (calibre `metadata.db`/`notes.db`, any app DB) on NFS. Even with `hard` mounts and NLM locking, transient NFS errors cause `SQLITE_IOERR`. Keep SQLite on local SSD; sync non-SQLite files to NAS if sharing is needed.
|
||
- **CIFS vs NFS for file permissions**: CIFS enforces server-side ACLs based on the SMB user. Files created via NFS by uid=99 with mode 600 are inaccessible via CIFS. Use NFS when container needs uid-mapped access to NFS-created files.
|
||
- **lsyncd for NAS sync**: inotify-based, syncs within ~15 seconds of any write. Config at `/etc/lsyncd/lsyncd.conf.lua`. Log at `/var/log/lsyncd.log`.
|
||
|
||
---
|
||
|
||
## Self-Healing (nas_family / nas_books / nas_owncloud / nas_audiobooks)
|
||
|
||
As of 2026-08-25, stale-handle recovery for immich, calibre, and oCIS is
|
||
automated instead of requiring manual `systemctl restart` + `docker restart`
|
||
(audiobookshelf added 2026-09-16, and its underlying mount fixed to actually
|
||
stop going stale on 2026-09-17 - see Known Issues):
|
||
|
||
- `<container>-mount-ready.service` (`immich-mount-ready`, `calibre-mount-ready`,
|
||
`ocis-mount-ready`, `audiobookshelf-mount-ready`) — `BindsTo=mnt-nas_*.mount`,
|
||
restarts the container whenever its mount unit restarts.
|
||
- `nfs-mount-heal.timer` (every 5 min) → `nfs-mount-heal.sh` — does a nested
|
||
file read (not just `ls` the mount root, which can succeed even when nested
|
||
handles are stale) against each of the four mounts. On failure it force-
|
||
remounts the mount unit, which cascades into the container restart above,
|
||
then sends a Telegram alert. This is a safety net for genuine transient
|
||
staleness - it is not a substitute for fixing a mount that's flapping for
|
||
a structural reason (e.g. sitting on autofs when it shouldn't be).
|
||
|
||
Host files backed up at `system-config/nfs-self-heal/` in this repo (see its
|
||
README for the reinstall procedure — these are systemd units, not
|
||
docker-compose, so they don't redeploy via Portainer).
|
||
|
||
---
|
||
|
||
## Recommended Migration (future)
|
||
|
||
Plex and qBittorrent still use the fragile Docker NFS named volume pattern (Immich and Audiobookshelf have both since migrated off it). The safer pattern is systemd permanent mount + Docker bind mount (as used by calibre-import, oCIS, and now audiobookshelf). This would involve:
|
||
1. Create `/mnt/nas_media` systemd permanent mount unit
|
||
2. Update compose files to use bind mounts instead of the external `nas_media` volume
|
||
3. `docker volume rm nas_media` after redeployment
|