Add NFS self-heal automation for immich/calibre/ocis, backed up in git
Stale NFS handles inside containers (ESTALE/errno -116) can persist even after the host-side mount looks healthy, requiring a manual remount + container restart. Adds systemd mount-ready hooks (BindsTo) to restart immich_server/calibre/ocis when their NFS mount unit restarts, plus a 5-minute health-check timer that does a real nested read and force-remounts on staleness. Host files backed up under system-config/nfs-self-heal/ so they can be reinstalled after a host rebuild.
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# NFS Self-Heal (host-level systemd units)
|
||||
|
||||
These files are **not** deployed by Docker/Portainer — they're systemd units and a
|
||||
script that live directly on the host at `/etc/systemd/system/` and
|
||||
`/usr/local/bin/`. They're checked in here purely as a backup so they can be
|
||||
restored if the host is rebuilt. Editing a file here does **not** change
|
||||
running behavior; you have to re-copy it to the host and reload systemd.
|
||||
|
||||
## What this does
|
||||
|
||||
Permanent NFS mounts (`mnt-nas_family.mount`, `mnt-nas_books.mount`,
|
||||
`mnt-nas_owncloud.mount`) can go stale — the host mount looks fine on a
|
||||
top-level `ls`, but file handles nested inside it return ESTALE
|
||||
(errno -116) to processes that opened them earlier, including containers
|
||||
that bind-mount the path. A container restart is required to pick up a
|
||||
fresh handle even after the host-side mount is healthy again. See
|
||||
`../../NAS-CONNECTION-STRATEGIES.md` for the full mount inventory and
|
||||
`../../immich/` / `../../calibre/MOUNT-HISTORY.md` for history on this
|
||||
failure mode (first hit immich on 2026-08-25; calibre-import had the same
|
||||
issue documented earlier but unresolved).
|
||||
|
||||
Two pieces:
|
||||
|
||||
1. **`<container>-mount-ready.service`** (one per affected container) —
|
||||
`BindsTo=mnt-nas_*.mount`, so restarting the mount unit automatically
|
||||
restarts the bound container. Currently covers `immich_server`,
|
||||
`calibre`, `ocis`. (Older instances of this same pattern —
|
||||
`backrest`, `kiwix`, `pmtiles`, `audiobookshelf`, `romm` — already
|
||||
exist on the host but aren't backed up here yet.)
|
||||
2. **`nfs-mount-heal.timer`** → **`nfs-mount-heal.service`** →
|
||||
**`nfs-mount-heal.sh`** — runs every 5 minutes, does a real nested
|
||||
read (not just `ls` the mount root) against `nas_family`, `nas_books`,
|
||||
and `nas_owncloud`. On failure it force-remounts the mount unit
|
||||
(`systemctl restart`), which cascades into the container restart via
|
||||
the hooks above, then re-checks and sends a Telegram alert either way.
|
||||
|
||||
## Reinstall after a host rebuild
|
||||
|
||||
```bash
|
||||
sudo cp *.service *.timer /etc/systemd/system/
|
||||
sudo cp nfs-mount-heal.sh /usr/local/bin/nfs-mount-heal.sh
|
||||
sudo chmod 755 /usr/local/bin/nfs-mount-heal.sh
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now immich-mount-ready.service
|
||||
sudo systemctl enable --now calibre-mount-ready.service
|
||||
sudo systemctl enable --now ocis-mount-ready.service
|
||||
sudo systemctl enable --now nfs-mount-heal.timer
|
||||
```
|
||||
|
||||
Requires the `mnt-nas_family.mount`, `mnt-nas_books.mount`, and
|
||||
`mnt-nas_owncloud.mount` systemd units to already exist (see
|
||||
`NAS-CONNECTION-STRATEGIES.md`) and `.credentials` at the repo root to be
|
||||
present on the host (the heal script sources it for the Telegram bot
|
||||
token/chat ID).
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Restart calibre after NAS books mount is ready
|
||||
After=mnt-nas_books.mount
|
||||
Requires=mnt-nas_books.mount
|
||||
BindsTo=mnt-nas_books.mount
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/docker restart calibre
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=mnt-nas_books.mount
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Restart immich_server after NAS family mount is ready
|
||||
After=mnt-nas_family.mount
|
||||
Requires=mnt-nas_family.mount
|
||||
BindsTo=mnt-nas_family.mount
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/docker restart immich_server
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=mnt-nas_family.mount
|
||||
@@ -0,0 +1,6 @@
|
||||
[Unit]
|
||||
Description=Deep health check and self-heal for permanent NFS mounts (immich/calibre/ocis)
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/nfs-mount-heal.sh
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/bin/bash
|
||||
# Deep health check + self-heal for permanent NFS mounts backing
|
||||
# immich, calibre, and ocis.
|
||||
#
|
||||
# A plain `ls` on a mount root can succeed even when nested file
|
||||
# handles inside it are stale (ESTALE / errno -116) -- that's the
|
||||
# failure mode that broke immich on 2026-08-25 and is documented as
|
||||
# a recurring issue for calibre-import in calibre/MOUNT-HISTORY.md.
|
||||
# This does an actual nested read; on failure it forces a full
|
||||
# remount of the NFS mount unit, which triggers the matching
|
||||
# <container>-mount-ready.service (BindsTo) to restart the container
|
||||
# automatically.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
source /home/poprhythm/docker-infrastructure/.credentials
|
||||
|
||||
# mount:unit:testdir:container
|
||||
CHECKS=(
|
||||
"/mnt/nas_family:mnt-nas_family.mount:/mnt/nas_family/immich-library/thumbs:immich_server"
|
||||
"/mnt/nas_books:mnt-nas_books.mount:/mnt/nas_books/calibre-import:calibre"
|
||||
"/mnt/nas_owncloud:mnt-nas_owncloud.mount:/mnt/nas_owncloud:ocis"
|
||||
)
|
||||
|
||||
alert() {
|
||||
curl -fsS -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
|
||||
-d chat_id="${TELEGRAM_CHAT_ID}" -d text="$1" > /dev/null 2>&1
|
||||
}
|
||||
|
||||
healthy() {
|
||||
local testdir="$1"
|
||||
local f
|
||||
f=$(timeout 10 find "$testdir" -maxdepth 3 -type f -print -quit 2>/dev/null) || return 1
|
||||
[ -n "$f" ] || return 1
|
||||
timeout 10 cat "$f" > /dev/null 2>&1
|
||||
}
|
||||
|
||||
for entry in "${CHECKS[@]}"; do
|
||||
IFS=':' read -r mount unit testdir container <<< "$entry"
|
||||
|
||||
healthy "$testdir" && continue
|
||||
|
||||
logger -t nfs-mount-heal "${mount} unhealthy (stale/unreachable), remounting ${unit}"
|
||||
|
||||
if timeout 30 systemctl restart "$unit"; then
|
||||
sleep 3
|
||||
if healthy "$testdir"; then
|
||||
alert "🔧 NFS self-heal: ${mount} went stale, remounted ${unit} and restarted ${container} automatically. All clear."
|
||||
else
|
||||
alert "⚠️ NFS self-heal: ${mount} was stale, remounted ${unit} and restarted ${container}, but it's still failing the health check. Needs a look."
|
||||
fi
|
||||
else
|
||||
alert "❗ NFS self-heal FAILED: ${mount} is unhealthy and systemctl restart ${unit} failed. Manual intervention needed."
|
||||
fi
|
||||
done
|
||||
@@ -0,0 +1,9 @@
|
||||
[Unit]
|
||||
Description=Run NFS mount self-heal check every 5 minutes
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=5min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Restart ocis after NAS owncloud mount is ready
|
||||
After=mnt-nas_owncloud.mount
|
||||
Requires=mnt-nas_owncloud.mount
|
||||
BindsTo=mnt-nas_owncloud.mount
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/docker restart ocis
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=mnt-nas_owncloud.mount
|
||||
Reference in New Issue
Block a user