Adds the 4 permanent .mount units (family/books/owncloud/audiobooks) and
audiobookshelf-mount-ready.service to the repo backup, which weren't
tracked here yet, and updates the README to document the autofs->permanent
mount fix applied to nas_audiobooks today so a host rebuild doesn't
recreate the broken automount version.
Root cause of the recurring nas_audiobooks ESTALE flapping (18 remounts in
82min on 9/15, 13+ continuous through several hours on 9/17): the mount
sat behind mnt-nas_audiobooks.automount instead of a permanent .mount unit,
exactly matching this doc's own pre-existing "Docker bind mounts + automount
= stale handles" gotcha. Switched to systemd enable --now on the .mount
unit directly (matching nas_family/nas_books/nas_owncloud), removed the
automount unit. Confirmed stable after the change - self-heal script no
longer needed as anything but a backstop for this share.
Host reboot broke the nas_media Docker NFS volume for 6+ hours (plex,
qbittorrent x3, navidrome, sonarr, jellyfin all down): nfs-utils 2.6.4 no
longer accepts the ":/path" + addr= device split, and separately Docker's
local volume driver can't do nfsvers=4 (calls mount(2) directly, skipping
the mount.nfs helper's version negotiation). Volume recreated with an
explicit host:/path device and vers=3. Updating docs/comments to match
the corrected live config so the next person doesn't recreate the same
broken volume from the stale example.
Host rebooted 2026-09-16 17:47 UTC, recreating every container at once and
producing 70 alerts in a single burst - all traced to benign init-time
patterns that will recur on every future reboot: systemd-executor's PAM
reads during boot, firefly-iii/fidi's s6 healthcheck shell, and romm's own
loopback redis/app connection at container start.
Distilled from tonight's session building SUB/WAVE anchor-playlist
shows by hand: a CLI (search/artist/album/playlist get/create/replace/
add/remove) plus a playlist-audit command that flags accidental
full-album dumps, replacing the one-off curl/Python snippets used
throughout. SKILL.md documents the gotchas hit along the way
(case-sensitive artist matching, deluxe-reissue duplicate tracks,
createPlaylist's full-replace semantics) and this session's curation
conventions (15-20 tracks/artist, prefer playlistStrict).
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
The prior fix dropped numCtx to 8192 to guarantee full GPU offload,
but real multi-turn djAgentPick conversations peak around 7.5k tokens
-- leaving almost no headroom before llama.cpp's context-shift drops
the front of the prompt (system instructions + tool defs, including
`done`). Result: "agent stopped without calling done" recurring every
few picks. 11264 is the largest context that still offloads all 29
layers on this GPU (tested empirically -- 12288 fell 15MB short and
dropped to 28/29), leaving ~3.7k tokens of margin over the observed
peak.
Also: OLLAMA_CONTEXT_LENGTH had drifted onto the running container via
an earlier manual `docker run` and was never in this compose file, so
a prior git-redeploy silently kept the manual value instead of the
committed one. Committing it here closes that drift.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
The ollama stack's GitConfig.Authentication.Password had gone empty
(same class of issue as the prior Gitea log-poisoning/token-rotation
work), causing redeploy to fail with "Unable to clone git repository
directory". This resupplies repositoryUsername/repositoryPassword
from GITEA_USER/GITEA_TOKEN on redeploy, scoped to a new subcommand
rather than changing default redeploy behavior for every stack.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
Observed KV cache size for qwen2.5:7b at fixed 16384 context varying
224MB-896MB between model loads with flash attention off. The larger
figure pushes total memory needs just past the GPU's free VRAM, so
some loads only offload 25/29 layers instead of 29/29 -- causing
DJ-agent pick latency to jump from ~1s to multiple minutes. This GPU
(6GB) has very little slack for this model/context combination even
after freeing obico and stable-diffusion's VRAM reservations.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
obico's own comment already notes its :cuda tag falls back to CPU
inference on this driver/GPU combo, but it was still reserving 826MB
of VRAM it never used productively. That squeeze was forcing ollama
to offload only 20/29 model layers to GPU, pushing the rest onto CPU
and causing severe latency (multi-minute LLM calls) that stalled
SUB/WAVE's track picking.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
0.64.0 re-encoded every internal ID in the DB (a documented breaking
change), which watchtower auto-applied and silently broke every
SUB/WAVE show's playlistIds reference. Pinning stops watchtower from
jumping versions without a deliberate review of release notes.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
ND_AUTOIMPORTPLAYLISTS defaults to true and was importing every .m3u it
found scanning the music folder (one per album from ripping software) as
a playlist -- 2059 of them, 1605 completely empty, hundreds duplicated.
Cleaned up the existing junk via the Subsonic API; this stops it from
coming back on the next scan.
User added several previously-recommended albums/compilations to the
library (AC/DC Ballbreaker, Rancid ...And Out Come the Wolves, Garbage,
Foo Fighters, Goo Goo Dolls, Melissa Etheridge x2, Big Shiny Tunes, MTV
Buzz Bin Vol. 2, etc.) and rescanned Navidrome. Re-running the matcher
across all 12 shows resolved many previous misses, which surfaced 6 new
cross-show duplicates (a song now matching in two shows at once). Each
was resolved by keeping the track on whichever show has its authentic/
better chart rank, matching the same policy used earlier this session.
Match counts after re-run: 1994 shows 13/29, 10/24, 10/27, 14/26; 1995
shows 22/30, 20/30, 19/30, 10/17; 1996 shows 17/35, 15/28, 19/29, 13/30.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
Adds Tunecaster Rock Top 30 for 1994-01-08, 1994-04-09, 1994-07-09, and
1994-10-15, checked against all existing shows' actual live Navidrome
playlist contents (not just the hand-compiled list of chart titles used so
far, which had already let one duplicate slip through -- see below) for
exact-track duplicates.
Also fixes a real duplicate: "Everclear - Santa Monica" was airing in both
the 1995-12-16 and 1996-01-13 shows, because the manual text-based dedup
check compared exact strings and missed that "Santa Monica" and "Santa
Monica (Watch The World Die)" resolve to the same library track (the
matcher's containment-boost logic correctly treats them as the same song;
the manual check didn't). Removed from 1995-12-16, kept on the thinner
1996-01-13 show.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
A playlistStrict show's no-repeat window scales down to a fraction of the
playlist size and disables entirely below 15 tracks (recency.ts's
effectiveNoRepeatWindow) -- every historical-chart playlist this script
builds is well under that (7-30 tracks), so repeats were essentially
unthrottled. playlistExhaust switches to a full-rotation window instead.
Also applied this fix retroactively to all 8 existing shows.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
Adds Tunecaster Rock Top 30 for 1996-01-13, 1996-04-13, 1996-07-13, and
1996-12-14, cross-checked against all 4 existing 1995 shows and each other
for exact-track duplicates.
Also fixes a real false-positive: "John Mellencamp - Just Another Day" was
matching Jon Secada's unrelated song of the same generic title, because a
perfect title score could carry a middling-but-not-absurd artist-name
coincidence (0.56) over the acceptance threshold. Added a hard artist
similarity floor (0.65) independent of the combined score.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
score_candidate compared library tags verbatim against chart artist/title, so a
reissue's composite artist credit (e.g. "Green Day • Billie Joe Armstrong, Mike
Dirnt, & Tre Cool") or a dropped subtitle ("Sour Times" vs "Sour Times (Nobody
Loves Me)") could push an otherwise-correct match just under the score
threshold. Added whole-word substring containment as a scoring boost for both
fields, which recovered several real misses.
Also adds three more 1995 Tunecaster Rock Top 30 weeks (Jan 14, Jun 10, Dec 16)
alongside the existing Oct 28 show, cross-checked against it for exact-track
duplicates (swapped for a different real single by the same artist, or dropped
where none existed in the library).
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
Given a hand-transcribed {station, date, tracks} JSON file (chart data is
sourced manually in conversation, never scraped in code -- ARSA's robots.txt
disallows automated fetching of its survey pages), matches tracks against the
Navidrome library, builds/updates a Navidrome playlist from the matches, and
wires it into a SUB/WAVE show with playlistStrict so it plays only that chart.
Claude-Session: https://claude.ai/code/session_01L7Rwa6guD5wK8F8tWQwcJX
Third shape variant: qbt-relink.sh handles season packs (N files
matching N files), qbt-relink-daily.sh handles single-file
date-based episodes (Jeopardy). This handles single-file torrents
carrying one standard SxxEyy episode (e.g. Lower Decks, downloaded
one file per torrent rather than as season packs) - matches by
parsed season/episode number against the single file in that
Season NN folder carrying the same numbers. Same safety model
throughout: one blocking pass, pre/post filesystem size check
independent of qBittorrent's self-reported state, idempotent.
Verified against a live torrent - clean single-pass relink.
Claude-Session: https://claude.ai/code/session_01KjrQwEec4vzukbnHiFpFmk
many single-file torrents
Runs a bounded-concurrency job pool (default 4) across every torrent
matching a name substring, logging each to its own file. The
orchestration here carries no data-safety risk beyond inefficiency
- each qbt-relink-daily.sh invocation is independently safe and
idempotent, so a driver bug can misreport or waste time but can't
corrupt data. Failures are logged and skipped, never auto-retried
within a run - re-running the whole batch picks up anything
incomplete since both scripts are idempotent.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
qbt-relink.sh assumes torrent file count matches the destination
folder's file count 1:1 - true for season packs, false for a daily
show like Jeopardy where each torrent holds one episode but the
destination Season <year> folder holds every episode that aired
that year. This matches by date parsed from the torrent's own
filename against the single dated file in that season folder,
using the same safety model (single blocking pass, pre/post
filesystem size check, idempotent, no repeated manual triggering).
Verified against a live torrent - clean single-pass relink,
post-check confirmed the file intact.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
The command_id extraction passed the full command-submission
response (which echoes back the whole files array) as a python argv
arg - same ARG_MAX issue as the api_post fix, one step later. Now
piped via stdin. The actual command submission already worked with
the first fix; this only affected reading back the response.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
api_post/api_put passed the JSON payload directly as a curl -d
command-line argument, which hit the OS ARG_MAX limit importing
Jeopardy's 1269-file library (a large date-based show is exactly
the case where a batch import payload gets big). Now written to a
temp file and passed via curl's -d @file instead.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Reflects the rewritten script's blocking single-pass model and the
hard rule against re-triggering stop/setLocation/recheck against the
same hash more than once. Marks Incident 3 as fixed and verified.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Root cause of the data-loss incident: repeated manual re-triggering
(stop/setLocation/recheck called multiple times across separate
debugging invocations) raced against qBittorrent's own automatic
incomplete-file management. The 7 clean successes earlier all
completed in one uninterrupted pass; the 2 that failed were the ones
manually re-triggered while investigating.
Redesign: the whole operation is now one blocking pass (stop ->
relocate -> rename -> recheck -> poll to completion -> verify ->
report) with a hard rule never to call recheck/stop/setLocation
against the same hash a second time while a previous run might
still be settling. Adds:
- A pre-flight filesystem manifest (name+size) of the destination,
and a post-recheck comparison against it - the real ground-truth
safety net, independent of trusting qBittorrent's self-reported
state.
- Idempotency: skips already-correctly-relinked torrents rather
than re-touching them. Found and fixed a real gap here during
testing - the first version trusted qBittorrent's per-file
"progress" alone, which can be stale (cached from before a move)
and produced a false "already good" on a torrent whose recheck
had never actually run at the new location. Now also requires the
tracked filenames to match the destination manifest.
- setLocation/setDownloadPath verification now polls briefly instead
of checking once immediately - both are asynchronous and a single
immediate check can read stale data (this exact bug false-failed
a real run during testing).
- Fixed manifest generation to use printf instead of `stat -c`'s own
\t escape handling, which silently emitted a literal backslash-t
instead of a real tab and broke `cut -f1` pairing.
Verified end-to-end against a real torrent (Babylon 5 S04, 69GB):
clean single-pass recheck, post-check confirmed all 22 files intact.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Babylon 5 S03/S04 (44 episodes) were destroyed during a relink
attempt despite every safety check the script performs (HTTP status,
save_path, content_path) passing. qBittorrent's own automatic
incomplete-file management took further unrequested action after
the script's calls succeeded - moving/truncating files at a
location it still considered incomplete - which none of the
script's checks could see coming since they only verify immediately
after its own calls.
No backup existed. Files could not be found anywhere on /data after
a full search. Marking qbt-relink.sh unsafe until the interaction
with qBittorrent's temp_path_enabled behavior is understood well
enough to prevent this.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
in qbt-relink.sh
Second live incident in the same batch: setLocation genuinely
succeeded and save_path updated correctly, but content_path (what
recheck actually reads) stayed pointed at qBittorrent's incomplete
staging path via a leftover per-torrent download_path override from
an earlier failed attempt. setLocation doesn't clear that override.
Fixed by also calling torrents/setDownloadPath (note: takes "id",
not "hashes") and verifying content_path directly before proceeding.
Also replaced alphabetical-sort file pairing with SxxEyy-parsed
matching, since sort order silently breaks on non-zero-padded
episode numbers (E9 sorts after E10) - a real risk across the
~320 remaining folders with inconsistent naming conventions.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Migrated shows default to unmonitored/no-search since they're
back-catalog imports. For a show still marked "continuing" on TVDB
(e.g. 3 Body Problem), this flips it (and all its seasons) to
monitored so new episodes flow through the automated tv-sonarr
pipeline instead of needing a manual scan/import each time one
drops. --search optionally triggers an immediate missing-episode
search.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
let a torrent start re-downloading
Batch-relinking 9 torrents, one setLocation call silently failed
(status ignored) while the script declared success and moved on.
Its recheck then ran against the original (now-empty) path, found
0% match, and qBittorrent started re-downloading the whole torrent
from scratch into its incomplete-files staging area. No lasting
harm (separate path from the real hardlinked copy, cleaned up), but
caught only by watching qBittorrent directly, not by anything the
script reported.
Fixes: every mutating call now goes through an api_call() helper
that checks the HTTP status and aborts on failure; the torrent is
stopped before any location/rename calls (qBittorrent 5.x renamed
pause/resume to stop/start) and left stopped after recheck rather
than auto-resuming, so a bad relink can never turn into an active
download. setLocation's effect is also verified via a follow-up
GET before proceeding to renameFile.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Sonarr's hardlink-import removes files from their original
torrent-named folder, but qBittorrent's own records still point
there - the next recheck or peer request flips those downloads to
"missing files". qbt-relink.sh re-points a torrent's location and
per-file names at the Sonarr-organized destination and triggers a
recheck, so seeding continues against the same underlying data
(same inode) instead of erroring out or needing a re-download.
Documented as a required step in the sonarr skill's per-show
migration workflow.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
claude-homelab's Sonarr skill only covers search/add/remove, not the
manual-import workflow the Jellyfin migration depends on. Wraps
lookup/add/scan/import into a script matching this repo's
portainer.sh conventions, replacing repetitive raw curl calls.
Also documents a naming-token gotcha hit while migrating the first
two pilot shows: Sonarr's series folder format needs the combined
{Series TitleYear} token, not {Series Title} ({Year}) - the latter
silently drops the year.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Coordinates with qBittorrent to auto-rename/organize new TV downloads
into the Show Name (Year)/Season NN/... layout Jellyfin needs, and
provides the same import engine for migrating the existing library.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Enables an nginx-proxy-manager proxy host (jellyfin:8096) so it can be
reached by hostname on the LAN instead of IP:port. No public
VIRTUAL_HOST/LETSENCRYPT_HOST since this stays internal-only for now.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Roku-accessible on the LAN via direct IP:8096; nginx proxy setup held
off for now. Reuses the existing nas_media volume and GPU passthrough
pattern from plex/docker-compose.yaml.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
find -maxdepth 3 couldn't reach immich's thumbs/<uuid>/XX/YY/file (4
levels deep), so it exhausted the 256x256 hash-bucket fan-out with no
match and timed out on every run -- 3 spurious remounts/restarts of
immich_server within 15 minutes of deploy, all falsely alerted as
staleness. Drop -maxdepth entirely; -quit already stops at the first
match via depth-first search, so it's fast regardless of tree depth.
Stale NFS handles inside containers (ESTALE/errno -116) can persist even
after the host-side mount looks healthy, requiring a manual remount +
container restart. Adds systemd mount-ready hooks (BindsTo) to restart
immich_server/calibre/ocis when their NFS mount unit restarts, plus a
5-minute health-check timer that does a real nested read and force-remounts
on staleness. Host files backed up under system-config/nfs-self-heal/ so
they can be reinstalled after a host rebuild.
Confirmed 5 false positives over 2026-08-20 to 2026-08-25, all connecting
to ::1:8888 at ~21:3x daily - never a real mining-pool destination since
that's loopback, and every attempt failed instantly (nothing listens on
8888). Couldn't identify the source process (falco loses metadata for it
before it can be captured), but a real miner pool can never be localhost,
so excluding loopback fixes the false-positive class without weakening
the rule for actual remote pool connections.
This alarm was originally the crude proxy for catching cryptominers, but
Falco now detects that directly via process/network behavior. It was
firing several times a day from Ollama's legitimate sustained CPU use
(subwave-controller driving chat completions), with no way to tell that
apart from a real problem using aggregate CPU % alone. Routed to netdata's
"silent" role so it stays visible on the dashboard but stops paging
Telegram/Cloud email.
Investigated recent Telegram alerts - no intrusion, both new hits trace to
benign one-offs: calibre's linuxserver.io s6-init NOPASSWD setup (same
pattern already whitelisted for obsidian) and lsyncd truncating its own
status log (not evidence-clearing).
Deploys a second Technitium instance on the container host (192.168.1.67)
to pair with the existing Pi-based instance at dns02 (192.168.1.45) via
native Technitium v14+ clustering for DNS redundancy.
poprhythm