api_post/api_put passed the JSON payload directly as a curl -d
command-line argument, which hit the OS ARG_MAX limit importing
Jeopardy's 1269-file library (a large date-based show is exactly
the case where a batch import payload gets big). Now written to a
temp file and passed via curl's -d @file instead.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Reflects the rewritten script's blocking single-pass model and the
hard rule against re-triggering stop/setLocation/recheck against the
same hash more than once. Marks Incident 3 as fixed and verified.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Root cause of the data-loss incident: repeated manual re-triggering
(stop/setLocation/recheck called multiple times across separate
debugging invocations) raced against qBittorrent's own automatic
incomplete-file management. The 7 clean successes earlier all
completed in one uninterrupted pass; the 2 that failed were the ones
manually re-triggered while investigating.
Redesign: the whole operation is now one blocking pass (stop ->
relocate -> rename -> recheck -> poll to completion -> verify ->
report) with a hard rule never to call recheck/stop/setLocation
against the same hash a second time while a previous run might
still be settling. Adds:
- A pre-flight filesystem manifest (name+size) of the destination,
and a post-recheck comparison against it - the real ground-truth
safety net, independent of trusting qBittorrent's self-reported
state.
- Idempotency: skips already-correctly-relinked torrents rather
than re-touching them. Found and fixed a real gap here during
testing - the first version trusted qBittorrent's per-file
"progress" alone, which can be stale (cached from before a move)
and produced a false "already good" on a torrent whose recheck
had never actually run at the new location. Now also requires the
tracked filenames to match the destination manifest.
- setLocation/setDownloadPath verification now polls briefly instead
of checking once immediately - both are asynchronous and a single
immediate check can read stale data (this exact bug false-failed
a real run during testing).
- Fixed manifest generation to use printf instead of `stat -c`'s own
\t escape handling, which silently emitted a literal backslash-t
instead of a real tab and broke `cut -f1` pairing.
Verified end-to-end against a real torrent (Babylon 5 S04, 69GB):
clean single-pass recheck, post-check confirmed all 22 files intact.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Babylon 5 S03/S04 (44 episodes) were destroyed during a relink
attempt despite every safety check the script performs (HTTP status,
save_path, content_path) passing. qBittorrent's own automatic
incomplete-file management took further unrequested action after
the script's calls succeeded - moving/truncating files at a
location it still considered incomplete - which none of the
script's checks could see coming since they only verify immediately
after its own calls.
No backup existed. Files could not be found anywhere on /data after
a full search. Marking qbt-relink.sh unsafe until the interaction
with qBittorrent's temp_path_enabled behavior is understood well
enough to prevent this.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
in qbt-relink.sh
Second live incident in the same batch: setLocation genuinely
succeeded and save_path updated correctly, but content_path (what
recheck actually reads) stayed pointed at qBittorrent's incomplete
staging path via a leftover per-torrent download_path override from
an earlier failed attempt. setLocation doesn't clear that override.
Fixed by also calling torrents/setDownloadPath (note: takes "id",
not "hashes") and verifying content_path directly before proceeding.
Also replaced alphabetical-sort file pairing with SxxEyy-parsed
matching, since sort order silently breaks on non-zero-padded
episode numbers (E9 sorts after E10) - a real risk across the
~320 remaining folders with inconsistent naming conventions.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Migrated shows default to unmonitored/no-search since they're
back-catalog imports. For a show still marked "continuing" on TVDB
(e.g. 3 Body Problem), this flips it (and all its seasons) to
monitored so new episodes flow through the automated tv-sonarr
pipeline instead of needing a manual scan/import each time one
drops. --search optionally triggers an immediate missing-episode
search.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
let a torrent start re-downloading
Batch-relinking 9 torrents, one setLocation call silently failed
(status ignored) while the script declared success and moved on.
Its recheck then ran against the original (now-empty) path, found
0% match, and qBittorrent started re-downloading the whole torrent
from scratch into its incomplete-files staging area. No lasting
harm (separate path from the real hardlinked copy, cleaned up), but
caught only by watching qBittorrent directly, not by anything the
script reported.
Fixes: every mutating call now goes through an api_call() helper
that checks the HTTP status and aborts on failure; the torrent is
stopped before any location/rename calls (qBittorrent 5.x renamed
pause/resume to stop/start) and left stopped after recheck rather
than auto-resuming, so a bad relink can never turn into an active
download. setLocation's effect is also verified via a follow-up
GET before proceeding to renameFile.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Sonarr's hardlink-import removes files from their original
torrent-named folder, but qBittorrent's own records still point
there - the next recheck or peer request flips those downloads to
"missing files". qbt-relink.sh re-points a torrent's location and
per-file names at the Sonarr-organized destination and triggers a
recheck, so seeding continues against the same underlying data
(same inode) instead of erroring out or needing a re-download.
Documented as a required step in the sonarr skill's per-show
migration workflow.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
claude-homelab's Sonarr skill only covers search/add/remove, not the
manual-import workflow the Jellyfin migration depends on. Wraps
lookup/add/scan/import into a script matching this repo's
portainer.sh conventions, replacing repetitive raw curl calls.
Also documents a naming-token gotcha hit while migrating the first
two pilot shows: Sonarr's series folder format needs the combined
{Series TitleYear} token, not {Series Title} ({Year}) - the latter
silently drops the year.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Coordinates with qBittorrent to auto-rename/organize new TV downloads
into the Show Name (Year)/Season NN/... layout Jellyfin needs, and
provides the same import engine for migrating the existing library.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Enables an nginx-proxy-manager proxy host (jellyfin:8096) so it can be
reached by hostname on the LAN instead of IP:port. No public
VIRTUAL_HOST/LETSENCRYPT_HOST since this stays internal-only for now.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
Roku-accessible on the LAN via direct IP:8096; nginx proxy setup held
off for now. Reuses the existing nas_media volume and GPU passthrough
pattern from plex/docker-compose.yaml.
Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
find -maxdepth 3 couldn't reach immich's thumbs/<uuid>/XX/YY/file (4
levels deep), so it exhausted the 256x256 hash-bucket fan-out with no
match and timed out on every run -- 3 spurious remounts/restarts of
immich_server within 15 minutes of deploy, all falsely alerted as
staleness. Drop -maxdepth entirely; -quit already stops at the first
match via depth-first search, so it's fast regardless of tree depth.
Stale NFS handles inside containers (ESTALE/errno -116) can persist even
after the host-side mount looks healthy, requiring a manual remount +
container restart. Adds systemd mount-ready hooks (BindsTo) to restart
immich_server/calibre/ocis when their NFS mount unit restarts, plus a
5-minute health-check timer that does a real nested read and force-remounts
on staleness. Host files backed up under system-config/nfs-self-heal/ so
they can be reinstalled after a host rebuild.
Confirmed 5 false positives over 2026-08-20 to 2026-08-25, all connecting
to ::1:8888 at ~21:3x daily - never a real mining-pool destination since
that's loopback, and every attempt failed instantly (nothing listens on
8888). Couldn't identify the source process (falco loses metadata for it
before it can be captured), but a real miner pool can never be localhost,
so excluding loopback fixes the false-positive class without weakening
the rule for actual remote pool connections.
This alarm was originally the crude proxy for catching cryptominers, but
Falco now detects that directly via process/network behavior. It was
firing several times a day from Ollama's legitimate sustained CPU use
(subwave-controller driving chat completions), with no way to tell that
apart from a real problem using aggregate CPU % alone. Routed to netdata's
"silent" role so it stays visible on the dashboard but stops paging
Telegram/Cloud email.
Investigated recent Telegram alerts - no intrusion, both new hits trace to
benign one-offs: calibre's linuxserver.io s6-init NOPASSWD setup (same
pattern already whitelisted for obsidian) and lsyncd truncating its own
status log (not evidence-clearing).
Deploys a second Technitium instance on the container host (192.168.1.67)
to pair with the existing Pi-based instance at dns02 (192.168.1.45) via
native Technitium v14+ clustering for DNS redundancy.
The rule checked proc.name against gitea's managed hook names, but git's
hook dispatcher always execs these as `bash ./hooks/<hookname>.d/gitea` -
proc.name is "bash" (the interpreter), never the hook name. That check
could never match, so this fired CRITICAL 3x (pre-receive/update/
post-receive) on every single push since it was added - including two
notifications the user got moments ago from this repo's own commits.
Fixed to check the actual invoked script path in proc.cmdline instead.
This commit's own push is the live verification.
"Redirect STDOUT/STDIN to Network Connection in Container" (a reverse-shell
detector) was firing legitimately: gitea's sshd/sshd-session dup2 the
accepted SSH socket onto stdio for every session (4x per connection), and
firefly-iii's wait-for-it.sh does the same TCP-readiness-check dance while
waiting for postgres. Both confirmed recurring via netdata's alert history,
not one-off. Scoped exceptions added to each's specific binary/cmdline,
not the whole container/image.
Verified: a gitea SSH login no longer alerts, while a real dup2-based
redirect (bash's /dev/tcp exec pattern) from an unrelated container still
fires - the exception is narrow, not a blanket disable.
"Run shell untrusted" was also flagged as noisy, but investigation showed
it fired exactly once, during my own rule-testing window, and never
before or since - left alone rather than building a permanent exception
for a self-caused test artifact.
Cron script (every 5m) pulls falco_rule_match raise transitions from
netdata's existing alert_transitions API - the same data already driving
the Telegram pipeline - and writes them as a Dashy custom-list JSON file.
No new containers/Redis/Falcosidekick needed for a quick-glance security
event feed on the dashboard.
conf.yml and the crontab entry live outside the repo (per existing
pattern) - conf.yml got a new custom-list widget block in the System
Widgets section, and crontab got a matching */5 entry alongside the
other host-cron jobs.
Found during live testing: netdata's cloud-provider auto-detection runs
curl --fail -s -m1 --noproxy * http://169.254.169.254 on every
startup/reconnect to check whether the host is in AWS/GCP/Azure - standard
monitoring-agent behavior, not a bug. Without this exception the rule
would have paged every time netdata restarts. Scoped to netdata's own
curl specifically so any other connection from that container still
alerts.
git-hook-tamper.yaml's condition used proc.exepath, which resolves to the
script interpreter's path (e.g. /bin/busybox) for shebang scripts, not the
script's own path - switched to proc.cmdline, which retains the originally
invoked path. Confirmed via live-testing both ways.
ssh-persistence/cloud-metadata-probe/db-spawned-process round out the
post-incident hardening pass with a few more incubating-ruleset adaptations.
Written after the 2026-08-10/11 gitea internal-API log-poisoning attack
that planted a malicious uploadpack.packObjectsHook backdoor. Catches
both the planting (unexpected exec from a hooks/ path) and the firing
(git-upload-pack/git-receive-pack spawning anything but its own
pack-objects binary), independent of how the hook config got written.
No repeat intrusion - investigated all Falco alerts since deployment
(2026-08-08 to 2026-08-16); the only crypto-miner rule hits were the
original test alerts. The daily self-resolving Telegram alerts the user
was seeing came from four known-benign sources: Obsidian's Electron
self-re-exec pattern and s6-init sudoers setup (fires ~daily on container
recreation), ttyd/Synchronet's telnet/pv/busybox-extras BBS tooling, and
ollama's nvidia-ctk ldconfig-refresh hook (memfd_create by design). Added
scoped exceptions for each via Falco's own rule extension points, same
pattern as the earlier pg_isready/gitea fixes.
Also fixed netdata's used_swap alarm, which recalculated from the
instantaneous raw sample every 10s with only a 30s notification debounce -
a brief swap spike was enough to page both Telegram and (via Netdata
Cloud) email. Widened the up-delay to 5m so only sustained swap pressure
notifies.
Verified end-to-end via a fresh fake-xmrig test after redeploying Falco
and reloading netdata: CRITICAL alarm fired and Telegram delivery
succeeded.
- 'Drop and execute new binary in container' fired on every git push/pull
to gitea over SSH (gitea's own binary re-executing itself for git-shell
hooks looks like container drift). Scoped exception via
known_drop_and_execute_activities to proc.name=gitea on that specific
image, not a blanket container whitelist, so other unexpected binaries
in that container still get caught.
- netdata alarm text was a static generic blurb requiring a manual
'docker logs falco' every time to find out what actually happened.
Now interpolates ${label:rule_name}/${label:priority} (exposed by
Falco's Prometheus metric) so the Telegram message names the specific
rule directly. Note: literal double-quotes in the info/summary text
broke netdata's alarm-notify.sh (silent delivery failure, exit 1) -
avoided.
- Added netdata/health.d and netdata/go.d to the repo for documentation;
netdata does not auto-deploy these from git, same as gitea's app.ini -
copy to /srv/netdata/config/ manually and restart the container.
Navidrome indexes the existing music library on nas_media for Subsonic
access. SUB/WAVE (byo-proxy variant, LAN-only) provides the Icecast
broadcast, Liquidsoap crossfade/ducking, and AI DJ controller, reusing
the existing Ollama instance over npm-network for the LLM and Piper
for TTS.
Detects unauthorized miners (and other runtime threats) via Falco's
syscall/eBPF monitoring, following the 2026-08-06 gitea/xmrig
compromise. Wired into netdata rather than a new notification channel:
Falco exposes a Prometheus /metrics endpoint, netdata scrapes it, and a
new health.d alarm pages through netdata's already-configured Telegram
bot - no new alerting infra needed.
Includes a custom process-name rule for known miner binaries (the stock
Stratum-protocol rule wouldn't have caught the actual gitea incident,
which used a bare host:port with no scheme prefix), an outbound
miner-pool-port rule as a second layer, and rule_matching: all in the
Falco config - without it, Falco silently drops all but the first
matching rule per event, which would have suppressed our custom rule
whenever a stock rule also matched the same process.
Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until
found manually — there was no alerting on sustained host CPU. Wired
netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram
bot (config is host-only, not committed). Also narrowed the dashboard port
from 0.0.0.0 to the host's LAN IP to reduce exposure.
Same category of risk as the gitea :nightly issue: floating dev/beta tags
let watchtower silently pull unvetted upstream builds. couchdb had no tag
at all (implicit :latest); filebrowser and statping only publish
beta/dev channels upstream so pinned to the exact version in use rather
than a moving target.
Pin image to digest instead of :nightly (DB schema already ahead of
stable 1.27, downgrading would break startup) and disable open
self-registration/OpenID signup, which attackers used to plant xmrig
via a git hook.
- quizmaster: removed entirely (test service, no longer used); also removed Portainer stack and /srv/quizmaster data
- leafweb: removed orphaned MSSQL compose file (no running container, no Portainer stack, no data on disk)
- authelia, gitea, couchdb, portainer, immich (server + ml): add com.centurylinklabs.watchtower.monitor-only=true so watchtower still notifies on new versions without auto-applying them
- ladder: restore restart: unless-stopped (was commented out)
- rackpeek: add missing TZ env var
- watchtower: move Telegram token/chat ID to .env instead of plaintext in compose
- plex: remove stale/dead PLEX_CLAIM token
- uptime-kuma: mount docker.sock read-only (only needs to read container state)
- home-assistant: drop privileged: true (no device passthrough configured)
- glances: drop privileged: true, bind dashboard port to 127.0.0.1
- couchdb, docker-registry, leafweb: bind ports to 127.0.0.1 (proxy/localhost-only access confirmed, no direct LAN clients)
immich-server auto-updated to v3.0.1 via watchtower but postgres stayed
pinned to the old pgvecto.rs image, causing a crash loop (missing vector
extension). Per official Immich migration guide.
open-webui was sending OAuth credentials via both client_secret_basic
(header) and client_secret_post (body) simultaneously, which Authelia
rejects. Setting OAUTH_TOKEN_ENDPOINT_AUTH_METHOD=client_secret_post
forces a single method and matches the updated Authelia client config.
WEBUI_SECRET_KEY is now a stable env var so watchtower image updates
no longer invalidate all user sessions.
These vhost.d entries were symlinks to a non-existent 'public' file.
Portainer refuses to clone repos containing symlinks, blocking all
git-based stack redeployments. Replace with empty regular files
(equivalent behavior: no extra nginx config = public access).
NFS named volumes go ESTALE when unRAID drives spin down and Docker cannot recover.
systemd automount handles NFS reconnection transparently; Docker just sees a bind mount.
100 arcade titles spanning Pac-Man/Namco classics, Street Fighter,
Mortal Kombat, Metal Slug, King of Fighters, Samurai Shodown,
Fatal Fury, beat-em-ups (TMNT, Simpsons, X-Men), and more.
Also adds IGDB_CLIENT_ID/SECRET env vars for metadata scraping.
32 Game Boy, 36 GBA, and 41 Amiga titles. Amiga includes the full
Lemmings trilogy plus Turrican, Monkey Island, Chaos Engine, Worms,
Sensible Soccer, and more. Note: Amiga requires Kickstart firmware
(not in No-Intro) to actually run.
poprhythm