Rewrite qbt-relink.sh with a real safety model, re-enable it

Root cause of the data-loss incident: repeated manual re-triggering
(stop/setLocation/recheck called multiple times across separate
debugging invocations) raced against qBittorrent's own automatic
incomplete-file management. The 7 clean successes earlier all
completed in one uninterrupted pass; the 2 that failed were the ones
manually re-triggered while investigating.

Redesign: the whole operation is now one blocking pass (stop ->
relocate -> rename -> recheck -> poll to completion -> verify ->
report) with a hard rule never to call recheck/stop/setLocation
against the same hash a second time while a previous run might
still be settling. Adds:
- A pre-flight filesystem manifest (name+size) of the destination,
  and a post-recheck comparison against it - the real ground-truth
  safety net, independent of trusting qBittorrent's self-reported
  state.
- Idempotency: skips already-correctly-relinked torrents rather
  than re-touching them. Found and fixed a real gap here during
  testing - the first version trusted qBittorrent's per-file
  "progress" alone, which can be stale (cached from before a move)
  and produced a false "already good" on a torrent whose recheck
  had never actually run at the new location. Now also requires the
  tracked filenames to match the destination manifest.
- setLocation/setDownloadPath verification now polls briefly instead
  of checking once immediately - both are asynchronous and a single
  immediate check can read stale data (this exact bug false-failed
  a real run during testing).
- Fixed manifest generation to use printf instead of `stat -c`'s own
  \t escape handling, which silently emitted a literal backslash-t
  instead of a real tab and broke `cut -f1` pairing.

Verified end-to-end against a real torrent (Babylon 5 S04, 69GB):
clean single-pass recheck, post-check confirmed all 22 files intact.

Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA

qbt-relink.sh

@@ -23,11 +23,37 @@
 # especially for anything with specials/extras where a torrent's files span
 # more than one destination folder (this script only handles one folder;
 # see the sonarr skill doc for that case).
+#
+# SAFETY MODEL (rewritten after a real data-loss incident - see the sonarr
+# skill doc, "Incident 3"):
+#
+# - The whole operation runs as ONE blocking pass: stop -> relocate -> rename
+#   -> recheck -> poll to completion -> verify -> report. It does NOT return
+#   control with "poll manually and come back" - that pattern is exactly what
+#   caused the incident, because re-running recheck/stop by hand later, after
+#   qBittorrent's own state had moved on, raced against qBittorrent's
+#   automatic incomplete-file management and let it destroy real files.
+#   NEVER call recheck/stop/setLocation against the same hash a second time
+#   while a previous invocation might still be settling - wait for this
+#   script to finish (or clearly fail) first.
+# - Before touching qBittorrent at all, it snapshots the destination folder's
+#   filenames+sizes (the ground truth - Sonarr already put the real files
+#   there). After the recheck settles, it re-reads the folder and compares.
+#   Any file that shrank, vanished, or changed size is treated as data loss,
+#   reported loudly, and the script does NOT attempt any further remediation
+#   (no retry, no re-trigger) - investigate by hand from a known-bad state
+#   rather than risk compounding it.
+# - If the torrent already looks correctly relinked (content_path matches and
+#   on-disk files match the expected sizes), the script does nothing further
+#   - idempotent, so re-running it after a partial failure is safe rather
+#   than repeating destructive steps.
 
 set -euo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="$HOME/.claude-homelab/.env"
+POLL_INTERVAL=15
+POLL_TIMEOUT_S=2400  # 40 min - generous for large multi-GB files over NFS
 
 if [[ ! -f "$ENV_FILE" ]]; then
     echo "Error: $ENV_FILE not found (qBittorrent creds live there, not this repo's .credentials)" >&2
@@ -60,10 +86,6 @@ if [[ "$login_status" != "200" ]]; then
 fi
 
 # api_call <label> <path> [curl data args...]
-# Every write call goes through this - checks the HTTP status explicitly
-# instead of trusting a silent curl call, which is exactly how a stalled
-# torrent went back to actively downloading (a failed setLocation was never
-# checked, so the script declared success and moved on anyway).
 api_call() {
     local label="$1" path="$2"
     shift 2
@@ -75,17 +97,102 @@ api_call() {
     fi
 }
 
-# Stop the torrent FIRST, before touching its location/files at all. A
+torrent_info() {
-# recheck runs fine on a stopped torrent, and this guarantees qBittorrent
+    curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH"
-# can never decide to start downloading missing pieces mid-relink - the
+}
-# failure mode that hit Babylon 5 S05. (qBittorrent 5.x renamed pause/resume
+
-# to stop/start; the old /pause endpoint 404s silently on this version.)
+# manifest_of <folder> -> "name\tsize" lines, sorted by name
-echo "Stopping torrent (safety - no download can start while relinking)..."
+# Uses printf, not `stat -c '%n\t%s'` - stat's own \t escape handling isn't
+# reliable across environments and silently produced a literal backslash-t
+# instead of a real tab the first time this was tried, breaking `cut -f1`.
+manifest_of() {
+    docker exec jellyfin sh -c "cd \"$1\" && for f in *; do [ -f \"\$f\" ] && printf '%s\t%s\n' \"\$f\" \"\$(stat -c%s \"\$f\")\"; done" 2>/dev/null | sort
+}
+
+# wait_for_field <jq-ish python expr on the info dict> <expected-value> <label>
+# setLocation/setDownloadPath are asynchronous - qBittorrent returns 200
+# before the internal move actually completes, so checking immediately can
+# see stale data. Poll briefly instead of a single immediate check.
+wait_for_path_field() {
+    local field="$1" expected="$2" match_mode="$3" label="$4"
+    local attempt actual
+    for attempt in $(seq 1 15); do
+        actual=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['$field'])")
+        if [[ "$match_mode" == "prefix" && "$actual" == "$expected"* ]]; then
+            echo "$actual"
+            return 0
+        elif [[ "$match_mode" == "exact" && "$actual" == "$expected" ]]; then
+            echo "$actual"
+            return 0
+        fi
+        sleep 1
+    done
+    echo "$actual"
+    return 1
+}
+
+echo "=== Pre-flight ==="
+if ! docker exec jellyfin sh -c "[ -d \"$NEW_FOLDER\" ]"; then
+    echo "Error: destination folder does not exist: $NEW_FOLDER" >&2
+    exit 1
+fi
+baseline_manifest=$(manifest_of "$NEW_FOLDER")
+baseline_count=$(echo "$baseline_manifest" | grep -c . || true)
+if [[ "$baseline_count" -eq 0 ]]; then
+    echo "Error: destination folder is empty: $NEW_FOLDER" >&2
+    exit 1
+fi
+echo "Destination has $baseline_count file(s) - recorded as ground truth."
+
+# --- Idempotency check: is this already correctly relinked? ---
+BASELINE_MANIFEST_FILE=$(mktemp)
+echo "$baseline_manifest" > "$BASELINE_MANIFEST_FILE"
+trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE"' EXIT
+
+current_info=$(torrent_info)
+current_content_path=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['content_path'])")
+if [[ "$current_content_path" == "$NEW_FOLDER"* ]]; then
+    echo "content_path already points at $NEW_FOLDER - checking if fully verified..."
+    current_progress=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['progress'])")
+    # Compare current qBittorrent-tracked file NAMES (not just count) and
+    # progress to the manifest. Progress alone is not enough to trust - it
+    # can be stale, cached from before a move, if a prior attempt was
+    # interrupted before ever triggering a fresh recheck at the new
+    # location (this happened for real: a torrent showed progress 1 for
+    # files under a subfolder that had already stopped existing on disk).
+    # Requiring the basename to appear in the manifest means renameFile
+    # must have actually already run - a stale-but-unrenamed torrent can no
+    # longer produce a false "already good".
+    already_good=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH" | python3 -c "
+import json, os, sys
+files = json.load(sys.stdin)
+manifest = {}
+with open(sys.argv[1]) as f:
+    for line in f:
+        line = line.rstrip(chr(10))
+        if not line: continue
+        name, size = line.rsplit(chr(9), 1)
+        manifest[name] = int(size)
+if len(files) != len(manifest):
+    print('no')
+else:
+    names_match = all(os.path.basename(f['name']) in manifest for f in files)
+    progress_ok = all(f.get('progress', 0) >= 0.999 for f in files)
+    print('yes' if (names_match and progress_ok) else 'no')
+" "$BASELINE_MANIFEST_FILE" 2>/dev/null || echo "no")
+    if [[ "$already_good" == "yes" ]]; then
+        echo "Already relinked and verified (progress ~$current_progress). Nothing to do."
+        exit 0
+    fi
+    echo "Not yet fully verified (progress: $current_progress) - proceeding carefully."
+fi
+
+# --- Stop the torrent before touching anything ---
+echo
+echo "=== Stopping torrent ==="
 api_call "stop" "torrents/stop" --data-urlencode "hashes=$HASH"
 
-# Current files inside the torrent (relative paths, in qBittorrent's index order)
 old_files_json=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH")
-
 torrent_name=$(python3 -c "
 import json,sys
 files = json.loads(sys.argv[1])
@@ -94,16 +201,13 @@ if not files:
     sys.exit(1)
 print(files[0]['name'].split('/')[0])
 " "$old_files_json")
-
 echo "Torrent folder: $torrent_name"
 echo "New folder:     $NEW_FOLDER"
 echo
 
-# New files actually on disk at the destination (via any container with the nas_media mount)
+new_files=$(echo "$baseline_manifest" | cut -f1)
-new_files=$(docker exec jellyfin sh -c "ls -1 \"$NEW_FOLDER\"" | sort)
 
-# Build the old->new pairing, preferring an explicit SxxEyy match over sort
+echo "=== Computing file pairing ==="
-# order (see header comment for why sort order alone is unsafe).
 python3 -c "
 import json, re, sys
 
@@ -131,8 +235,7 @@ if all(old_keys.values()) and all(new_keys.values()) and len(set(old_keys.values
     pairs = [(old_by_key[k], new_by_key[k]) for k in sorted(common)]
 else:
     print('WARNING: could not parse a unique SxxEyy from every filename - falling back to alphabetical-sort pairing.', file=sys.stderr)
-    print('         Double-check the pairing below carefully before it applies (this is exactly the failure mode', file=sys.stderr)
+    print('         Double-check the pairing below carefully before it applies.', file=sys.stderr)
-    print('         that silently mismatches non-zero-padded episode numbers like E9 vs E10).', file=sys.stderr)
     pairs = list(zip(sorted(old_keys), sorted(new_keys)))
 
 with open('/tmp/qbt_relink_flat.tsv', 'w') as f:
@@ -143,54 +246,33 @@ with open('/tmp/qbt_relink_flat.tsv', 'w') as f:
 " "$old_files_json" "$new_files"
 
 echo
-echo "--- Applying ---"
+echo "=== Applying (single pass - stop/relocate/rename/recheck, no retries) ==="
 
 echo "setLocation -> $NEW_FOLDER"
 api_call "setLocation" "torrents/setLocation" \
     --data-urlencode "hashes=$HASH" \
     --data-urlencode "location=$NEW_FOLDER"
 
-# Don't just trust the 200 - confirm the torrent's save_path actually changed
+if ! actual_path=$(wait_for_path_field "save_path" "$NEW_FOLDER" exact "save_path"); then
-# before touching anything else. This is exactly the check that was missing
+    echo "Error: setLocation did not take effect after 15s (save_path is still '$actual_path'). Torrent is stopped - not proceeding." >&2
-# when Babylon 5 S05 silently kept its old location and started re-downloading.
-actual_path=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH" \
-    | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['save_path'])")
-if [[ "$actual_path" != "$NEW_FOLDER" ]]; then
-    echo "Error: setLocation did not take effect (save_path is still '$actual_path'). Torrent is stopped - not proceeding." >&2
     exit 1
 fi
 echo "  confirmed: save_path is now $actual_path"
 
-# If qBittorrent's "Keep incomplete torrents in" feature (temp_path_enabled)
+echo "setDownloadPath -> $NEW_FOLDER (clears any stale incomplete-path override)"
-# is on, and this torrent was ever flagged as needing to download (e.g. an
-# earlier relink attempt failed and its recheck found 0% at the wrong path),
-# it can carry a separate download_path override that content_path actually
-# follows instead of save_path - setLocation alone does NOT clear this. This
-# bit Babylon 5 S03/S04/S05 in the same incident: recheck kept running
-# against the stale incomplete-staging path even after setLocation reported
-# success. Note: this endpoint takes "id", not "hashes" (undocumented
-# inconsistency vs. every other torrents/* endpoint here).
-echo "setDownloadPath -> $NEW_FOLDER (in case a stale download_path override exists)"
 status=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" -X POST "$QBITTORRENT_URL/api/v2/torrents/setDownloadPath" \
     --data-urlencode "id=$HASH" --data-urlencode "path=$NEW_FOLDER")
-# 400 here just means the torrent has no download_path override to clear
-# (the common case) - only treat other failures as fatal.
 if [[ "$status" != "200" && "$status" != "400" ]]; then
     echo "Error: setDownloadPath failed (http $status)" >&2
     exit 1
 fi
 
-content_path=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH" \
+if ! content_path=$(wait_for_path_field "content_path" "$NEW_FOLDER" prefix "content_path"); then
-    | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['content_path'])")
+    echo "Error: content_path is '$content_path', not under '$NEW_FOLDER' after 15s. Not proceeding." >&2
-if [[ "$content_path" != "$NEW_FOLDER"* ]]; then
-    echo "Error: content_path is '$content_path', not under '$NEW_FOLDER'. A recheck would silently fail against the wrong path - not proceeding." >&2
     exit 1
 fi
 echo "  confirmed: content_path is now $content_path"
 
-# setLocation above already moved the torrent's root to $NEW_FOLDER itself
-# (the season folder) - so new paths here are bare filenames, not prefixed
-# with the season folder name again.
 while IFS=$'\t' read -r old new; do
     api_call "renameFile" "torrents/renameFile" \
         --data-urlencode "hash=$HASH" \
@@ -199,12 +281,77 @@ while IFS=$'\t' read -r old new; do
     echo "  renamed: $old -> $new"
 done < /tmp/qbt_relink_flat.tsv
 
-echo "recheck..."
+echo
+echo "=== Rechecking (recheck triggered exactly once) ==="
 api_call "recheck" "torrents/recheck" --data-urlencode "hashes=$HASH"
 
-echo "Done. Torrent is left STOPPED so nothing can auto-resume downloading"
+elapsed=0
-echo "before you've verified the recheck passed. Poll with:"
+final_state=""
+while [[ "$elapsed" -lt "$POLL_TIMEOUT_S" ]]; do
+    sleep "$POLL_INTERVAL"
+    elapsed=$((elapsed + POLL_INTERVAL))
+    state=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['state'])")
+    if [[ "$state" != "checkingUP" && "$state" != "checkingDL" && "$state" != "checkingResumeData" ]]; then
+        final_state="$state"
+        break
+    fi
+    echo "  still checking (${elapsed}s elapsed): $state"
+done
+
+if [[ -z "$final_state" ]]; then
+    echo
+    echo "TIMED OUT after ${POLL_TIMEOUT_S}s still checking. DO NOT re-trigger recheck or resume by hand -" >&2
+    echo "that repeated-intervention pattern is what caused the data-loss incident. Instead re-run this" >&2
+    echo "exact script invocation again later (it's idempotent) or investigate read-only first." >&2
+    exit 1
+fi
+echo "  settled: $final_state"
+
+echo
+echo "=== Post-check: verifying files against the pre-flight manifest ==="
+FINAL_MANIFEST_FILE=$(mktemp)
+manifest_of "$NEW_FOLDER" > "$FINAL_MANIFEST_FILE"
+trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE"' EXIT
+
+mismatch=$(python3 -c "
+import sys
+
+def load(path):
+    d = {}
+    with open(path) as f:
+        for line in f:
+            line = line.rstrip(chr(10))
+            if not line:
+                continue
+            name, size = line.rsplit(chr(9), 1)
+            d[name] = size
+    return d
+
+baseline = load(sys.argv[1])
+final = load(sys.argv[2])
+problems = []
+for name, size in baseline.items():
+    if name not in final:
+        problems.append(f'MISSING: {name}')
+    elif final[name] != size:
+        problems.append(f'SIZE CHANGED: {name} ({size} -> {final[name]})')
+for name in final:
+    if name not in baseline:
+        problems.append(f'UNEXPECTED NEW FILE: {name}')
+print(chr(10).join(problems))
+" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE")
+
+if [[ -n "$mismatch" ]]; then
+    echo "CRITICAL: files at $NEW_FOLDER changed during relink - possible data loss:" >&2
+    echo "$mismatch" >&2
+    echo >&2
+    echo "The torrent is left STOPPED. Do NOT re-run this script or touch qBittorrent further" >&2
+    echo "until this is investigated by hand - re-running blind is what caused the original incident." >&2
+    exit 1
+fi
+
+echo "All $baseline_count file(s) match the pre-flight manifest - no data loss detected."
+echo
+echo "Done. Torrent is left STOPPED on purpose. Verify state before starting it:"
 echo "  bash ~/.claude/plugins/cache/claude-homelab/homelab-core/*/skills/qbittorrent/scripts/qbit-api.sh info $HASH"
-echo "Once it shows 100% progress / stoppedUP (not stoppedDL - stoppedDL means the"
+echo "Then start it from the qBittorrent WebUI once you're satisfied."
-echo "recheck found missing/mismatched pieces, do not resume), start it again from"
-echo "the qBittorrent WebUI (this script does not auto-resume, on purpose)."