Rewrite qbt-relink.sh with a real safety model, re-enable it
Root cause of the data-loss incident: repeated manual re-triggering (stop/setLocation/recheck called multiple times across separate debugging invocations) raced against qBittorrent's own automatic incomplete-file management. The 7 clean successes earlier all completed in one uninterrupted pass; the 2 that failed were the ones manually re-triggered while investigating. Redesign: the whole operation is now one blocking pass (stop -> relocate -> rename -> recheck -> poll to completion -> verify -> report) with a hard rule never to call recheck/stop/setLocation against the same hash a second time while a previous run might still be settling. Adds: - A pre-flight filesystem manifest (name+size) of the destination, and a post-recheck comparison against it - the real ground-truth safety net, independent of trusting qBittorrent's self-reported state. - Idempotency: skips already-correctly-relinked torrents rather than re-touching them. Found and fixed a real gap here during testing - the first version trusted qBittorrent's per-file "progress" alone, which can be stale (cached from before a move) and produced a false "already good" on a torrent whose recheck had never actually run at the new location. Now also requires the tracked filenames to match the destination manifest. - setLocation/setDownloadPath verification now polls briefly instead of checking once immediately - both are asynchronous and a single immediate check can read stale data (this exact bug false-failed a real run during testing). - Fixed manifest generation to use printf instead of `stat -c`'s own \t escape handling, which silently emitted a literal backslash-t instead of a real tab and broke `cut -f1` pairing. Verified end-to-end against a real torrent (Babylon 5 S04, 69GB): clean single-pass recheck, post-check confirmed all 22 files intact. Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
This commit is contained in:
+199
-52
@@ -23,11 +23,37 @@
|
||||
# especially for anything with specials/extras where a torrent's files span
|
||||
# more than one destination folder (this script only handles one folder;
|
||||
# see the sonarr skill doc for that case).
|
||||
#
|
||||
# SAFETY MODEL (rewritten after a real data-loss incident - see the sonarr
|
||||
# skill doc, "Incident 3"):
|
||||
#
|
||||
# - The whole operation runs as ONE blocking pass: stop -> relocate -> rename
|
||||
# -> recheck -> poll to completion -> verify -> report. It does NOT return
|
||||
# control with "poll manually and come back" - that pattern is exactly what
|
||||
# caused the incident, because re-running recheck/stop by hand later, after
|
||||
# qBittorrent's own state had moved on, raced against qBittorrent's
|
||||
# automatic incomplete-file management and let it destroy real files.
|
||||
# NEVER call recheck/stop/setLocation against the same hash a second time
|
||||
# while a previous invocation might still be settling - wait for this
|
||||
# script to finish (or clearly fail) first.
|
||||
# - Before touching qBittorrent at all, it snapshots the destination folder's
|
||||
# filenames+sizes (the ground truth - Sonarr already put the real files
|
||||
# there). After the recheck settles, it re-reads the folder and compares.
|
||||
# Any file that shrank, vanished, or changed size is treated as data loss,
|
||||
# reported loudly, and the script does NOT attempt any further remediation
|
||||
# (no retry, no re-trigger) - investigate by hand from a known-bad state
|
||||
# rather than risk compounding it.
|
||||
# - If the torrent already looks correctly relinked (content_path matches and
|
||||
# on-disk files match the expected sizes), the script does nothing further
|
||||
# - idempotent, so re-running it after a partial failure is safe rather
|
||||
# than repeating destructive steps.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="$HOME/.claude-homelab/.env"
|
||||
POLL_INTERVAL=15
|
||||
POLL_TIMEOUT_S=2400 # 40 min - generous for large multi-GB files over NFS
|
||||
|
||||
if [[ ! -f "$ENV_FILE" ]]; then
|
||||
echo "Error: $ENV_FILE not found (qBittorrent creds live there, not this repo's .credentials)" >&2
|
||||
@@ -60,10 +86,6 @@ if [[ "$login_status" != "200" ]]; then
|
||||
fi
|
||||
|
||||
# api_call <label> <path> [curl data args...]
|
||||
# Every write call goes through this - checks the HTTP status explicitly
|
||||
# instead of trusting a silent curl call, which is exactly how a stalled
|
||||
# torrent went back to actively downloading (a failed setLocation was never
|
||||
# checked, so the script declared success and moved on anyway).
|
||||
api_call() {
|
||||
local label="$1" path="$2"
|
||||
shift 2
|
||||
@@ -75,17 +97,102 @@ api_call() {
|
||||
fi
|
||||
}
|
||||
|
||||
# Stop the torrent FIRST, before touching its location/files at all. A
|
||||
# recheck runs fine on a stopped torrent, and this guarantees qBittorrent
|
||||
# can never decide to start downloading missing pieces mid-relink - the
|
||||
# failure mode that hit Babylon 5 S05. (qBittorrent 5.x renamed pause/resume
|
||||
# to stop/start; the old /pause endpoint 404s silently on this version.)
|
||||
echo "Stopping torrent (safety - no download can start while relinking)..."
|
||||
torrent_info() {
|
||||
curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH"
|
||||
}
|
||||
|
||||
# manifest_of <folder> -> "name\tsize" lines, sorted by name
|
||||
# Uses printf, not `stat -c '%n\t%s'` - stat's own \t escape handling isn't
|
||||
# reliable across environments and silently produced a literal backslash-t
|
||||
# instead of a real tab the first time this was tried, breaking `cut -f1`.
|
||||
manifest_of() {
|
||||
docker exec jellyfin sh -c "cd \"$1\" && for f in *; do [ -f \"\$f\" ] && printf '%s\t%s\n' \"\$f\" \"\$(stat -c%s \"\$f\")\"; done" 2>/dev/null | sort
|
||||
}
|
||||
|
||||
# wait_for_field <jq-ish python expr on the info dict> <expected-value> <label>
|
||||
# setLocation/setDownloadPath are asynchronous - qBittorrent returns 200
|
||||
# before the internal move actually completes, so checking immediately can
|
||||
# see stale data. Poll briefly instead of a single immediate check.
|
||||
wait_for_path_field() {
|
||||
local field="$1" expected="$2" match_mode="$3" label="$4"
|
||||
local attempt actual
|
||||
for attempt in $(seq 1 15); do
|
||||
actual=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['$field'])")
|
||||
if [[ "$match_mode" == "prefix" && "$actual" == "$expected"* ]]; then
|
||||
echo "$actual"
|
||||
return 0
|
||||
elif [[ "$match_mode" == "exact" && "$actual" == "$expected" ]]; then
|
||||
echo "$actual"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "$actual"
|
||||
return 1
|
||||
}
|
||||
|
||||
echo "=== Pre-flight ==="
|
||||
if ! docker exec jellyfin sh -c "[ -d \"$NEW_FOLDER\" ]"; then
|
||||
echo "Error: destination folder does not exist: $NEW_FOLDER" >&2
|
||||
exit 1
|
||||
fi
|
||||
baseline_manifest=$(manifest_of "$NEW_FOLDER")
|
||||
baseline_count=$(echo "$baseline_manifest" | grep -c . || true)
|
||||
if [[ "$baseline_count" -eq 0 ]]; then
|
||||
echo "Error: destination folder is empty: $NEW_FOLDER" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Destination has $baseline_count file(s) - recorded as ground truth."
|
||||
|
||||
# --- Idempotency check: is this already correctly relinked? ---
|
||||
BASELINE_MANIFEST_FILE=$(mktemp)
|
||||
echo "$baseline_manifest" > "$BASELINE_MANIFEST_FILE"
|
||||
trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE"' EXIT
|
||||
|
||||
current_info=$(torrent_info)
|
||||
current_content_path=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['content_path'])")
|
||||
if [[ "$current_content_path" == "$NEW_FOLDER"* ]]; then
|
||||
echo "content_path already points at $NEW_FOLDER - checking if fully verified..."
|
||||
current_progress=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['progress'])")
|
||||
# Compare current qBittorrent-tracked file NAMES (not just count) and
|
||||
# progress to the manifest. Progress alone is not enough to trust - it
|
||||
# can be stale, cached from before a move, if a prior attempt was
|
||||
# interrupted before ever triggering a fresh recheck at the new
|
||||
# location (this happened for real: a torrent showed progress 1 for
|
||||
# files under a subfolder that had already stopped existing on disk).
|
||||
# Requiring the basename to appear in the manifest means renameFile
|
||||
# must have actually already run - a stale-but-unrenamed torrent can no
|
||||
# longer produce a false "already good".
|
||||
already_good=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH" | python3 -c "
|
||||
import json, os, sys
|
||||
files = json.load(sys.stdin)
|
||||
manifest = {}
|
||||
with open(sys.argv[1]) as f:
|
||||
for line in f:
|
||||
line = line.rstrip(chr(10))
|
||||
if not line: continue
|
||||
name, size = line.rsplit(chr(9), 1)
|
||||
manifest[name] = int(size)
|
||||
if len(files) != len(manifest):
|
||||
print('no')
|
||||
else:
|
||||
names_match = all(os.path.basename(f['name']) in manifest for f in files)
|
||||
progress_ok = all(f.get('progress', 0) >= 0.999 for f in files)
|
||||
print('yes' if (names_match and progress_ok) else 'no')
|
||||
" "$BASELINE_MANIFEST_FILE" 2>/dev/null || echo "no")
|
||||
if [[ "$already_good" == "yes" ]]; then
|
||||
echo "Already relinked and verified (progress ~$current_progress). Nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
echo "Not yet fully verified (progress: $current_progress) - proceeding carefully."
|
||||
fi
|
||||
|
||||
# --- Stop the torrent before touching anything ---
|
||||
echo
|
||||
echo "=== Stopping torrent ==="
|
||||
api_call "stop" "torrents/stop" --data-urlencode "hashes=$HASH"
|
||||
|
||||
# Current files inside the torrent (relative paths, in qBittorrent's index order)
|
||||
old_files_json=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH")
|
||||
|
||||
torrent_name=$(python3 -c "
|
||||
import json,sys
|
||||
files = json.loads(sys.argv[1])
|
||||
@@ -94,16 +201,13 @@ if not files:
|
||||
sys.exit(1)
|
||||
print(files[0]['name'].split('/')[0])
|
||||
" "$old_files_json")
|
||||
|
||||
echo "Torrent folder: $torrent_name"
|
||||
echo "New folder: $NEW_FOLDER"
|
||||
echo
|
||||
|
||||
# New files actually on disk at the destination (via any container with the nas_media mount)
|
||||
new_files=$(docker exec jellyfin sh -c "ls -1 \"$NEW_FOLDER\"" | sort)
|
||||
new_files=$(echo "$baseline_manifest" | cut -f1)
|
||||
|
||||
# Build the old->new pairing, preferring an explicit SxxEyy match over sort
|
||||
# order (see header comment for why sort order alone is unsafe).
|
||||
echo "=== Computing file pairing ==="
|
||||
python3 -c "
|
||||
import json, re, sys
|
||||
|
||||
@@ -131,8 +235,7 @@ if all(old_keys.values()) and all(new_keys.values()) and len(set(old_keys.values
|
||||
pairs = [(old_by_key[k], new_by_key[k]) for k in sorted(common)]
|
||||
else:
|
||||
print('WARNING: could not parse a unique SxxEyy from every filename - falling back to alphabetical-sort pairing.', file=sys.stderr)
|
||||
print(' Double-check the pairing below carefully before it applies (this is exactly the failure mode', file=sys.stderr)
|
||||
print(' that silently mismatches non-zero-padded episode numbers like E9 vs E10).', file=sys.stderr)
|
||||
print(' Double-check the pairing below carefully before it applies.', file=sys.stderr)
|
||||
pairs = list(zip(sorted(old_keys), sorted(new_keys)))
|
||||
|
||||
with open('/tmp/qbt_relink_flat.tsv', 'w') as f:
|
||||
@@ -143,54 +246,33 @@ with open('/tmp/qbt_relink_flat.tsv', 'w') as f:
|
||||
" "$old_files_json" "$new_files"
|
||||
|
||||
echo
|
||||
echo "--- Applying ---"
|
||||
echo "=== Applying (single pass - stop/relocate/rename/recheck, no retries) ==="
|
||||
|
||||
echo "setLocation -> $NEW_FOLDER"
|
||||
api_call "setLocation" "torrents/setLocation" \
|
||||
--data-urlencode "hashes=$HASH" \
|
||||
--data-urlencode "location=$NEW_FOLDER"
|
||||
|
||||
# Don't just trust the 200 - confirm the torrent's save_path actually changed
|
||||
# before touching anything else. This is exactly the check that was missing
|
||||
# when Babylon 5 S05 silently kept its old location and started re-downloading.
|
||||
actual_path=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)[0]['save_path'])")
|
||||
if [[ "$actual_path" != "$NEW_FOLDER" ]]; then
|
||||
echo "Error: setLocation did not take effect (save_path is still '$actual_path'). Torrent is stopped - not proceeding." >&2
|
||||
if ! actual_path=$(wait_for_path_field "save_path" "$NEW_FOLDER" exact "save_path"); then
|
||||
echo "Error: setLocation did not take effect after 15s (save_path is still '$actual_path'). Torrent is stopped - not proceeding." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " confirmed: save_path is now $actual_path"
|
||||
|
||||
# If qBittorrent's "Keep incomplete torrents in" feature (temp_path_enabled)
|
||||
# is on, and this torrent was ever flagged as needing to download (e.g. an
|
||||
# earlier relink attempt failed and its recheck found 0% at the wrong path),
|
||||
# it can carry a separate download_path override that content_path actually
|
||||
# follows instead of save_path - setLocation alone does NOT clear this. This
|
||||
# bit Babylon 5 S03/S04/S05 in the same incident: recheck kept running
|
||||
# against the stale incomplete-staging path even after setLocation reported
|
||||
# success. Note: this endpoint takes "id", not "hashes" (undocumented
|
||||
# inconsistency vs. every other torrents/* endpoint here).
|
||||
echo "setDownloadPath -> $NEW_FOLDER (in case a stale download_path override exists)"
|
||||
echo "setDownloadPath -> $NEW_FOLDER (clears any stale incomplete-path override)"
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" -X POST "$QBITTORRENT_URL/api/v2/torrents/setDownloadPath" \
|
||||
--data-urlencode "id=$HASH" --data-urlencode "path=$NEW_FOLDER")
|
||||
# 400 here just means the torrent has no download_path override to clear
|
||||
# (the common case) - only treat other failures as fatal.
|
||||
if [[ "$status" != "200" && "$status" != "400" ]]; then
|
||||
echo "Error: setDownloadPath failed (http $status)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
content_path=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH" \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin)[0]['content_path'])")
|
||||
if [[ "$content_path" != "$NEW_FOLDER"* ]]; then
|
||||
echo "Error: content_path is '$content_path', not under '$NEW_FOLDER'. A recheck would silently fail against the wrong path - not proceeding." >&2
|
||||
if ! content_path=$(wait_for_path_field "content_path" "$NEW_FOLDER" prefix "content_path"); then
|
||||
echo "Error: content_path is '$content_path', not under '$NEW_FOLDER' after 15s. Not proceeding." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " confirmed: content_path is now $content_path"
|
||||
|
||||
# setLocation above already moved the torrent's root to $NEW_FOLDER itself
|
||||
# (the season folder) - so new paths here are bare filenames, not prefixed
|
||||
# with the season folder name again.
|
||||
while IFS=$'\t' read -r old new; do
|
||||
api_call "renameFile" "torrents/renameFile" \
|
||||
--data-urlencode "hash=$HASH" \
|
||||
@@ -199,12 +281,77 @@ while IFS=$'\t' read -r old new; do
|
||||
echo " renamed: $old -> $new"
|
||||
done < /tmp/qbt_relink_flat.tsv
|
||||
|
||||
echo "recheck..."
|
||||
echo
|
||||
echo "=== Rechecking (recheck triggered exactly once) ==="
|
||||
api_call "recheck" "torrents/recheck" --data-urlencode "hashes=$HASH"
|
||||
|
||||
echo "Done. Torrent is left STOPPED so nothing can auto-resume downloading"
|
||||
echo "before you've verified the recheck passed. Poll with:"
|
||||
elapsed=0
|
||||
final_state=""
|
||||
while [[ "$elapsed" -lt "$POLL_TIMEOUT_S" ]]; do
|
||||
sleep "$POLL_INTERVAL"
|
||||
elapsed=$((elapsed + POLL_INTERVAL))
|
||||
state=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['state'])")
|
||||
if [[ "$state" != "checkingUP" && "$state" != "checkingDL" && "$state" != "checkingResumeData" ]]; then
|
||||
final_state="$state"
|
||||
break
|
||||
fi
|
||||
echo " still checking (${elapsed}s elapsed): $state"
|
||||
done
|
||||
|
||||
if [[ -z "$final_state" ]]; then
|
||||
echo
|
||||
echo "TIMED OUT after ${POLL_TIMEOUT_S}s still checking. DO NOT re-trigger recheck or resume by hand -" >&2
|
||||
echo "that repeated-intervention pattern is what caused the data-loss incident. Instead re-run this" >&2
|
||||
echo "exact script invocation again later (it's idempotent) or investigate read-only first." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo " settled: $final_state"
|
||||
|
||||
echo
|
||||
echo "=== Post-check: verifying files against the pre-flight manifest ==="
|
||||
FINAL_MANIFEST_FILE=$(mktemp)
|
||||
manifest_of "$NEW_FOLDER" > "$FINAL_MANIFEST_FILE"
|
||||
trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE"' EXIT
|
||||
|
||||
mismatch=$(python3 -c "
|
||||
import sys
|
||||
|
||||
def load(path):
|
||||
d = {}
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
line = line.rstrip(chr(10))
|
||||
if not line:
|
||||
continue
|
||||
name, size = line.rsplit(chr(9), 1)
|
||||
d[name] = size
|
||||
return d
|
||||
|
||||
baseline = load(sys.argv[1])
|
||||
final = load(sys.argv[2])
|
||||
problems = []
|
||||
for name, size in baseline.items():
|
||||
if name not in final:
|
||||
problems.append(f'MISSING: {name}')
|
||||
elif final[name] != size:
|
||||
problems.append(f'SIZE CHANGED: {name} ({size} -> {final[name]})')
|
||||
for name in final:
|
||||
if name not in baseline:
|
||||
problems.append(f'UNEXPECTED NEW FILE: {name}')
|
||||
print(chr(10).join(problems))
|
||||
" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE")
|
||||
|
||||
if [[ -n "$mismatch" ]]; then
|
||||
echo "CRITICAL: files at $NEW_FOLDER changed during relink - possible data loss:" >&2
|
||||
echo "$mismatch" >&2
|
||||
echo >&2
|
||||
echo "The torrent is left STOPPED. Do NOT re-run this script or touch qBittorrent further" >&2
|
||||
echo "until this is investigated by hand - re-running blind is what caused the original incident." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "All $baseline_count file(s) match the pre-flight manifest - no data loss detected."
|
||||
echo
|
||||
echo "Done. Torrent is left STOPPED on purpose. Verify state before starting it:"
|
||||
echo " bash ~/.claude/plugins/cache/claude-homelab/homelab-core/*/skills/qbittorrent/scripts/qbit-api.sh info $HASH"
|
||||
echo "Once it shows 100% progress / stoppedUP (not stoppedDL - stoppedDL means the"
|
||||
echo "recheck found missing/mismatched pieces, do not resume), start it again from"
|
||||
echo "the qBittorrent WebUI (this script does not auto-resume, on purpose)."
|
||||
echo "Then start it from the qBittorrent WebUI once you're satisfied."
|
||||
|
||||
Reference in New Issue
Block a user