Root cause of the data-loss incident: repeated manual re-triggering (stop/setLocation/recheck called multiple times across separate debugging invocations) raced against qBittorrent's own automatic incomplete-file management. The 7 clean successes earlier all completed in one uninterrupted pass; the 2 that failed were the ones manually re-triggered while investigating. Redesign: the whole operation is now one blocking pass (stop -> relocate -> rename -> recheck -> poll to completion -> verify -> report) with a hard rule never to call recheck/stop/setLocation against the same hash a second time while a previous run might still be settling. Adds: - A pre-flight filesystem manifest (name+size) of the destination, and a post-recheck comparison against it - the real ground-truth safety net, independent of trusting qBittorrent's self-reported state. - Idempotency: skips already-correctly-relinked torrents rather than re-touching them. Found and fixed a real gap here during testing - the first version trusted qBittorrent's per-file "progress" alone, which can be stale (cached from before a move) and produced a false "already good" on a torrent whose recheck had never actually run at the new location. Now also requires the tracked filenames to match the destination manifest. - setLocation/setDownloadPath verification now polls briefly instead of checking once immediately - both are asynchronous and a single immediate check can read stale data (this exact bug false-failed a real run during testing). - Fixed manifest generation to use printf instead of `stat -c`'s own \t escape handling, which silently emitted a literal backslash-t instead of a real tab and broke `cut -f1` pairing. Verified end-to-end against a real torrent (Babylon 5 S04, 69GB): clean single-pass recheck, post-check confirmed all 22 files intact. Claude-Session: https://claude.ai/code/session_01HZQK6jHmdTpFjFZM8FUnqA
358 lines
15 KiB
Bash
Executable File
358 lines
15 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Re-point a qBittorrent torrent at files Sonarr has since renamed/moved.
|
|
#
|
|
# When Sonarr imports an existing download via hardlink, it removes the file
|
|
# from its original torrent-named folder (only the new hardlinked copy
|
|
# remains) - so qBittorrent's record of that torrent silently starts
|
|
# pointing at nothing. This re-links the torrent to the new location so it
|
|
# keeps seeding the same underlying data (same inode, so the hash check
|
|
# passes) instead of erroring out as "missing files".
|
|
#
|
|
# Usage:
|
|
# ./qbt-relink.sh <torrent-hash> <new-folder>
|
|
#
|
|
# <new-folder> should be the Sonarr season folder (or show folder, for a
|
|
# single-season show) containing the renamed files - e.g.
|
|
# ./qbt-relink.sh 8601b9b9... "/data/video/tv/Andor (2022)/Season 02"
|
|
#
|
|
# Pairs the torrent's files with the files in <new-folder> by SxxEyy episode
|
|
# number parsed out of each filename (falls back to alphabetical sort only if
|
|
# a name can't be parsed, with a loud warning - alphabetical sort silently
|
|
# mismatches on non-zero-padded episode numbers, e.g. "E9" sorts after "E10").
|
|
# Always prints the proposed pairing before applying - check it makes sense,
|
|
# especially for anything with specials/extras where a torrent's files span
|
|
# more than one destination folder (this script only handles one folder;
|
|
# see the sonarr skill doc for that case).
|
|
#
|
|
# SAFETY MODEL (rewritten after a real data-loss incident - see the sonarr
|
|
# skill doc, "Incident 3"):
|
|
#
|
|
# - The whole operation runs as ONE blocking pass: stop -> relocate -> rename
|
|
# -> recheck -> poll to completion -> verify -> report. It does NOT return
|
|
# control with "poll manually and come back" - that pattern is exactly what
|
|
# caused the incident, because re-running recheck/stop by hand later, after
|
|
# qBittorrent's own state had moved on, raced against qBittorrent's
|
|
# automatic incomplete-file management and let it destroy real files.
|
|
# NEVER call recheck/stop/setLocation against the same hash a second time
|
|
# while a previous invocation might still be settling - wait for this
|
|
# script to finish (or clearly fail) first.
|
|
# - Before touching qBittorrent at all, it snapshots the destination folder's
|
|
# filenames+sizes (the ground truth - Sonarr already put the real files
|
|
# there). After the recheck settles, it re-reads the folder and compares.
|
|
# Any file that shrank, vanished, or changed size is treated as data loss,
|
|
# reported loudly, and the script does NOT attempt any further remediation
|
|
# (no retry, no re-trigger) - investigate by hand from a known-bad state
|
|
# rather than risk compounding it.
|
|
# - If the torrent already looks correctly relinked (content_path matches and
|
|
# on-disk files match the expected sizes), the script does nothing further
|
|
# - idempotent, so re-running it after a partial failure is safe rather
|
|
# than repeating destructive steps.
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="$HOME/.claude-homelab/.env"
|
|
POLL_INTERVAL=15
|
|
POLL_TIMEOUT_S=2400 # 40 min - generous for large multi-GB files over NFS
|
|
|
|
if [[ ! -f "$ENV_FILE" ]]; then
|
|
echo "Error: $ENV_FILE not found (qBittorrent creds live there, not this repo's .credentials)" >&2
|
|
exit 1
|
|
fi
|
|
# shellcheck source=/dev/null
|
|
source "$ENV_FILE"
|
|
|
|
if [[ -z "${QBITTORRENT_URL:-}" || -z "${QBITTORRENT_USERNAME:-}" || -z "${QBITTORRENT_PASSWORD:-}" ]]; then
|
|
echo "Error: QBITTORRENT_URL/USERNAME/PASSWORD not set in $ENV_FILE" >&2
|
|
exit 1
|
|
fi
|
|
|
|
HASH="${1:-}"
|
|
NEW_FOLDER="${2:-}"
|
|
if [[ -z "$HASH" || -z "$NEW_FOLDER" ]]; then
|
|
echo "Usage: $0 <torrent-hash> <new-folder>" >&2
|
|
exit 1
|
|
fi
|
|
|
|
COOKIE_JAR=$(mktemp)
|
|
trap 'rm -f "$COOKIE_JAR"' EXIT
|
|
|
|
login_status=$(curl -s -o /dev/null -w "%{http_code}" -c "$COOKIE_JAR" -X POST "$QBITTORRENT_URL/api/v2/auth/login" \
|
|
--data-urlencode "username=$QBITTORRENT_USERNAME" \
|
|
--data-urlencode "password=$QBITTORRENT_PASSWORD")
|
|
if [[ "$login_status" != "200" ]]; then
|
|
echo "Error: qBittorrent login failed (http $login_status)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# api_call <label> <path> [curl data args...]
|
|
api_call() {
|
|
local label="$1" path="$2"
|
|
shift 2
|
|
local status
|
|
status=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" -X POST "$QBITTORRENT_URL/api/v2/$path" "$@")
|
|
if [[ "$status" != "200" ]]; then
|
|
echo "Error: $label failed (http $status)" >&2
|
|
exit 1
|
|
fi
|
|
}
|
|
|
|
torrent_info() {
|
|
curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/info" --data-urlencode "hashes=$HASH"
|
|
}
|
|
|
|
# manifest_of <folder> -> "name\tsize" lines, sorted by name
|
|
# Uses printf, not `stat -c '%n\t%s'` - stat's own \t escape handling isn't
|
|
# reliable across environments and silently produced a literal backslash-t
|
|
# instead of a real tab the first time this was tried, breaking `cut -f1`.
|
|
manifest_of() {
|
|
docker exec jellyfin sh -c "cd \"$1\" && for f in *; do [ -f \"\$f\" ] && printf '%s\t%s\n' \"\$f\" \"\$(stat -c%s \"\$f\")\"; done" 2>/dev/null | sort
|
|
}
|
|
|
|
# wait_for_field <jq-ish python expr on the info dict> <expected-value> <label>
|
|
# setLocation/setDownloadPath are asynchronous - qBittorrent returns 200
|
|
# before the internal move actually completes, so checking immediately can
|
|
# see stale data. Poll briefly instead of a single immediate check.
|
|
wait_for_path_field() {
|
|
local field="$1" expected="$2" match_mode="$3" label="$4"
|
|
local attempt actual
|
|
for attempt in $(seq 1 15); do
|
|
actual=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['$field'])")
|
|
if [[ "$match_mode" == "prefix" && "$actual" == "$expected"* ]]; then
|
|
echo "$actual"
|
|
return 0
|
|
elif [[ "$match_mode" == "exact" && "$actual" == "$expected" ]]; then
|
|
echo "$actual"
|
|
return 0
|
|
fi
|
|
sleep 1
|
|
done
|
|
echo "$actual"
|
|
return 1
|
|
}
|
|
|
|
echo "=== Pre-flight ==="
|
|
if ! docker exec jellyfin sh -c "[ -d \"$NEW_FOLDER\" ]"; then
|
|
echo "Error: destination folder does not exist: $NEW_FOLDER" >&2
|
|
exit 1
|
|
fi
|
|
baseline_manifest=$(manifest_of "$NEW_FOLDER")
|
|
baseline_count=$(echo "$baseline_manifest" | grep -c . || true)
|
|
if [[ "$baseline_count" -eq 0 ]]; then
|
|
echo "Error: destination folder is empty: $NEW_FOLDER" >&2
|
|
exit 1
|
|
fi
|
|
echo "Destination has $baseline_count file(s) - recorded as ground truth."
|
|
|
|
# --- Idempotency check: is this already correctly relinked? ---
|
|
BASELINE_MANIFEST_FILE=$(mktemp)
|
|
echo "$baseline_manifest" > "$BASELINE_MANIFEST_FILE"
|
|
trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE"' EXIT
|
|
|
|
current_info=$(torrent_info)
|
|
current_content_path=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['content_path'])")
|
|
if [[ "$current_content_path" == "$NEW_FOLDER"* ]]; then
|
|
echo "content_path already points at $NEW_FOLDER - checking if fully verified..."
|
|
current_progress=$(echo "$current_info" | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['progress'])")
|
|
# Compare current qBittorrent-tracked file NAMES (not just count) and
|
|
# progress to the manifest. Progress alone is not enough to trust - it
|
|
# can be stale, cached from before a move, if a prior attempt was
|
|
# interrupted before ever triggering a fresh recheck at the new
|
|
# location (this happened for real: a torrent showed progress 1 for
|
|
# files under a subfolder that had already stopped existing on disk).
|
|
# Requiring the basename to appear in the manifest means renameFile
|
|
# must have actually already run - a stale-but-unrenamed torrent can no
|
|
# longer produce a false "already good".
|
|
already_good=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH" | python3 -c "
|
|
import json, os, sys
|
|
files = json.load(sys.stdin)
|
|
manifest = {}
|
|
with open(sys.argv[1]) as f:
|
|
for line in f:
|
|
line = line.rstrip(chr(10))
|
|
if not line: continue
|
|
name, size = line.rsplit(chr(9), 1)
|
|
manifest[name] = int(size)
|
|
if len(files) != len(manifest):
|
|
print('no')
|
|
else:
|
|
names_match = all(os.path.basename(f['name']) in manifest for f in files)
|
|
progress_ok = all(f.get('progress', 0) >= 0.999 for f in files)
|
|
print('yes' if (names_match and progress_ok) else 'no')
|
|
" "$BASELINE_MANIFEST_FILE" 2>/dev/null || echo "no")
|
|
if [[ "$already_good" == "yes" ]]; then
|
|
echo "Already relinked and verified (progress ~$current_progress). Nothing to do."
|
|
exit 0
|
|
fi
|
|
echo "Not yet fully verified (progress: $current_progress) - proceeding carefully."
|
|
fi
|
|
|
|
# --- Stop the torrent before touching anything ---
|
|
echo
|
|
echo "=== Stopping torrent ==="
|
|
api_call "stop" "torrents/stop" --data-urlencode "hashes=$HASH"
|
|
|
|
old_files_json=$(curl -s -b "$COOKIE_JAR" -G "$QBITTORRENT_URL/api/v2/torrents/files" --data-urlencode "hash=$HASH")
|
|
torrent_name=$(python3 -c "
|
|
import json,sys
|
|
files = json.loads(sys.argv[1])
|
|
if not files:
|
|
print('ERROR: no files returned for this hash - check the hash is correct', file=sys.stderr)
|
|
sys.exit(1)
|
|
print(files[0]['name'].split('/')[0])
|
|
" "$old_files_json")
|
|
echo "Torrent folder: $torrent_name"
|
|
echo "New folder: $NEW_FOLDER"
|
|
echo
|
|
|
|
new_files=$(echo "$baseline_manifest" | cut -f1)
|
|
|
|
echo "=== Computing file pairing ==="
|
|
python3 -c "
|
|
import json, re, sys
|
|
|
|
def ep_key(name):
|
|
m = re.search(r'[Ss](\d{1,4})[Ee](\d{1,4})', name)
|
|
return (int(m.group(1)), int(m.group(2))) if m else None
|
|
|
|
old_files = json.loads(sys.argv[1])
|
|
new_files = sys.argv[2].strip().split(chr(10)) if sys.argv[2].strip() else []
|
|
|
|
if len(old_files) != len(new_files):
|
|
print(f'ERROR: file count mismatch - torrent has {len(old_files)} files, destination has {len(new_files)}', file=sys.stderr)
|
|
sys.exit(1)
|
|
|
|
old_keys = {f['name']: ep_key(f['name']) for f in old_files}
|
|
new_keys = {n: ep_key(n) for n in new_files}
|
|
|
|
if all(old_keys.values()) and all(new_keys.values()) and len(set(old_keys.values())) == len(old_files) and len(set(new_keys.values())) == len(new_files):
|
|
old_by_key = {v: k for k, v in old_keys.items()}
|
|
new_by_key = {v: k for k, v in new_keys.items()}
|
|
common = set(old_by_key) & set(new_by_key)
|
|
if common != set(old_by_key) or common != set(new_by_key):
|
|
print('ERROR: episode-number sets differ between torrent and destination - not safe to auto-pair. Check both manually.', file=sys.stderr)
|
|
sys.exit(1)
|
|
pairs = [(old_by_key[k], new_by_key[k]) for k in sorted(common)]
|
|
else:
|
|
print('WARNING: could not parse a unique SxxEyy from every filename - falling back to alphabetical-sort pairing.', file=sys.stderr)
|
|
print(' Double-check the pairing below carefully before it applies.', file=sys.stderr)
|
|
pairs = list(zip(sorted(old_keys), sorted(new_keys)))
|
|
|
|
with open('/tmp/qbt_relink_flat.tsv', 'w') as f:
|
|
for old, new in pairs:
|
|
print(f' {old}')
|
|
print(f' -> {new}')
|
|
f.write(f'{old}\t{new}\n')
|
|
" "$old_files_json" "$new_files"
|
|
|
|
echo
|
|
echo "=== Applying (single pass - stop/relocate/rename/recheck, no retries) ==="
|
|
|
|
echo "setLocation -> $NEW_FOLDER"
|
|
api_call "setLocation" "torrents/setLocation" \
|
|
--data-urlencode "hashes=$HASH" \
|
|
--data-urlencode "location=$NEW_FOLDER"
|
|
|
|
if ! actual_path=$(wait_for_path_field "save_path" "$NEW_FOLDER" exact "save_path"); then
|
|
echo "Error: setLocation did not take effect after 15s (save_path is still '$actual_path'). Torrent is stopped - not proceeding." >&2
|
|
exit 1
|
|
fi
|
|
echo " confirmed: save_path is now $actual_path"
|
|
|
|
echo "setDownloadPath -> $NEW_FOLDER (clears any stale incomplete-path override)"
|
|
status=$(curl -s -o /dev/null -w "%{http_code}" -b "$COOKIE_JAR" -X POST "$QBITTORRENT_URL/api/v2/torrents/setDownloadPath" \
|
|
--data-urlencode "id=$HASH" --data-urlencode "path=$NEW_FOLDER")
|
|
if [[ "$status" != "200" && "$status" != "400" ]]; then
|
|
echo "Error: setDownloadPath failed (http $status)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if ! content_path=$(wait_for_path_field "content_path" "$NEW_FOLDER" prefix "content_path"); then
|
|
echo "Error: content_path is '$content_path', not under '$NEW_FOLDER' after 15s. Not proceeding." >&2
|
|
exit 1
|
|
fi
|
|
echo " confirmed: content_path is now $content_path"
|
|
|
|
while IFS=$'\t' read -r old new; do
|
|
api_call "renameFile" "torrents/renameFile" \
|
|
--data-urlencode "hash=$HASH" \
|
|
--data-urlencode "oldPath=$old" \
|
|
--data-urlencode "newPath=$new"
|
|
echo " renamed: $old -> $new"
|
|
done < /tmp/qbt_relink_flat.tsv
|
|
|
|
echo
|
|
echo "=== Rechecking (recheck triggered exactly once) ==="
|
|
api_call "recheck" "torrents/recheck" --data-urlencode "hashes=$HASH"
|
|
|
|
elapsed=0
|
|
final_state=""
|
|
while [[ "$elapsed" -lt "$POLL_TIMEOUT_S" ]]; do
|
|
sleep "$POLL_INTERVAL"
|
|
elapsed=$((elapsed + POLL_INTERVAL))
|
|
state=$(torrent_info | python3 -c "import json,sys; print(json.load(sys.stdin)[0]['state'])")
|
|
if [[ "$state" != "checkingUP" && "$state" != "checkingDL" && "$state" != "checkingResumeData" ]]; then
|
|
final_state="$state"
|
|
break
|
|
fi
|
|
echo " still checking (${elapsed}s elapsed): $state"
|
|
done
|
|
|
|
if [[ -z "$final_state" ]]; then
|
|
echo
|
|
echo "TIMED OUT after ${POLL_TIMEOUT_S}s still checking. DO NOT re-trigger recheck or resume by hand -" >&2
|
|
echo "that repeated-intervention pattern is what caused the data-loss incident. Instead re-run this" >&2
|
|
echo "exact script invocation again later (it's idempotent) or investigate read-only first." >&2
|
|
exit 1
|
|
fi
|
|
echo " settled: $final_state"
|
|
|
|
echo
|
|
echo "=== Post-check: verifying files against the pre-flight manifest ==="
|
|
FINAL_MANIFEST_FILE=$(mktemp)
|
|
manifest_of "$NEW_FOLDER" > "$FINAL_MANIFEST_FILE"
|
|
trap 'rm -f "$COOKIE_JAR" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE"' EXIT
|
|
|
|
mismatch=$(python3 -c "
|
|
import sys
|
|
|
|
def load(path):
|
|
d = {}
|
|
with open(path) as f:
|
|
for line in f:
|
|
line = line.rstrip(chr(10))
|
|
if not line:
|
|
continue
|
|
name, size = line.rsplit(chr(9), 1)
|
|
d[name] = size
|
|
return d
|
|
|
|
baseline = load(sys.argv[1])
|
|
final = load(sys.argv[2])
|
|
problems = []
|
|
for name, size in baseline.items():
|
|
if name not in final:
|
|
problems.append(f'MISSING: {name}')
|
|
elif final[name] != size:
|
|
problems.append(f'SIZE CHANGED: {name} ({size} -> {final[name]})')
|
|
for name in final:
|
|
if name not in baseline:
|
|
problems.append(f'UNEXPECTED NEW FILE: {name}')
|
|
print(chr(10).join(problems))
|
|
" "$BASELINE_MANIFEST_FILE" "$FINAL_MANIFEST_FILE")
|
|
|
|
if [[ -n "$mismatch" ]]; then
|
|
echo "CRITICAL: files at $NEW_FOLDER changed during relink - possible data loss:" >&2
|
|
echo "$mismatch" >&2
|
|
echo >&2
|
|
echo "The torrent is left STOPPED. Do NOT re-run this script or touch qBittorrent further" >&2
|
|
echo "until this is investigated by hand - re-running blind is what caused the original incident." >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "All $baseline_count file(s) match the pre-flight manifest - no data loss detected."
|
|
echo
|
|
echo "Done. Torrent is left STOPPED on purpose. Verify state before starting it:"
|
|
echo " bash ~/.claude/plugins/cache/claude-homelab/homelab-core/*/skills/qbittorrent/scripts/qbit-api.sh info $HASH"
|
|
echo "Then start it from the qBittorrent WebUI once you're satisfied."
|