Files
SPRUCE-scraper/run_metadata_scan.sh
poprhythm 6390f5d529 Scraping resilience, metadata tooling, and repository hygiene
Consolidates mosaic and session hardening (login retry, skip processed scans, no retry on 404, started_at), progress reporting (Markdown tables, by-year rollup, rolling-window rate/ETA), and metadata workflow scripts (run_metadata_scan.sh, scan_progress_report.py, export_machine_metadata.py). Adds mosaic reconstruction sample JPEGs referenced by the report. Updates .gitignore for backup/ and .claude/; sample_random_scans helper is documented for branch testing/sample-runs only (see README).
2026-05-14 19:52:53 -04:00

16 lines
463 B
Bash

#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENV="/tmp/spruce_venv"
if [[ ! -x "$VENV/bin/python" ]]; then
echo "Setting up venv at $VENV..."
python3 -m venv "$VENV"
"$VENV/bin/python" -m ensurepip --upgrade
"$VENV/bin/pip" install -q -r "$SCRIPT_DIR/requirements.txt"
fi
echo "Starting metadata-only scan of all machines..."
"$VENV/bin/python" "$SCRIPT_DIR/scraper.py" --metadata-only "$@"