Scraping resilience, metadata tooling, and repository hygiene
Consolidates mosaic and session hardening (login retry, skip processed scans, no retry on 404, started_at), progress reporting (Markdown tables, by-year rollup, rolling-window rate/ETA), and metadata workflow scripts (run_metadata_scan.sh, scan_progress_report.py, export_machine_metadata.py). Adds mosaic reconstruction sample JPEGs referenced by the report. Updates .gitignore for backup/ and .claude/; sample_random_scans helper is documented for branch testing/sample-runs only (see README).
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
VENV="/tmp/spruce_venv"
|
||||
|
||||
if [[ ! -x "$VENV/bin/python" ]]; then
|
||||
echo "Setting up venv at $VENV..."
|
||||
python3 -m venv "$VENV"
|
||||
"$VENV/bin/python" -m ensurepip --upgrade
|
||||
"$VENV/bin/pip" install -q -r "$SCRIPT_DIR/requirements.txt"
|
||||
fi
|
||||
|
||||
echo "Starting metadata-only scan of all machines..."
|
||||
"$VENV/bin/python" "$SCRIPT_DIR/scraper.py" --metadata-only "$@"
|
||||
Reference in New Issue
Block a user