Scraping resilience, metadata tooling, and repository hygiene
Consolidates mosaic and session hardening (login retry, skip processed scans, no retry on 404, started_at), progress reporting (Markdown tables, by-year rollup, rolling-window rate/ETA), and metadata workflow scripts (run_metadata_scan.sh, scan_progress_report.py, export_machine_metadata.py). Adds mosaic reconstruction sample JPEGs referenced by the report. Updates .gitignore for backup/ and .claude/; sample_random_scans helper is documented for branch testing/sample-runs only (see README).
This commit is contained in:
@@ -97,10 +97,8 @@ python scraper.py --machine "BW3-20 [AMR-26]" --mosaic-only
|
||||
# Download mosaics for all machines
|
||||
python scraper.py --mosaic-only
|
||||
|
||||
# One random completed scan per machine: mosaic + all tiles (from machines.txt; uses --list-scans + --scan-id)
|
||||
# MOSAIC_ONLY=1 ./scripts/sample_random_scans.sh machines.txt # optional: mosaics only, no tiles
|
||||
# cp scripts/machines.example.txt machines.txt # then edit: one label per line
|
||||
# ./scripts/sample_random_scans.sh machines.txt
|
||||
# One random completed scan per machine (helper script): check out branch `testing/sample-runs`,
|
||||
# then see `scripts/sample_random_scans.sh` and `docs/sample_random_scans_run_progress.md`.
|
||||
|
||||
# Download all tiles for a specific scan
|
||||
python scraper.py --machine "BW3-20 [AMR-26]" --scan-id 158374 --workers 4
|
||||
|
||||
Reference in New Issue
Block a user