Kolpack-Software-Consulting/SPRUCE-scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add sample-run checkpoint and disk-space investigation manifest

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
poprhythm
2026-05-14 19:39:34 -04:00
co-authored by Cursor
parent 3a3083b4e3
commit 1e1695a27d
2 changed files with 351 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sample_random_scans_run_progress.md
+50
View File
@@ -0,0 +1,50 @@
# `sample_random_scans.sh` run progress (checkpoint)
Snapshot from terminal session **9** (repo: `/Users/igt/Documents/spruce_scraper`), as of when the machine was about to be restarted. **Date:** 2026-04-26.
## Active run (incomplete)
A **full scan** was in progress: **mosaic + all tiles** (worker count from `config.yaml`), with scan listing using **`--list-scans-first-page-only`** (one page, up to 320 scan IDs, uniform random choice among that page).
| Item | Value |
|------|--------|
| Script | `./scripts/sample_random_scans.sh` |
| Machines file | `machines.txt` (12 machines) |
| Config | `config.yaml` |
| State files | `archives/scans.csv`, `archives/tiles.csv`, `archives/.progress.json` |
### Where it stopped
The run was on **step [9/12]**, machine **BW3-17 [AMR-20]**, **scan ID 153772**.
- **Mosaic:** HTTP **404** for `…/RootView_Database/153772/mosaic.jpg` (same pattern as other scans: tiles still available).
- **Tiles:** **33784** total; progress bar showed roughly **5%** completed — last log line observed was on the order of **~1736 / 33784** tiles (exact count advances continuously; re-check `archives/.progress.json` or resume to see current).
**Not yet started** in this full-scan pass: steps **[10/12]–[12/12]**: **BW3-19 [AMR-21]**, **BW3-20 [AMR-26]**, **BW3-21 [AMR-17]** (lines 12–14 of `machines.txt`).
### Skipped machine in this pass
- **[4/12] BW2-8 [AMR-25]:** `SKIPPED` — `scraper.py --list-scans --list-scans-first-page-only` exited with **code 1** (could not get scan list or pick an ID). The script continued with the next machine.
### Completed machines in this full-scan pass (steps 1–3, 5–8)
| Step | Machine | Scan ID | Mosaic | Tiles downloaded |
|------|---------|---------|--------|------------------|
| 1 | BW1-4 [AMR-15] | 71478 | 404 | 56 |
| 2 | BW1-6 [AMR-19] | 156875 | saved | 72 |
| 3 | BW1-7 [AMR-18] | 10837 | 404 | 1170 |
| 4 | BW2-8 [AMR-25] | — | — | skipped |
| 5 | BW2-10 [AMR-22] | 146368 | saved | 156 |
| 6 | BW2-11 [AMR-23] | 160022 | saved | 529 |
| 7 | BW2-13 [AMR-24] | 156957 | saved | 143 |
| 8 | BW3-16 [AMR-16] | 77300 | 404 | 400 |
## After restart
1. `cd` to the repo and activate the same venv as before.
2. Re-run **`./scripts/sample_random_scans.sh`** with the **same mode** (full scan — default if that is what you used). The scraper **resumes** from `archives/.progress.json` and will continue **BW3-17** scan **153772** (remaining tiles) before moving to later machines, unless you change options or data manually.
## Other runs in the same log (for context)
- Earlier **`DRY_FLAG[@]: unbound variable`** errors from the script were fixed in later invocations.
- A **mosaic-only** pass over all 12 machines completed with banner: *12 machine(s) with mosaic step completed, 0 skipped* (random scan per machine from the first page of IDs). That is a **separate** completed run from the **in-progress full scan** above.