fix(ocr): keep multi-page OCR from deadlocking on a worker's session lock

[sasuketorii]

Summary

Multi-page OCR could hang forever at 0% CPU when the engine runs more than one OCR worker (pipeline_concurrency() of 2 or 3, which needs 8 or more cores). OarOcrEngine::recognize spread pages over a shared rayon pool and picked each page's worker by rayon::current_thread_index(). oar-ocr-core runs rayon work while it holds a worker's ONNX session mutex: OrtInfer::infer_first_output_f32 calls its closure under the lock, and the recognizer's closure decodes with CTCLabelDecode::argmax_predictions (par_chunks_exact). While that inner join waited, work stealing could start another page on the same thread, which maps to the same worker, and that page blocked on the mutex its own thread already held.

One pool thread of a hung pdf2md <100-page scan> --json --ocr auto (Apple M2 Max, 3 workers), sampled with sample:

OarOcrEngine::recognize_page
  TextRecognitionPredictor::predict -> CRNNModel::forward_refs
    OrtInfer::infer_first_output_f32              <- locks the worker's session
      CTCLabelDecode::argmax_predictions -> par_chunks_exact -> join (waiting)
        OarOcrEngine::recognize_page              <- stolen page, same worker
          TextRecognitionPredictor::predict -> CRNNModel::forward_refs
            OrtInfer::infer_first_output_f32
              Mutex::lock -> __psynch_mutexwait

The main thread was waiting in ThreadPool::install and every other pool thread was asleep. With fewer than 8 cores the engine builds one worker and never takes this branch, so a 4-core machine does not hit it.

Changes

• Each OcrWorker owns a one-thread rayon pool (pdf-inspector-ocr-{k}), and every page runs inside its worker's install. The rayon calls oar-ocr makes under the session lock are join-based (par_iter, par_chunks*), so on a one-thread pool they finish from the thread's own deque and never pick up another page.
• dispatch_pages_on_workers keeps the old branch: one worker or one page runs in page order on workers[0]'s pool; otherwise map_pages_on_workers starts one scoped dispatch thread per worker, each taking the next page from a shared cursor. Dispatch threads only wait on install, so a caller that is itself a rayon worker (for example files.par_iter() around process_pdf_with_ocr) does not tie up the global pool: all of the library's rayon work stays on the worker pools.
• Pages come back in order, the lowest-index error still wins, and a panic stops dispatch and resumes on the caller with its original payload.
• If a worker's pool can't start (a thread limit, for example), the engine keeps the workers whose pools did start; if the first one fails, it runs a single worker sequentially on the caller's thread, the same fallback the shared pool had. A warning is logged either way.

Tests

• Unit tests with fake workers: page order, empty input, lowest-index error, panic payload, and that every page runs on its own worker's pool thread in both the sequential and the parallel branch. The pool-startup fallback is covered for all pools starting, a later pool failing, and the first pool failing.
• Re-entry probe: each fake worker holds a try_locked mutex across a short sleep-based inner par_iter; no re-entry for one caller or for four concurrent callers sharing one worker set. A 200-page version of this probe re-entered in about 45% of rounds against the previous shared-pool dispatch (release build).
• Global-pool callers: current_num_threads().clamp(1, 8) global-pool tasks dispatch pages with inner rayon work under a 30 s timeout. Scoped dispatch threads without per-worker pools time out here, because every global worker waits on its scope while the inner work is queued on the global pool.
• cargo fmt --all -- --check, cargo clippy -- -D warnings, cargo clippy --features ocr -- -D warnings, cargo test and cargo test --features ocr pass. I don't have access to pdf-evals, so it was not run.

Footprint

Whole-document --json --ocr auto on 100-page scans (2294×1770 RGB page images, Apple M2 Max, 3 workers):

• main hung at 0% CPU in 2 of 14 runs over two scans; the second hang's stack matched the one above.
• Alternating runs on one scan: main 161.5 s, 175.0 s, then the hang; this branch 157.2 s, 162.1 s, 163.0 s. CPU time and peak RSS (about 6.55 GB) were about the same. These timings are from an earlier revision of this branch with the same dispatch; the PR's first commit finished the same scan in 158.3 s with 6.55 GB peak RSS, and the follow-up commit only changes what happens when a pool fails to start.
• The OCR Markdown of all 100 pages is identical between main and this branch.

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Shadow auto-approve: would not auto-approve because issues were found.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Shadow auto-approve: would require human review. Reworks multi-page OCR dispatch to per-worker single-thread pools to fix a deadlock, adding a nonblocking fallback and two new OarOcrError variants. Needs human sign-off on the concurrency tradeoff and the public API change.

Re-trigger cubic</sub>