Parallel lattice loading

[ChristopherMayes]

Adds a reusable primitive for initializing multiple SubprocessTao instances concurrently, and rewires the pytao-constraints tool to use it.

Each SubprocessTao owns a private pipe, subprocess, FIFO, and shared-memory segment, and pytao.errors uses contextvars rather than global state. Nothing is shared between instances and no code path calls os.chdir, so initialization can be driven from a thread pool: the threads only block on I/O with independent child processes.

New public API

• pytao.parallel_subprocess_taos(startups, *, jobs=None) — context manager accepting TaoStartup objects or plain dicts of init arguments.
• pytao.TaoInitResult — per-item outcome (startup, tao, error, elapsed_time, ok).
• pytao.subproc.resolve_job_count() / MAX_AUTO_JOBS — job-count policy, defaulting to CPU count capped at 8.

Results are streamed as a bounded-prefetch pipeline rather than materialized as a list. Up to jobs instances are initialized ahead of the consumer, and each is closed as soon as the consumer advances past it. This caps live subprocesses at jobs regardless of how many startups are supplied, while still overlapping caller work on one instance with initialization of the next.

Initialization failures are captured per item rather than raised, so one bad lattice does not prevent the rest from loading. Everything still open is closed when the block exits, including on early break or an exception in the body.

with parallel_subprocess_taos(lattices, jobs=4) as results:
    for res in results:
        if res.ok:
            print(res.tao.version())
        else:
            print(f"failed: {res.error}")

Note that res.tao is closed once the loop advances and must not be retained.

Constraints tool

pytao-constraints gains -j / --jobs N. Lattice loading previously ran sequentially; it now loads in parallel by default. _run_lattice was split so that the loader owns process lifetime and _observe_lattice only evaluates observables against an already-initialized instance.

Output is unchanged aside from the loading header, which reports the job count when greater than one. Lattices are reported in configuration order and observations are byte-identical to a sequential run — ordering now falls out of the streaming design rather than requiring a post-hoc re-sort.

Measured on a 9-lattice configuration: 3.27 s at -j 1, 1.28 s at -j 4, 0.56 s at -j 9.

Tests

Eight tests for the loader in test_subproc.py covering serial/parallel parity, dict and TaoStartup inputs, ordering, isolated initialization failure, the live-subprocess bound, and cleanup on both exception and early break. Three tests in test_run.py covering job-count resolution and parallel/serial equivalence of observations and constraint results.

Notes for review

• Branched from FIX: pytao should not specify Tao's default #186 rather than master, so the diff currently includes that commit. It will shrink once FIX: pytao should not specify Tao's default #186 lands.
• reference_results.json is regenerated here: FIX: pytao should not specify Tao's default #186 changes the hook_init_file default to None, which appears in the recorded startup for each lattice. As merged, FIX: pytao should not specify Tao's default #186 does not include this regeneration, so test_cli_reference_results fails on that commit standalone.
• Unrelated, and not addressed here: Tao silently truncates its init command line at 200 characters. When the truncation lands on a valid unique flag prefix the run proceeds with different settings than requested; otherwise it fails with an opaque THIS MATCHES TO MULTIPLE COMMAND LINE ARGUMENTS error. A length guard in make_tao_init would be worthwhile.

---

Drafted with Claude via GitHub Copilot in VS Code; all changes reviewed and tested locally.