Skip to content

Run PARI operations from Python on a dedicated owner thread - #215

Open
cxzhong wants to merge 9 commits into
sagemath:mainfrom
cxzhong:agent/cypari2-threadsafe-runtime
Open

cxzhong wants to merge 9 commits into
sagemath:mainfrom
cxzhong:agent/cypari2-threadsafe-runtime

Conversation

@cxzhong

@cxzhong cxzhong commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Python thread-pool workers can enter libpari without an initialized PARI context and crash (#107). This change routes Python API operations that touch libpari through one permanent owner thread. Managed Gen values can then outlive the thread that produced them and be used or released by other Python threads.

Calls from Python threads are serialized. Live stack values are moved to the owner's clone heap before a request completes, including values retained through callback side effects or exceptions. Python operator behavior and lazy iteration are preserved. Callbacks run in a snapshot of the submitting caller's contextvars context and can make nested CyPari2 calls. PARI's native parallel computations remain available.

Compatibility

  • Cython: Pari() no longer initializes the submitting thread's PARI stack. Move complete computations using raw GEN values into pari.run_on_owner(function, *args, **kwargs), returning Python values or managed Gen objects. The existing new_gen_from_double and new_t_COMPLEX_from_double helpers dispatch automatically.
  • Signals: cysignals shares its jump state across threads. Requests encountering an existing foreign sig_on() frame raise RuntimeError; callers must leave that frame before submitting a request. Independent external cysignals-protected operations must not run concurrently with the owner.
  • Callbacks: an owner callback must not wait for another thread to make a CyPari2 call. Suppressing a nested PariError aborts the enclosing evaluation because PARI has reset its evaluator. Python callbacks in native PARI workers are rejected; use nbthreads = 1 for those callbacks.
  • Processes: use multiprocessing's spawn method, or fork before initialization. Descendants forked after initialization reject CyPari2 use, including after subsequent forks.

Performance

Thread dispatch adds a fixed cost per request, which is substantial for small operations. run_on_owner() provides a public batching entry point. The included benchmark compares small arithmetic, larger PARI calculations, and batched execution.

Validation

Built and installed the wheel on Linux x86_64 with Python 3.14.4, PARI 2.17.3 (pthread), Cython 3.3.0, and cysignals 1.13.1.

  • make check: 2,493 doctests and 53 unit tests passed.
  • Regression coverage includes operator dispatch, nested callbacks, foreign signal frames, Cython integration, and repeated forks.
  • git diff --check passed.

Closes #107

cxzhong and others added 3 commits September 29, 2026 21:51
After the no-editable wheel install, launching Python from the repository root shadows the installed cypari2 extension with the source package. Start the two context tests from the tests directory so they import the installed wheel, matching CI.
@dimpase
dimpase force-pushed the agent/cypari2-threadsafe-runtime branch from 2f03e7e to 18ce454 Compare September 30, 2026 02:54
dimpase and others added 4 commits September 29, 2026 21:59
uv caches local directory packages using pyproject.toml metadata, so the persistent Darwin runner can install an older cypari2 wheel after a source-only change. Force a fresh project build before running tests.
Owner requests run under a snapshot of the submitting thread context, including reentrant Python callbacks. Correct the threading guide and cover concurrent callers with distinct ContextVar values.

Co-authored-by: Codex <codex@openai.com>
The owner thread toggles this flag while signal handlers on other threads read it. Use the atomic type from cysignals rather than volatile sig_atomic_t, which only addresses signal access on one thread.

Co-authored-by: Codex <codex@openai.com>
Document the fixed dispatch cost of small PARI operations and provide comparable measurements for main and this PR. The private owner-call cases isolate the round-trip overhead.

Co-authored-by: Codex <codex@openai.com>
@cxzhong
cxzhong marked this pull request as ready for review October 7, 2026 12:45
@cxzhong cxzhong changed the title Make CyPari2's Python API thread-safe Run PARI operations from Python on a dedicated owner thread Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make cypari2 threadsafe

2 participants