Skip to content

Rebuild watchdog's os._exit orphans a mid-task ProcessPoolExecutor worker #3396

Description

@ayushcodes10

Summary

Filing separately from #3341, which is being closed as fixed by #3210. @Yo-TR's comment there (#3341 (comment)) flagged a real, separate finding worth tracking on its own: when a git-hook rebuild watchdog fires while a ProcessPoolExecutor worker is still mid-task, os._exit(1) kills only the rebuild process itself. It skips every cleanup path, including the pool's own context-manager shutdown, so a worker that is still running becomes orphaned (reparented to PID 1 on POSIX) and keeps going unsupervised for as long as whatever it was doing takes. A worker stuck in catastrophic regex backtracking (the exact shape #3341 fixed) was observed surviving 2.5 days that way on macOS.

Where

Both git-hook rebuild bodies in graphify/hooks.py (_REBUILD_BODY_COMMIT, _REBUILD_BODY_CHECKOUT) arm a GRAPHIFY_REBUILD_TIMEOUT watchdog. On platforms without signal.SIGALRM, the fallback is a threading.Timer whose callback calls os._exit(1) directly with no cleanup:

def _bail():
    print(f'[graphify hook] graphify rebuild exceeded {_timeout}s', flush=True)
    os._exit(1)

extract.py already documents this exact risk for its own 1-worker special case (# ...the parent's rebuild watchdog (os._exit) can orphan a worker that is mid-task), but that only covers max_workers == 1; a real multi-worker pool spawned for a larger corpus is not covered.

Suggested fix

Terminate any live worker processes before exiting, as @Yo-TR suggested. The watchdog fires on a separate timer thread with no reference to the pool object, but multiprocessing.active_children() enumerates every live worker process regardless of which thread asks. Kill (SIGKILL), not terminate (SIGTERM) — a worker stuck in a C-level call like regex backtracking never gets a chance to act on SIGTERM.

Environment (from the original report): graphifyy 0.9.53 → 0.9.55, CPython 3.14.7 (Homebrew), macOS 26 (Darwin 25.5.0).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions