Skip to content

Start terminals without forking the server - #250

Open
krassowski wants to merge 26 commits into
jupyter:mainfrom
krassowski:fork-free-spawn
Open

krassowski wants to merge 26 commits into
jupyter:mainfrom
krassowski:fork-free-spawn

Conversation

@krassowski

@krassowski krassowski commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

References

  • Add a test demonstrating the forking hazards - first commit should fail on CI
Details
FAILED tests/fork_hazards_test.py::test_lock_held_by_another_thread_does_not_hang - AssertionError: assert 'timed out\nstarted\n' == 'started\n'
  
  + timed out
    started
FAILED tests/fork_hazards_test.py::test_objects_of_other_threads_are_not_finalized - assert "False\n['row 0', 'row 1', 'row 2', 'row 0', 'row 1', 'row 2']\n" == "True\n['row 0', 'row 1', 'row 2']\n"
  
  - True
  - ['row 0', 'row 1', 'row 2']
  + False
  + ['row 0', 'row 1', 'row 2', 'row 0', 'row 1', 'row 2']
============================== 2 failed in 15.48s ==============================

Code changes

  • Starts the shell from a small python -I -S helper launched with os.posix_spawn(), so no server code runs in the child (option 1 in Start terminals without forking the server #249).
  • Starts the helper with the server's environment, which the startup check tests, and passes env to os.execve() through a pipe.
  • Blocks SIGINT in the helper until it calls setsid(), so a Ctrl-C to the server does not stop a terminal that is starting.
  • Falls back to the fork on PyPy (no os.posix_spawn()), in frozen apps, and when a first test start of the helper with sys.executable fails or takes longer than 30 s.
  • Raises TimeoutError when the shell has not started after 30 s.
  • Raises OSError with the exit code and the last line that the helper printed, when the helper exits before exec.
  • Closes the terminal and kills the helper when an open fails, for example with too many open files.
  • Resets SIGXFSZ as well as SIGPIPE on both paths, and starts the shell from the helper with no blocked signals.
  • Passes env to the fork unchanged, so that env=None gives the shell the environment of the server on that path too.

User-facing changes

Before: opening a terminal forks the server, and extension and library code runs in the copy before the shell starts (#249).

  • A hang in the copy hangs the server too.
  • The copy can delete or rewrite files that the server still uses.
  • A working directory that the server cannot enter makes the server stop answering requests.

After: no server code runs before the shell starts.

  • An open in a working directory that cannot be entered fails with PermissionError.
  • Ctrl-C works in the terminals of a server that ignores SIGINT, for example jupyter lab & in a script.
  • An open blocks the server for 14 to 20 ms at every server size measured. On a fresh server, that is about 10 ms longer than the fork.

The table gives medians of 30 opens on Ubuntu 26.04 with Python 3.15:

Server memory Before After
34 MiB 3.4 ms 13.7 ms
1 GiB 42 ms 14 ms
4 GiB 113 ms 14 ms
8 GiB 182 ms 20 ms

Backwards-incompatible changes

None

AI usage

  • YES: Some or all of the content of this PR was generated by AI.
  • YES: The human author has carefully reviewed this PR and run this code (keep this PR "draft" until the answer is YES)
  • AI tools and models used: Claude Code with Claude Opus 5.5

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unbounded waitpid() calls can freeze terminal creation beyond EXEC_TIMEOUT; both startup waits need deadline enforcement, termination, and asynchronous reaping of overdue processes.

1 open issue
What changed in this PR

Moves POSIX terminal startup to a small os.posix_spawn() helper, avoiding server-fork hazards in multithreaded Jupyter servers while retaining compatibility fallbacks.

Changes:

  • Adds helper-based startup with environment transfer, terminal setup, signal handling, and startup error reporting.
  • Retains fork fallback and adds the PTYPROCESS_USE_FORK override.
  • Adds regression coverage and updates documentation and test configuration.
File Description
tests/​spawn_test.py Tests helper startup, terminal behavior, errors, cleanup, and fallbacks.
tests/​fork_hazards_test.py Tests protection against fork-related deadlocks and finalizer side effects.
terminado/​management.py Selects helper or fork startup and updates fallback environment and signal handling.
terminado/​_spawn.py Implements helper availability checks, spawning, communication, and cleanup.
terminado/​_spawn_helper.py Configures the terminal, signals, and descriptors before executing the command.
README.md Documents helper startup, fallback conditions, and the forced-fork option.
pyproject.toml Adjusts warning filters, coverage exclusions, and helper lint exceptions.

Review effort: Balanced

Comment thread terminado/_spawn.py Resolved
Comment thread .github/workflows/test.yml Resolved
@Carreau

Carreau commented Nov 13, 2026

Copy link
Copy Markdown
Member

Made a separate PR to drop 3.10, we can remove it from here once the other is merged, I just don't want to spend time fixing test on a version we won't support.

spawn() blocked on the error pipe until the helper called exec. A
chdir() into a stalled network mount, or any process that keeps the
pipe open, then froze the whole server. spawn() now waits at most
EXEC_TIMEOUT seconds, then kills the helper, reaps it in a daemon
thread and raises TimeoutError.
The helper decodes its arguments with the locale in env, but encoded the
file name of an error as UTF-8, and spawn() decoded it as UTF-8. With a
Latin-1 locale in env, a missing cwd named café came back as café. The
helper now uses os.fsencode() and spawn() uses os.fsdecode(), so the
bytes of the name reach spawn() unchanged. A test would need a non-UTF-8
locale, not available on CI so no test here.
krassowski and others added 3 commits November 13, 2026 14:35
On macOS, the output of a helper that exits before exec is lost when the
last slave of the pty is closed. The parent closed its slave right after
posix_spawn(), so the error message lost the last line of the traceback.
Keep the slave open until that output has been read.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Assign the duration to `elapsed` before asserting, so that the rewritten
assertion in the CI log shows a readable number.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Carreau

Carreau commented Nov 13, 2026

Copy link
Copy Markdown
Member

Rebase on main with the help of haiku, the test should be passing, This does not contain the previous commit about dropping 3.10.

@Carreau

Carreau commented Nov 17, 2026

Copy link
Copy Markdown
Member

I renamed _close to _close_all_fd as it was unclear this was taking a *kwarg to me. I'm +1 to merge now.

@krassowski I'll let you review my changes if you wish and will merge next monday otherwise.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants