Skip to content

gh-158570: Fix ProcessPoolExecutor shutdown deadlock when the result pipe is full - #158571

Open
tfzee wants to merge 2 commits into
python:mainfrom
tfzee:fix-ppe-shutdown-result-pipe-deadlock
Open

tfzee wants to merge 2 commits into
python:mainfrom
tfzee:fix-ppe-shutdown-result-pipe-deadlock

Conversation

@tfzee

@tfzee tfzee commented Oct 1, 2026 •

Copy link
Copy Markdown

On shutdown, each worker puts its pid on the result queue before exiting,
while the executor manager thread waits for the workers in _join_executor_internals() without reading that queue.
If the result pipe fills up, for example because Linux shrank it under fs.pipe-user-pages-soft on a machine with many workers,
an exiting worker blocks forever writing to it while holding the queue's write lock.
Every other worker then blocks behind it, and shutdown(wait=True) never returns.

This change drains the result queue while joining each worker.
When _join_executor_internals() is reached there are no pending work items,
so the only messages left are worker exit notifications and they can be discarded.

The new test makes the hang deterministic with two workers.
Right before the workers are told to exit, a thread puts a PIPE_MAX_SIZE payload on the result queue.
This is to emulate the alot of workers exiting and writing their results back.
That thread blocks while holding the queue's write lock, so without the fix every exiting worker blocks behind it.

Testing:

  • The new test passes with the fork, forkserver and spawn start methods, and hangs without the fix.
  • ./python -m test test_concurrent_futures passes.
  • The reproducer from the issue (400 workers, 4 KiB result pipe) now shuts down in about 0.2s;
    without the fix it hangs.

Prepared with help from an AI assistant (Claude). I reviewed and verified the change.

@python-cla-bot

python-cla-bot Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant