Use asyncio’s loop.run_in_executor() with a ProcessPoolExecutor to run CPU-heavy synchronous functions outside the event-loop thread. On Linux with Python 3.14, the default multiprocessing start method is forkserver; if your program needs a different method, select it explicitly. Reliable use also depends on importable worker functions, picklable inputs and results, deliberate shutdown, and tests that exercise the process context you deploy.
Run CPU-bound functions without blocking the event loop
Calling CPU-heavy synchronous code directly from a coroutine blocks the event loop while that code runs. The asyncio development guide advises against this; the event-loop documentation shows run_in_executor() with a process pool as an option for CPU-bound work.
A basic pattern is to create the pool in the parent process, submit work through the running loop, and protect the program entry point:
import asyncio
from concurrent.futures import ProcessPoolExecutor
# Define workers at module scope so child processes can import them.
def cpu_bound(value):
return value * value
async def main():
with ProcessPoolExecutor() as pool:
loop = asyncio.get_running_loop()
result = await loop.run_in_executor(pool, cpu_bound, 12)
print(result)
if __name__ == "__main__":
asyncio.run(main())The example prints 144 if the submitted work completes successfully. Keep the event-loop coordination in the parent: Python documents that a coroutine or callback cannot be scheduled directly from a separate multiprocessing process. Use the executor integration or an explicit interprocess communication mechanism instead.
#1 Best Overall
Make submitted work importable and serializable
- Define worker functions at module scope. A function defined only in a REPL session or a lambda is not a reliable process-pool target.
- Ensure the callable, its arguments, and its return value can be pickled. With
spawn, the child must be able to import the main module and unpickle the target and arguments. - Keep the
if __name__ == "__main__":guard around program startup. The process-pool documentation requires this guard for the multiprocessing-backed example. - Do not have a submitted function call executor or future methods on the same process pool; Python warns that this can deadlock.
These requirements are described in the concurrent.futures documentation.
Choose a multiprocessing start method deliberately
Python 3.14.8 documentation, consulted on 2026-10-07, identifies forkserver as the default start method on supported POSIX platforms, including Linux. That differs from older assumptions that Linux will use fork. The available methods have different startup, inheritance, and safety trade-offs:
| Method | How workers start | Practical implications |
|---|---|---|
forkserver |
A server process forks workers when requested. | Python 3.14 made this the POSIX default. The server is generally single-threaded and avoids inheriting unnecessary resources from the application process. |
spawn |
Starts a fresh interpreter and passes the resources needed to run the child. | Python describes startup as slower than fork or forkserver. The child must import the main module and unpickle the target and arguments. |
fork |
Duplicates the parent interpreter and inherits its resources. | Safely forking a multithreaded process is problematic. Since Python 3.14, fork is not the default on any platform and must be requested explicitly. |
These behaviors are documented in Python’s multiprocessing documentation. The default described here is specific to Python 3.14 on supported POSIX systems; check the documentation for the Python version and platform you actually deploy.
Rank #2
Set a context locally when needed
For an explicit choice, use multiprocessing.get_context("forkserver") (or another supported method) and pass that context through the pool’s mp_context argument. Prefer a local context to changing a global start-method setting when only one part of an application needs a choice. Python advises library authors to let users supply a multiprocessing context, and warns that synchronization objects created under different contexts may not be compatible.
The ProcessPoolExecutor also accepts max_tasks_per_child to replace workers after a configured number of tasks. Its default is no limit; when no context is supplied, enabling it selects spawn, and it is incompatible with fork. Consider that worker replacement changes startup costs and should be tested with the context you intend to use. See the executor documentation.
Measure performance on the workload you need to run
A process pool can run work on multiple processors and avoid the GIL limitation described in Python’s multiprocessing introduction. It also adds process startup and interprocess communication costs. Python describes spawn startup as comparatively slow and recommends avoiding large transfers between processes; manager-based proxy sharing is more flexible but slower than shared memory.
The official documentation does not establish a general speedup, benchmark result, or task-size threshold that applies across Linux applications. Do not assume a process pool will be faster for every job. Compare a sequential baseline with the candidate process configurations using the same representative workload, input sizes, and machine.
Record enough detail to make a comparison useful
- Measure end-to-end latency and throughput, and distinguish pool startup from steady-state work.
- Record Python version, start method, worker count, machine and workload characteristics, and whether startup is included.
- Track serialization and data-transfer volume, since moving large inputs or results between processes can consume the benefit of parallel work.
- Observe event-loop responsiveness as well as total work time; the goal is to keep CPU-heavy work from blocking the application’s asynchronous coordination.
These are engineering measurement recommendations based on the documented costs, not a benchmark protocol or performance guarantee published by Python.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Make process communication and shutdown part of correctness
Processes do not share ordinary Python state in the same way as coroutines in one process. Multiprocessing queues and pipes serialize values, so avoid unnecessary shared state and bulk data transfers. Use the simplest communication design that fits the work.
- Join processes you start. On POSIX, an exited process that has not been joined can remain a zombie; Python calls explicit joining good practice.
- Drain queued output before joining producers. A process that has placed data on a multiprocessing queue can wait for its feeder thread to flush buffered output. If the parent joins before consuming that output, the program can deadlock.
- Prefer orderly shutdown to routine termination. Python warns that terminating a process while it uses a lock, semaphore, pipe, or queue can leave that shared resource broken or unavailable. Avoid abrupt termination when shared resources may be in use.
These lifecycle cautions come from the multiprocessing programming guidelines. If your application uses queues or other shared resources, define who consumes pending messages and who joins each producer before implementing shutdown.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Handle worker failures and cancellation in the parent
If a worker in a ProcessPoolExecutor terminates abnormally, the executor raises BrokenProcessPool. Surface that failure to the application rather than treating it as a successful result. Decide which tasks, if any, are safe to retry: whether an operation can be repeated without unwanted side effects depends on the application. Then close or recreate the pool according to the application’s recovery design.
Cancellation and shutdown behavior should also be an explicit part of that design. Specify what the parent does when a caller stops waiting, how outstanding work is treated, and how children and shared resources are cleaned up. Do not rely on abrupt process termination as a general substitute for this policy.
Test async behavior and real process behavior separately
Use an async-aware test framework for coroutine behavior. Python’s unittest.IsolatedAsyncioTestCase accepts coroutine test functions, creates an event loop for each test, and cancels remaining tasks at the end. Async unit tests alone do not establish that process creation, serialization, or cleanup works in the deployed environment; add process integration tests for those paths.
Process integration test checklist
- Run the actual supported start context, including each context your application claims to support.
- Submit module-level worker functions with representative picklable inputs and verify returned results.
- Exercise successful completion and a worker exception or abnormal exit; verify the parent observes and handles failure.
- Test cancellation and shutdown through the application’s actual coordination path.
- If using queues, verify pending output is drained before producers are joined.
- Verify child processes are joined and application-owned resources are cleaned up after success and failure.
Testing only one context is not evidence that another will work: start methods differ in import and inheritance behavior, contexts can have incompatible synchronization objects, and worker replacement has its own restrictions. Keep performance tests separate from correctness tests, and report their environment and whether startup time is included.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
