October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsSlow PC?RecommendedPC slow today? Run a repair scan before it gets worseResolve common Windows issues and optimize system performance.Scan NowOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk5 min

How to Use Async Multiprocessing on Linux Safely

Use asyncio for coordination, process pools for CPU-bound Python functions, and subprocess APIs for external programs. Learn the Linux start-method changes and safe lifecycle patterns.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use asyncio to coordinate asynchronous I/O, a process pool to run CPU-bound Python functions, and asyncio subprocess APIs to manage external programs. On Linux, check your Python version and multiprocessing start method: Python 3.14 changed the POSIX default from fork to forkserver. The safe choice depends on how your application starts, what it passes to workers, and whether it needs to launch Python functions or other programs.

First, choose the kind of work you need to run

“Async multiprocessing” can describe two different patterns. In the first, an asyncio event loop submits Python functions to a process pool and awaits their results. In the second, asyncio launches and monitors external subprocesses. These approaches solve different problems; neither means that CPU-heavy synchronous work should run on the event-loop thread.

Approach Use it for What asyncio manages
ProcessPoolExecutor with loop.run_in_executor CPU-bound Python callables that can be imported and whose arguments and results can be serialized as required by the selected start method. The future representing the submitted function’s result. The worker runs in another process; it does not run an asyncio coroutine for you.
asyncio.create_subprocess_exec Running a known executable with an argument list. Asynchronous communication with the child and waiting for it to finish.
asyncio.create_subprocess_shell Commands that genuinely need shell syntax. Asynchronous management of the shell process; your application must also handle shell quoting safely.

For CPU-bound Python code, running a synchronous function directly in the event loop blocks that thread and delays other tasks and I/O. Python’s asyncio development guide advises that blocking CPU-bound code should not be called directly and recommends using an executor, including a process pool when appropriate.

Check the multiprocessing start method before relying on defaults

On Linux, the default changed with Python 3.14. The Python 3.14.8 multiprocessing documentation says forkserver is now the default on POSIX, which includes Linux; fork is no longer the default on any platform. Older instructions that assume Linux always defaults to fork are therefore version-dependent.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
#1 Best Overall
Lenovo IdeaPad Slim 3 Linux Laptop, 15.6" FHD Touchscreen Laptop, 8-Core AMD Ryzen 7 5825U, 16GB RAM, 512GB SSD, Keypad, SD Card Reader, Stylus Pen + External Portable SSD + USB Hub, Linux Ubuntu OS
  • Powerful Linux Laptop: This IdeaPad Slim 3 Laptop comes pre-installed with Ubuntu Linux, offering fast performance, robust security, and a clean, user-friendly experience. Enjoy full customization, seamless hardware compatibility, and access to thousands of open-source apps. Whether you're working, creating, or coding, it's built to keep up with everything you do.
  • A Multitasking Master: The latest AMD Ryzen 7 5825U processor (up to 4.5 GHz) delivers powerful performance with 8 cores and 16 threads for smooth multitasking. Integrated AMD Radeon Graphics provide crisp visuals for streaming, browsing, photo editing, and casual gaming. With smart machine intelligence, it adapts to your needs for a fast, responsive experience.
  • 15.6" Full HD Display: The IdeaPad Slim 3 boasts an 88% screen-to-body ratio for a floating, edge-to-edge visual experience. TÜV Low Blue Light certification reduces eye strain, making it perfect for long work or study sessions.
  • Military-Grade Durability: The smart IdeaPad Slim 3 combines portability and durability, letting you work, study, and play on the go. With a profile 10% slimmer than the previous generation, it's lightweight yet military-grade rugged, ready for anything, anywhere.
  • Versatile Connectivity: Enjoy the security of a built-in webcam with a privacy shutter. Connect effortlessly with multiple ports: 2x USB A, 1x USB C, 1x HDMI, 1x SD Card Reader, 1x Headphone/Microphone combo. Bundle comes with Stylus Pen, 256GB Portable SSD and 5-in-1 Docking Station.

The start method determines how worker processes are created and affects startup, inherited resources, and code requirements:

  • spawn: Starts a fresh Python interpreter and inherits fewer resources from the parent. It has more startup overhead than fork. Worker functions and arguments must meet the method’s importability and pickling requirements.
  • fork: Creates a child that initially resembles its parent and inherits its resources. Python warns, “Note that safely forking a multithreaded process is problematic.” Since an asyncio application may coexist with other threads, do not treat fork as automatically safe.
  • forkserver: Uses a server process to create workers and is the Python 3.14 POSIX default. It also requires code and arguments suitable for its process-creation and serialization rules.

Python may issue a DeprecationWarning when it can detect multiple threads and fork is selected, starting in Python 3.12. Check the Python version and the context your application actually uses; choose a method deliberately when compatibility or deployment requirements make the default unsuitable.

Rank #2
HP 17 Business Laptop - Linux Mint Cinnamon - Intel Quad-Core i5-10210U, 32GB RAM, 1TB PCIe NVMe SSD + 1TB Storage HDD, 17.3" Inch HD+ (1600x900) Display
  • Intel Core i5-10210U (up to 4.2GHz) - 1TB PCIe NVMe + 1TB HDD - 32GB DDR4 SDRAM
  • 17.3" HD+ (1600x900) Display, Intel UHD Graphics 620
  • Built in HD 720p Webcam with Microphone - Bluetooth Version4.2
  • I/O Ports: 2x USB 3.1 (Data Only), 1x USB 2.0, 1x HDMI, 1x Headphone/Microphone Combo Jack
  • Linux Mint Cinnamon 64-Bit - 6-Row Keyboard w/ Full Numberpad

Run CPU-bound Python functions through a process pool

Define workers at module scope, keep process creation behind the main-module guard, and create the executor in an application scope with a clear shutdown path. This illustrative pattern uses ProcessPoolExecutor with its default context; adapt the context and lifecycle to the Python versions and deployment modes you support.

import asyncio
from concurrent.futures import ProcessPoolExecutor


def cpu_work(value: int) -> int:
    return value * value


async def main() -> None:
    loop = asyncio.get_running_loop()
    with ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(pool, cpu_work, 12)
        print(result)


if __name__ == "__main__":
    asyncio.run(main())

The awaited call returns 144 for this worker. The function runs in a worker process rather than blocking the event-loop thread while it computes. The example does not specify an explicit multiprocessing context, so the selected default can vary by Python version and platform.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #3
Lenovo Business Laptop - Linux Mint (Cinnamon) - Intel i5-1335U, 16GB RAM, 256GB SSD, 15.6" FHD 1920x1080 Display, Full Keyboard, Fast Charging
  • Intel Core i5-1335U Processor (12M Cache, 12 Threads, up to 4.6 GHz) - 256GB Solid State Drive - 16GB DDR4 SDRAM
  • 15.6" FHD (1920x1080) Non-Touch Anti-Glare Display - Intel UHD 620 Integrated Graphics - Stereo Speakers
  • 720p HD Webcam with Privacy Shutter. Integrated Microphone - Intel Dual Band Wireless-AC (2x2) 8265, Bluetooth Version 4.2
  • I/O Ports: 2x USB 3.0, 1x USB 3.1 Type-C 3.1, Headphone/Mic Combo Port, 4-in-1 Card Reader, HDMI, Kensington Mini-Lock Slot
  • Linux Mint (Cinnamon) 64-Bit - Keyboard with Full NumberPad - Fast Charging

Make worker inputs safe to transfer

With spawn and forkserver, submitted functions need to be importable by the child, and arguments must be picklable. Keep worker functions at module scope rather than defining them inside a coroutine or another function. Pass required data explicitly instead of relying on parent-process globals or inherited resources.

If an application or library uses multiprocessing internally, context choices should be compatible across the components that share multiprocessing objects. Python notes that objects from different contexts may not be compatible; for example, a lock created in a fork context cannot be passed to a spawn or forkserver child. Python’s context guidance recommends that libraries allow users to supply a context rather than imposing one.

Make executor shutdown part of application shutdown

The example’s with block scopes the executor and shuts it down after the awaited task completes. In a longer-running application, manage the executor in an explicit application lifecycle and shut it down as part of orderly shutdown; do not create unmanaged pools and leave cleanup to interpreter finalization. If using the lower-level multiprocessing.Pool API, use its context manager or explicitly close or terminate it. The multiprocessing documentation warns that unmanaged pools can hang during finalization.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Launch external programs with asyncio subprocess APIs

If the work is an external command rather than a Python function, use asyncio’s subprocess interface instead of submitting it to a process pool. Prefer create_subprocess_exec when you can name the executable and pass arguments separately:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Best Value
Sale
GMKtec G3S Mini PC Intel N95 Processor (Up to 3.4GHz) 8GB RAM 256GB M.2 SSD
  • 12th Intel Alder Lake N95 Processor – The GMKtec G3 S Mini PC is powered by the 12th Gen Intel N95 processor with 4 cores, 4 threads, 6MB cache and a burst frequency up to 3.4GHz. Compared with N100/N5105/N5100/N5095, the N95 delivers up to 36% overall performance improvement. Perfect for routine tasks, office work, and home entertainment, this compact mini desktop is more convenient than traditional bulky PCs.
  • 8GB RAM & 256GB SSD Storage – Pre-installed with 8GB DDR4 memory and a fast 256GB M.2 2242 SSD, the G3 S mini desktop offers quicker startup, smoother multitasking, and faster file transfers. Enjoy seamless performance whether you’re working on multiple applications, browsing, or streaming content.
  • Rich Interfaces & Connectivity – The G3 S mini computer comes equipped with USB 3.2 (up to 10Gbps), dual HDMI 2.0 (4K@60Hz), and a 3.5mm audio jack. With support for WiFi 5, Bluetooth 5.0, and Gigabit Ethernet (RJ45 1000MbE), it connects easily with monitors, projectors, printers, office equipment, and other peripherals, making it versatile for both home and business use.
  • Dual 4K Display Support – Featuring upgraded Intel UHD Graphics (up to 1000MHz), the G3 S supports 4K video playback and AV1 decoding for a smooth viewing experience. With dual HDMI outputs, you can connect two 4K@60Hz displays simultaneously, enabling efficient multitasking for work and entertainment.
  • GMKtec WARRANTY - GMKtec offers a 1-year limited GMKtec's warranty for each mini PC, starting from the date of the purchase. All defects due to design and workmanship are covered. With a professional after sales team always ready to attend to your needs, you can simply relax and enjoy your mini PC.
import asyncio


async def run_command() -> None:
    process = await asyncio.create_subprocess_exec(
        "python3", "--version",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    stdout, stderr = await process.communicate()
    print("exit status:", process.returncode)
    print(stdout.decode().strip())


asyncio.run(run_command())

communicate() reads the configured output streams and waits for the process to finish; wait() is also asynchronous. Keep a reference to the process object while the child is running. The asyncio subprocess documentation warns that garbage collection of a still-running process object kills the child.

Use a shell only when the command needs one

create_subprocess_shell adds shell parsing, which creates an input-quoting and injection risk. Python assigns the application responsibility for quoting whitespace and special characters to avoid shell injection vulnerabilities. Do not interpolate untrusted input into a shell command. If shell syntax is genuinely needed, construct the command carefully and use shlex.quote() for values that must be inserted into a shell string.

Account for deployment and resource cleanup

  • Frozen executables on POSIX: The multiprocessing documentation says spawn and forkserver generally cannot be used with frozen executables on POSIX. Packaging can therefore affect which start method is practical.
  • Named shared resources: spawn and forkserver use a resource tracker for named resources such as semaphores and shared memory. Abrupt signal termination can leave resources that need attention.
  • Performance expectations: A process pool has startup and data-transfer costs. The official guidance does not establish a universal throughput advantage or benchmark; performance depends on the workload and deployment.

Use the current concurrent.futures reference alongside the multiprocessing and asyncio documentation when choosing executor behavior. For most applications, the practical decision is: submit serializable CPU-bound Python functions to a managed process pool, launch external tools with create_subprocess_exec, and reserve shell execution for commands that truly require shell syntax.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.