To run multiple Python jobs in parallel, give each job an identifier and callable, submit it to an executor, then collect its result or failure from the returned Future. Python’s standard-library concurrent.futures supplies the shared interface; you can start with threads, switch to processes when the workload calls for them, and add bounded submission when the input is too large to queue all at once.
Define the runner’s job contract first
A useful runner needs to do more than start work. For every job, decide what identifies it, what function and arguments it receives, when its outcome is reported, and how the pool is shut down. Make those promises explicit before choosing a concurrency backend.
As an Amazon Associate I earn from qualifying purchases.
- Identity: assign a stable job ID or retain the input that identifies the job.
- Outcome: report either the return value or the exception raised by that job.
- Ordering: choose submission order or completion order for reporting.
- Failure policy: decide whether one job’s failure should stop collection, be recorded while other jobs continue, or be aggregated and reported at the end.
- Shutdown: decide who owns the executor and when the caller waits for outstanding work.
Keep result collection in the controlling thread where practical. Worker functions that append to shared collections introduce synchronization and make ownership less clear; collecting each Future’s outcome in one place keeps the basic runner easier to reason about.
Start with the shared Executor and Future interface
The standard-library concurrent.futures API provides an abstract Executor interface implemented by concrete pools. Calling submit(fn, *args, **kwargs) schedules a callable and immediately returns a Future, which represents work that may finish later.
#1 Best Overall
Keep a mapping from each Future to its job ID. That association is essential when jobs finish in a different order from the order in which they were submitted:
from concurrent.futures import ThreadPoolExecutor, as_completed
def run_job(job_id, fn, *args, **kwargs):
return fn(*args, **kwargs)
def run_jobs(jobs):
# Each job is (job_id, callable, args, kwargs).
future_to_job_id = {}
with ThreadPoolExecutor() as executor:
for job_id, fn, args, kwargs in jobs:
future = executor.submit(run_job, job_id, fn, *args, **kwargs)
future_to_job_id[future] = job_id
for future in as_completed(future_to_job_id):
job_id = future_to_job_id[future]
try:
result = future.result()
except Exception as exc:
print(f"{job_id} failed: {exc!r}")
else:
print(f"{job_id} completed: {result!r}")The wrapper is optional; the executor can submit a job callable and its arguments directly. The important part is retaining the Future-to-job mapping. Calling future.result() returns the callable’s value, or raises the exception that callable raised.
Choose how results should arrive
The collection method is part of the runner’s behavior, not just an implementation detail. as_completed() supports completion-order handling; Executor.map() yields corresponding results in input order.
Recommended Free Tools
Completion order with as_completed()
Use this when callers should hear about a fast job without waiting for an earlier, slower submission. For each Future, look up its job ID and call result() inside a try block. Catching Exception at this boundary allows collection of other independent jobs to continue; remove the catch or record failures and raise them later if your policy is fail-fast or aggregate reporting.
Input order with map()
Use executor.map(fn, inputs) when results should correspond to inputs in their original order. A slow early job can delay delivery of later results even if those jobs have already completed. Exceptions are raised when the corresponding result is retrieved, so callers should account for that in their error policy.
Pick a backend for the workload and programming style
ThreadPoolExecutor and ProcessPoolExecutor share the Executor interface, but they execute work differently. Python’s concurrency overview frames the choice around whether work is CPU-bound or I/O-bound and whether the preferred style is event-driven cooperative multitasking or preemptive multitasking. Neither executor promises a universal speedup: benchmark representative jobs with the Python version and hardware you will actually use.
| Option | Typical fit | Important trade-off |
|---|---|---|
ThreadPoolExecutor |
Blocking I/O jobs, such as callables that spend time waiting | Runs threads within one process; start here for a straightforward pool-based I/O runner and measure the real workload. |
ProcessPoolExecutor |
CPU-bound Python computation when separate processes suit the work | Workers need picklable functions and values, and process startup and data transfer are part of the design. |
asyncio |
Event-driven coroutine code | It is a different concurrency model and programming style, rather than a drop-in executor backend for ordinary synchronous callables. |
Choose based on the task and the way the application is structured. For synchronous jobs that block on I/O, threads are a typical simple first option. For computation that can use separate processes, test a process pool while accounting for its portability constraints. No worker count or throughput figure is appropriate for every runner; tune against your actual job duration, input size, and target environment.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Bound the work when inputs may be large
Submission volume affects memory use and how much work is already queued when you need to stop. In the Python 3.13 documentation, Executor.map() collects its input iterables immediately. Avoid feeding it an unbounded or very large source without considering that eager consumption.
A bounded runner keeps only a limited number of jobs in flight, then submits another job as one completes. This controls queued work as well as active work; select the limit based on the application’s memory budget and workload, and measure rather than assuming one value is universally best.
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
def bounded_results(executor, jobs, max_in_flight):
jobs = iter(jobs)
pending = {}
def submit_one(job):
job_id, fn, args, kwargs = job
future = executor.submit(fn, *args, **kwargs)
pending[future] = job_id
for _ in range(max_in_flight):
try:
submit_one(next(jobs))
except StopIteration:
break
while pending:
done, _ = wait(pending, return_when=FIRST_COMPLETED)
for future in done:
job_id = pending.pop(future)
try:
yield job_id, future.result(), None
except Exception as exc:
yield job_id, None, exc
try:
submit_one(next(jobs))
except StopIteration:
pass
# The caller owns executor lifecycle and chooses how to handle each outcome.
with ThreadPoolExecutor() as pool:
for job_id, result, error in bounded_results(pool, jobs, max_in_flight=8):
if error is not None:
print(f"{job_id} failed: {error!r}")
else:
print(f"{job_id} completed: {result!r}")This example yields each outcome as it becomes available and catches ordinary task exceptions so the caller can continue. It uses a fixed in-flight limit; choose an appropriate value for your application rather than treating the example’s 8 as a recommendation. If using a newer API’s buffering feature instead, check the documentation for the exact Python version you deploy.
Make shutdown and cancellation behavior explicit
A with block around an executor shuts it down on exit and waits for pending work to finish. This is convenient when the runner owns the whole batch and should not return before its jobs are done.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Future.cancel() succeeds only if execution has not started; it cannot forcibly stop a running call. Likewise, shutdown(cancel_futures=True) cancels futures that have not started, not work already running. If a job must respond promptly to a stop request, design a cooperative cancellation mechanism into that job rather than relying on Future cancellation to interrupt it.
Make process-pool jobs portable
Process workers run in separate processes, so the functions and values sent to them must be picklable, and the worker process must be able to import the main module. Define worker functions at module level and protect process-launching code in portable scripts with the main guard:
from concurrent.futures import ProcessPoolExecutor
def calculate(value):
return value * value
def main():
with ProcessPoolExecutor() as executor:
print(list(executor.map(calculate, [2, 3, 4])))
if __name__ == "__main__":
main()Do not call Executor or Future methods from a callable running in a process pool; the Python documentation warns that this can cause deadlock. The Python 3.13 documentation also notes that the multiprocessing default start method changes away from fork in Python 3.14. If your application depends on fork, request the multiprocessing context explicitly and test that configuration on the Python versions you support.
Decide whether a local runner is enough
This design handles local concurrent execution. If the requirement is for durable jobs that survive process restarts, scheduled work, or execution across multiple machines, those are separate requirements and call for a system designed around persistence or distributed coordination rather than just an in-process executor.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
