What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To keep a scheduled Python job from creating more native threads than intended, set the thread-limit variables for the BLAS or OpenMP library it actually loads before Python starts, then inspect the pools from the job’s own runtime environment. A scheduler’s CPU allocation and Python’s worker count do not automatically limit every numerical library.
Why can a Python job use more CPU cores than n_jobs?
NumPy can call a multithreaded BLAS backend for linear algebra. Those native threads are created by the linked numerical library, not by Python’s array-call syntax. If an outer layer also launches processes or workers, each worker may start its own native threads, multiplying total concurrency.
As an Amazon Associate I earn from qualifying purchases.
Scikit-learn illustrates the arithmetic with a hypothetical job using 8 CPUs, 8 joblib jobs and 8 OpenMP threads per job: that setup can produce 64 threads. This is an explanatory example, not a performance measurement. Scikit-learn describes the condition as: “Over-subscription happens when a program is running too many threads at the same time.” Scikit-learn’s parallelism guide explains the nested-parallelism issue.
Free tools Windows power users keep installed
One-click scans. No signup required.
The relevant comparison is total concurrent processes multiplied by the native threads each process may use, measured against the CPUs allocated to the task. More threads are not automatically faster: the goal is to place parallelism where it helps the workload without oversubscribing the available CPUs.
#1 Best Overall
How do I limit OpenBLAS threads in Python?
Set the variable for the backend that is loaded in the scheduled process. Put it in the job’s launch environment so it is present before the Python interpreter and numerical libraries start. For example, if the loaded backend is OpenBLAS, set OPENBLAS_NUM_THREADS=1; if it is MKL, use MKL_NUM_THREADS=1. Other relevant controls include BLIS_NUM_THREADS and OMP_NUM_THREADS. The scikit-learn parallelism documentation describes these controls and their interactions.
For a shell-based job launcher, the pattern is to assign the variable in the command environment before invoking the same Python executable the scheduler uses. Adapt the assignment syntax to the scheduler or shell in use; shell syntax is not interchangeable across every scheduler. The available evidence does not establish platform-specific recipes for cron, systemd, Slurm, Kubernetes or cloud batch services.
- Choose the library-specific variable only after identifying the loaded implementation.
- Use
OMP_NUM_THREADSdeliberately: it can affect OpenMP code and may also affect a linked BLAS implementation. - Do not assume that setting one variable limits every native pool. The library and build determine which controls are honored.
Package names alone are not enough to identify the active backend: a pip or conda installation does not prove which BLAS library is loaded at runtime. NumPy’s global state documentation explains that linear algebra may be backed by a multithreaded BLAS library.
How should I combine scheduler CPUs, worker processes and native threads?
Decide which layer should own most of the parallel work. If the application launches many processes, limiting each process’s native pool can prevent the combined concurrency from exceeding the CPUs assigned to the job. If there are few processes and the numerical operation is the main work, a larger native pool may be appropriate. There is no universal best thread count.
Joblib’s loky backend can limit supported third-party native thread pools in its child processes. For an explicit per-worker cap, configure parallel_config(inner_max_num_threads=...). See the joblib parallelism guide. This mitigation applies to joblib-managed workers; it is not a scheduler-wide policy and does not automatically configure independent processes launched by a scheduler or unrelated multiprocessing code.
Account for the actual CPU allocation, the number of simultaneous processes, and the native threads per process. In particular, avoid combining a high outer n_jobs count with unrestricted native parallelism unless you have verified that the resulting concurrency suits the allocated resources.
How can I check which BLAS library NumPy is using?
Inspect the numerical pools from the same Python executable, package environment, inherited variables and launch configuration used by the scheduled job. Scikit-learn documents this command for inspecting pools loaded through NumPy and SciPy:
python -m threadpoolctl -i numpy scipyThe output identifies loaded libraries and their thread-pool information. Compare the reported implementation and thread limits with the settings you intended to apply. Scikit-learn also demonstrates checking how the loaded library responds to OMP_NUM_THREADS; see its parallelism guide and NumPy global state reference.
Best Value
Run the inspection as part of the scheduled job, or through an equivalent launch that reproduces its environment. If an interactive shell reports different pools or limits, first check whether it uses a different Python executable, installed environment or inherited scheduler variable.
Quick Recap
What should I verify before changing the thread limit?
- Allocated CPUs: identify the CPUs assigned to this particular task, rather than assuming the host’s full core count is available.
- Concurrent workers: count outer processes or jobs that can run at the same time.
- Loaded pools: inspect the actual BLAS and OpenMP libraries used by the deployed process.
- Effective limits: confirm the chosen environment variables are honored by those libraries.
- Representative runtime: after confirming the settings, benchmark representative work under the real allocation. No general performance improvement can be promised from a particular thread count.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
