What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The shortest path to GPU inference on Modal is to declare the container image, Python dependencies, GPU type and model function in one Python file, then run that file with modal run. When the function works, modal deploy turns it into a persistent application. For HTTP traffic, expose the function through a Modal Server or web-server function; for MLflow models, the Modal plugin can package artifacts, place them in a Volume, create FastAPI routes and return an HTTPS endpoint.
This approach moves container building, GPU provisioning and autoscaling into Modal while you pay for active resource usage. You still have to make the model fit the selected GPU, control cold-start behavior, persist large weights sensibly and validate the deployed API under your real concurrency pattern.
As an Amazon Associate I earn from qualifying purchases.
What a Modal GPU deployment actually contains
A Modal application is infrastructure described in Python rather than a manually maintained virtual machine. The file defines four things that matter to inference:
- Image: the operating-system base and installed packages.
- Execution unit: the function or server that receives work.
- GPU: the accelerator attached to that unit, such as an H100 or A100.
- Invocation model: an on-demand function for jobs and experiments, or a persistent HTTP service for API traffic.
Modal builds the image, starts a compatible container when work arrives, attaches the requested accelerator and tears resources down when they are no longer needed. Its documentation describes this as serverless execution and per-second usage billing: Modal operates the underlying hosts while your code specifies the environment. GPU availability, limits and prices can change, so treat the GPU name in a file as a request, not a permanent capacity guarantee.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
The distinction between a one-off invocation and a deployed service is important. modal run is the development and verification path; modal deploy registers a persistent application that can expose HTTP routes and remain available for callers.
Prerequisites and account setup
- Create a Modal account. GPU execution requires a valid payment method on the account.
- Install the SDK. In the Python environment where you will edit the application, run
pip install modal. - Authenticate the CLI. Run
modal setupand complete the browser-based authentication flow. - Prepare a local project. Keep the Modal Python file, any application code, a dependency lockfile if you use one, and a clear model-configuration file together. Do not rely on packages that happen to be installed on your laptop; the remote image is the runtime that matters.
Run the setup commands from the same environment you will use to invoke the application. If your shell cannot find modal, activate the intended virtual environment or install the package into the interpreter that owns your project.
Build the smallest working GPU application
The official minimal pattern starts with a modal.App, creates a Debian-based image, installs the Transformers framework with its PyTorch extra, and attaches an H100 to the inference function. The model is initialized inside that remote GPU container, not on the machine issuing the command.
import modal
app = modal.App("text-classifier")
image = (
modal.Image.debian_slim()
.uv_pip_install("transformers[torch]")
)
@app.function(gpu="h100", image=image)
def classify(text: str):
from transformers import pipeline
classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
device_map="cuda",
)
return classifier(text)
This example demonstrates the deployment shape rather than a production optimization. Loading the pipeline inside the function guarantees that CUDA-dependent initialization happens in the GPU-backed container. In a real service, avoid rebuilding the model for every request: initialize it in a container-startup or lifecycle hook supported by the SDK version you have installed, then reuse the in-memory object for subsequent calls in that warm container.
Why device_map="cuda" matters
Transformers must place the model on the accelerator. A CPU-only image or a function that initializes before the GPU is attached can produce device errors, silently move work to the CPU or exhaust host memory. Keep GPU-specific imports and initialization in the remote function or its supported startup hook, and verify that the framework version in the image supports the selected CUDA runtime.
Pin dependencies for repeatable builds
Use explicit versions for the model framework, tokenizer and serving libraries when reproducibility matters. Rebuild the image after changing dependencies; an already-created image will not automatically include a package added to your local environment. Large optional libraries should be installed only when the model needs them, because every extra layer increases build time and may increase cold-start transfer time.
Package model code, dependencies and weights
There are three common weight-delivery strategies. Select one deliberately instead of allowing a production container to depend on an undocumented local cache.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallDownload weights during startup
The function can download a model from its registry the first time a container starts. This keeps the image smaller and makes model updates independent of image builds, but the first request pays for network transfer and startup. Use this pattern for experimentation or models that are small enough to initialize quickly. Cache the downloaded files in the container when the platform reuses that container; caching does not make the data durable across all future containers.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Bake weights into the image
Installing or copying weights during image construction makes startup more predictable after the image is available. The trade-off is a large image, slower builds and a new image build whenever weights change. This is practical when the artifact is stable and startup latency is more important than build speed.
Store weights in a Modal Volume
A Volume separates model artifacts from the image and is useful when several server instances need the same files. Mount the Volume using the current Modal SDK syntax, upload or materialize the weights once, and have the server read them from the mounted path. Confirm that the model code handles a missing or partially populated Volume; a deployment should fail clearly rather than attempt an expensive download on every request.
For all three approaches, package tokenizer files, configuration, custom Python modules and any generation templates along with the weights. A checkpoint without its tokenizer or preprocessing code is not a deployable model.
Validate locally with modal run
Save the application as, for example, inference.py, then invoke it from the project directory:
modal run inference.pyUse this phase to check four failure boundaries before exposing an endpoint:
- The image builds and all imports resolve.
- The requested GPU is available to your account and the model initializes without an out-of-memory error.
- Representative inputs produce the expected schema, labels or generated text.
- Initialization and inference complete within the timeout and memory limits you intend to use.
modal run is an invocation of your development entry point; it is not a permanently addressable production service. It is therefore the right place to print tensor shapes, inspect one result and test a smaller sample before you commit to an always-on API design.
Deploy a persistent application with modal deploy
When the function behaves correctly, deploy the file:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minutemodal deploy inference.pyModal registers the application and, for HTTP routes, reports a URL. Official examples also expose interactive Swagger documentation for deployed routes. Treat the URL as an operational credential: restrict access with the authentication mechanism your application requires, and do not publish it in source control or client-side code without an access plan.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Deployment does not remove the need to choose a lifecycle policy. A service that scales to zero can save money during idle periods but incurs a cold start when the first request arrives. A warm, continuously available process reduces that first-request delay while consuming resources between requests. Measure both behaviors with the model and concurrency you actually expect.
Serve inference over HTTP
For production request/response traffic, use a Modal Server or a web-server function. Servers are designed for low-latency HTTP handling, can use GPUs, can mount Volumes and Secrets, and can be clustered across multiple GPU nodes. The exact decorator and server configuration names vary with the installed SDK, so check the current API reference when translating the following pattern to your version.
import modal
app = modal.App("sentiment-api")
image = modal.Image.debian_slim().uv_pip_install("transformers[torch]", "fastapi")
@app.function(image=image, gpu="h100")
@modal.fastapi_endpoint(method="POST")
def predict(request: dict):
# Load or reuse the model in the container's supported startup hook.
# Validate request fields before invoking the model.
text = request["text"]
return {"result": run_model(text)}
Keep request validation, model invocation and response serialization separate. Reject malformed JSON, missing fields and inputs that exceed your model’s context or memory budget before they reach the GPU. Return a stable error shape so clients can distinguish validation failures, capacity failures and model errors.
Recommended Free Tools
Use Secrets for credentials
Private model registries, databases and observability systems should receive credentials through Modal Secrets rather than literals in the image or source file. Mount the required Secret on the function or server using the SDK’s current configuration syntax. Rotate credentials independently of model code and ensure logs never print tokens or authorization headers.
Use a Volume for shared artifacts
Mounting a Volume lets multiple server instances read the same model files without rebuilding each image. Plan updates as an atomic release: upload a complete version to a new directory, validate it, then change the configuration to point to that version. Avoid overwriting files that a live process may be reading.
Deploy an MLflow model with the Modal plugin
If your model is already packaged as an MLflow artifact, the Modal deployment plugin can automate much of the serving setup. Its workflow extracts model artifacts and dependencies, uploads the files to a Modal Volume, generates FastAPI endpoints, builds the container, deploys it and returns an HTTPS endpoint.
- Install and configure the MLflow and Modal tooling in the environment used for deployment.
- Point the plugin at the MLflow model artifact and its dependency specification.
- Let the plugin place artifacts in the selected Modal Volume rather than downloading them separately in every request.
- Review the generated image and endpoint configuration, especially the Python and framework versions.
- Deploy and test the returned HTTPS URL with a representative payload.
The generated interface includes /invocations and /predict_stream FastAPI endpoints. Confirm which endpoint your client should call, the expected MLflow input schema and whether streaming is appropriate for your model. The plugin reduces boilerplate, but it does not decide whether the chosen GPU has enough memory or whether your preprocessing code is safe for concurrent requests.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Choose a GPU and serving architecture
Do not select an accelerator solely by name. Evaluate the complete workload: model weights, tokenizer, runtime allocations, attention or KV cache, batch size and the concurrency you need.
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
| Decision axis | Question to answer | What to measure |
|---|---|---|
| Model memory fit | Does the checkpoint and runtime fit without paging or allocation failures? | Peak GPU memory during initialization and the longest realistic request. |
| Latency | How quickly must a single interactive request complete? | Cold-start time separately from warm-request time; report p50 and tail latency. |
| Throughput | How many requests or generated tokens must the service process? | Requests per second or tokens per second at the target batch and concurrency. |
| Scaling behavior | Is traffic bursty, steady or continuously high? | Scale-from-zero delay, queue depth and behavior when several containers start together. |
| Operational complexity | How much infrastructure should your team manage? | Container, autoscaling, rollout and GPU-host operations delegated to Modal versus self-managed infrastructure. |
| Cost | What does the workload consume while active? | GPU, CPU and memory seconds, including startup and idle-warm time. |
Modal notebook documentation describes configurations with up to eight A100s or H100s. That limit and the set of available GPU types are current capabilities, not a promise that every region or account can obtain them at any moment. Multi-GPU configurations are useful when a model cannot fit on one device or when parallelism improves throughput, but they add communication and placement complexity.
Functions versus Servers
Use a regular GPU function for batch jobs, experiments, asynchronous work or traffic that can tolerate startup. Use a Server when clients need an HTTP contract and predictable handling of concurrent requests. A clustered Server is the appropriate direction when one process or one GPU cannot provide the memory or throughput required, provided your model supports the needed parallelism.
Understand cost without relying on stale numbers
Modal’s billing is usage-based: active CPU, memory and GPU time are the important drivers. A scale-to-zero function generally avoids GPU charges while no container is running, whereas a warm service continues consuming resources between requests. Image builds, weight downloads and failed initializations can also affect how much time a deployment spends in active infrastructure.
Exact GPU prices and availability are volatile. Check Modal’s current pricing page immediately before committing to a capacity plan, and record the region, GPU type, memory allocation and date of any estimate. A benchmark on one accelerator is not a price quote for a different model, batch size or traffic pattern.
Cold starts, batching and concurrency
Cold-start latency includes image retrieval, dependency import, weight access and model initialization. Reduce it by keeping images focused, placing large artifacts in a fast shared Volume when appropriate, and avoiding repeated model construction per request. The correct lifecycle hook depends on the SDK version; verify that initialization occurs once per warm container rather than once per invocation.
Batching can improve throughput but may increase the latency of the first item in a batch. Set a maximum batch size and a maximum wait window, then measure tail latency under the expected arrival rate. For generative models, concurrency also increases KV-cache memory, so a configuration that succeeds for one request may fail under parallel load.
Benchmark at least these cases:
- First request after scaling from zero.
- Warm single-request latency.
- Target concurrent requests with realistic input and output lengths.
- Long-running load until autoscaling has stabilized.
- Deployment and model-update time, including artifact transfer.
Modal has published a TensorRT-LLM example exceeding 25,000 output tokens per second on a single NVIDIA H100. That is a configuration-specific example, not a universal performance promise; the model, engine, batching and measurement method determine whether your result is comparable.
Troubleshoot common deployment failures
Package or import errors
If a module imports locally but not remotely, it is missing from the Modal image. Add it to the image’s dependency installation, rebuild and run the file again. Check that the package name used by the installer matches the import name and that optional framework extras, such as the PyTorch extra for Transformers, are included.
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
CUDA or device errors
Verify that the function has a GPU decorator, the framework build is compatible with the image’s CUDA runtime and the model is explicitly placed on CUDA. Keep initialization inside the remote GPU function or its startup hook. A CPU-only local test cannot prove that the remote CUDA stack is correct.
Out-of-memory errors
Measure peak memory during model load and inference, not just the checkpoint size. Lower batch size, shorten input or output limits, use a smaller or quantized model when supported, or select a GPU with more memory. If the model still cannot fit, consider a multi-GPU Server and confirm that the model framework supports its parallelism strategy.
Every request downloads the model
This usually means weights are initialized inside the request path without a persistent cache or Volume. Move initialization to the container startup lifecycle, mount a durable artifact location, and verify that the process reuses the loaded object for warm requests.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →The deployed URL works but clients receive errors
Check the HTTP method, JSON shape and route name first. For MLflow deployments, verify whether the client is calling /invocations or /predict_stream and whether its payload matches the generated schema. Inspect server logs for validation failures separately from model exceptions, and return structured status codes rather than exposing raw stack traces.
Deployment cannot obtain the requested GPU
GPU capacity and availability can vary by region and time. Try an alternative supported GPU that still satisfies the memory and latency requirements, or change the scaling plan. Do not silently fall back to CPU if the model’s service-level objective requires GPU inference.
Security and production operations checklist
- Keep registry, database and API credentials in Secrets, never in source or image layers.
- Authenticate HTTP requests and apply rate limits before expensive generation work.
- Log request identifiers, latency, status and model version; redact prompts, tokens and personal data as required by your policy.
- Version model artifacts and image dependencies so a rollback can restore both code and weights.
- Set input-size and output-size limits to prevent accidental or abusive GPU exhaustion.
- Test deployment from a clean client, not only from the developer’s authenticated shell.
- Monitor cold starts, warm latency, GPU utilization, memory headroom, queue depth and error rate.
End-to-end deployment checklist
- Install the SDK with
pip install modaland authenticate usingmodal setup. - Define a
modal.Appand an image containing every runtime dependency. - Attach a GPU in the function or Server declaration.
- Load the model remotely, place it on CUDA and verify one representative prediction.
- Choose a weight strategy: startup download, image-baked artifact or Modal Volume.
- Run the file with
modal run your_file.pyand test failure cases. - Select a Server or web-server function when callers need HTTP.
- Deploy with
modal deploy your_file.py, then test the reported URL and Swagger documentation where available. - Measure cold and warm latency, throughput, concurrency, memory and active-resource cost.
- For MLflow, verify the generated Volume, image, endpoint paths and input schema before handing the URL to clients.
Frequently Asked Questions
What is the difference between modal run and modal deploy?
modal run invokes the application for development and validation. modal deploy registers a persistent application that can expose HTTP endpoints for ongoing use.
Can a Modal Server use more than one GPU?
Yes. Modal’s Server model supports GPU-backed deployments and can be clustered across multiple GPU nodes. The model framework must support the parallelism approach you choose.
Free tools Windows power users keep installed
One-click scans. No signup required.
How should I store large model weights?
Use a Modal Volume when you want artifacts separate from the image and readable by multiple server instances. Ensure the files are complete and versioned before switching live processes to a new release.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
