To run a local AI model on an NVIDIA DGX Spark, first complete its DGX OS setup, then choose a runtime that supports your model files and the way you plan to use them. Ollama is a straightforward starting point; llama.cpp offers direct control for GGUF models, while vLLM, SGLang, TensorRT, and PyTorch with CUDA are options for more configurable inference deployments. If you want an agent rather than just a model endpoint, NVIDIA also documents a guided NemoClaw setup that uses a local Ollama model.
Understand what you are setting up
A model runtime loads model weights and generates responses, often through a command-line interface or an API. An agent harness adds a layer for workflows, tools, and integrations. It may connect to external services even when the model itself runs locally.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Dell NVIDIA Tesla V100 GPU SXM2 32GB NWWWX by DELL | $854.96 | Buy on Amazon |
| 2 |
|
Gigabyte NVIDIA GeForce RTX 3060 Gaming OC V2 Graphics Card - 12GB GDDR6, 192-bit, PCI-E 4.0,... | $695.00 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
NVIDIA lists Ollama, llama.cpp, TensorRT, SGLang, vLLM, and PyTorch with CUDA among local AI runtime choices for DGX Spark. The right one depends on the model format, memory use, API requirements, desired throughput, and how much configuration you want to manage—not on a single universal ranking. NVIDIA’s DGX Spark product information describes the hardware and its intended local AI use.
Recommended Free Tools
Prepare DGX Spark before installing a runtime
Complete the first-boot process and check NVIDIA’s current system guidance before changing drivers, CUDA, or other software. The DGX Spark documentation hub links to first-boot instructions, OS and component updates, release notes, and recovery information. Follow the current instructions for your system rather than assuming a toolkit or driver command from an older guide is still appropriate.
#1 Best Overall
- GPU Chipset: NVIDIA
- Memory: HBM2
- Programming Interface: CUDA
- Memory Capacity: 32GB
- Slot Compatibility: SXM2
The sources cited here do not establish the current point-release versions of DGX OS, CUDA, Ollama, or each inference backend. Check the applicable release notes and the chosen runtime’s official installation guidance for versions and supported installation steps.
Choose a runtime for your model and workload
| Runtime | When it may fit | What to check |
|---|---|---|
| Ollama | A relatively straightforward way to run a local model; NVIDIA also uses it in its guided NemoClaw agent flow. | Confirm that the model you want is available in a compatible form and that its memory needs suit your workload. |
| llama.cpp | Running GGUF model files with direct control through a local server or command-line interface. | Check current build and CUDA guidance for DGX OS. Community build recipes are not NVIDIA support instructions. |
| vLLM | A configurable inference-server option when its model support and serving interface fit your application. | Verify the model format, current installation requirements, and memory behavior for your model and context length. |
| SGLang | An alternative serving backend for workloads whose model support and API needs match it. | Check current compatibility, setup requirements, and expected behavior for your use case. |
| TensorRT | An NVIDIA-listed runtime option when its supported model path and deployment requirements fit. | Check the relevant NVIDIA documentation for the supported model, precision, and setup path. |
| PyTorch with CUDA | A flexible option when you want to build or manage inference in a PyTorch-based workflow. | Check current CUDA and framework compatibility guidance rather than changing system components by guesswork. |
NVIDIA’s local AI guidance recommends selecting a backend with the operating system, model format, GPU architecture and memory, API needs, and throughput target in mind. These options are not interchangeable: a weight format supported by one runtime may require conversion or a different checkpoint for another.
Select model size and precision with memory in mind
NVIDIA specifies 128 GB of unified memory and up to 1 petaFLOP at FP4 for DGX Spark. Its product information also states inference support for models up to 200 billion parameters. These are NVIDIA capability claims, not a guarantee that every model at that size will fit or perform well. Quantization, context length, runtime overhead, and workload all affect actual memory use and responsiveness. See NVIDIA’s DGX Spark specifications and local AI guidance.
Free tools Windows power users keep installed
One-click scans. No signup required.
NVIDIA suggests Q4_K_M quantized checkpoints as a starting point for llama.cpp and NVFP4 for vLLM or PyTorch. Treat these as backend-specific recommendations to evaluate, not universal compatibility rules or assurances of a particular quality or speed. Shortlist models based on your memory and performance needs, then test them on representative tasks and review the outputs yourself.
Use NVIDIA’s guided NemoClaw route for an agent setup
NVIDIA’s June 1, 2026 walkthrough combines an agent harness, OpenShell sandbox runtime, and a local Ollama model. Its express flow downloads Qwen3.6-35B. This is a guided agent-oriented setup, not a prerequisite for running other local models.
Rank #2
- NVIDIA Ampere Streaming Multiprocessors: Building blocks for the world's fastest, most efficient GPUs, the all-new Ampere SM brings twice the FP32 throughput and improved energy efficiency
- 2nd Generation RT Cores - Experience 2x the 1st Generation RT Cores throughput, plus competitive RT and shading for a whole new level of ray-tracing performance
- 【3rd Generation Tensor Cores】Get up to 2X the throughput with structural sparsity and advanced AI algorithms such as DLSS
- Core Clock: 1837MHz
- WINDFORCE 3X Cooler
- Complete DGX Spark first boot and consult the NVIDIA Spark playbook for the current procedure.
- Run the installer command shown in the playbook:
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash. This installs software; the subsequent flow also downloads model weights. - Follow the installer prompts, accept the applicable licenses, and choose the express installation option if that is the path you want.
- Allow the setup to configure local Ollama and download Qwen3.6-35B, then use the gateway token to open the agent Web UI as described in the playbook.
Installer behavior and instructions can change. Review the current playbook before running the command, and only proceed if you understand and accept the software installation and model download it initiates. NVIDIA’s walkthrough reports up to 2.6× faster inference for Qwen3.6-35B using its NVFP4 checkpoint and vLLM optimizations; that is an NVIDIA-reported result for the described setup, not an independent benchmark or a general performance promise. See the NVIDIA Technical Blog walkthrough.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Check the agent’s privacy and network boundaries
Local inference does not by itself mean an agent operates offline. NVIDIA describes OpenShell as providing sandboxing, access controls, privacy protections, and operational guardrails, while the NemoClaw flow also discusses integrations and configurable external network destinations. Before using an agent with sensitive material, inspect its network policy, enabled integrations, accessible files, and available tools. Restrict those permissions to what the task requires; do not infer that model weights running locally keep every prompt, tool call, or integration on the machine.
For a standalone model runtime, assess the process separately: know which application is making requests, what API it exposes, and whether your own client or surrounding software sends data to remote services.
Validate the setup on your own tasks
Once the runtime is installed and the model is loaded, test it with representative prompts and context lengths before relying on it. Compare response quality, memory use, and speed against your actual needs. NVIDIA’s guidance is to evaluate candidate models on a task-specific dataset and include human review; a model that loads successfully is not necessarily the right model for your work.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
