To run an open-weight model more safely, choose a compatible local inference runtime, restrict access to its API, and isolate any model-generated code in a separate, stricter execution environment. A sandbox around an agent does not necessarily contain the model it calls or limit that model’s access to host memory and GPU resources.
How the deployment boundaries fit together
Think of the setup as four separate parts: the model artifact, the inference runtime, the execution boundary, and the network boundary. Keeping them distinct helps avoid treating “sandboxed” as a guarantee that every component is isolated.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
MINISFORUM MS-02 Ultra Workstation Mini PC, Intel Core Ultra 9 285HX (24C/24T, up to 5.5GHz), PCIe... | $1,659.00 | Buy on Amazon |
| 2 |
|
GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | $3,649.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
- Model artifact: The weights and their format. Docker Model Runner documents GGUF for llama.cpp and Safetensors for vLLM; it downloads and caches models locally before use. Docker Model Runner
- Inference runtime: The process that loads the weights and handles prompts. llama.cpp and vLLM differ in supported formats, platforms, and workload strengths.
- Execution boundary: The container or operating-system sandbox around a model engine, agent, or code-execution process. An agent sandbox may call a model service that runs outside it.
- Network boundary: The rules that determine which clients can reach the model API and where code-execution workloads can connect outbound.
A typical data path looks like this:
Client or agent → permitted model API → inference runtime → model weights
└→ generated-code tool → stricter execution sandbox
Keep administrative access in mind as well as prompt-and-response traffic. Docker says that any client able to reach the Docker Model Runner API can pull, load, and run models, as well as submit inference requests. Docker Model Runner: networking and security
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesChoose an inference runtime for your hardware and workload
Docker’s guidance is a useful starting point, not a benchmark or a guarantee for every model. Check the current platform matrix, model requirements, and runtime version before deploying.
#1 Best Overall
- High-Performance AI Processor：The MS-02 Ultra features an Intel Core Ultra 9 285HX (24C/24T, up to 5.5 GHz, 13 TOPS NPU), delivering fast and efficient performance for AI inference, algorithm development, and media workloads. A PCIe x16 expansion slot supports desktop-class GPU upgrades for advanced model training and accelerated computing tasks. It's ideal for creators, engineers, and teams handling intensive parallel workloads.
- 4 × M.2 PCIe 4.0 + 4 × DDR5 SODIMM slots:Four DDR5 SODIMM slots support up to 256 GB of memory, while ECC helps maintain data integrity in mission-critical environments. Four PCIe 4.0 M.2 slots support up to 24 TB of storage, supporting RAID 0/1/5/10, combining high-speed performance with data protection. It allows for the creation of independent scratch disks, media libraries, and project drives, providing high-throughput for production workflows.
- PCIe & USB 4.0 v2: Up to three PCIe slots can be equipped, including a dual-slot x16 GPU. The main slot supports PCIe 5.0, meeting the needs of high-bandwidth creative and computing workloads. USB 4.0 v2 (80Gbps) supports high-bandwidth external storage and displays.
- Ultra-fast Networking: Wi-Fi 7 further enhances wireless performance with next-generation speeds and low-latency stability. Intelligent bandwidth switching optimizes throughput in different network environments, ensuring optimal performance for enterprise or local networks. Dual 25GbE ports (providing up to approximately 3.125 GB/s bandwidth, about 25 times faster than traditional 1GbE), enabling seamless large-scale file transfers and parallel computing. 10GbE and 2.5GbE ports, with support for Intel vPro technology, ensure enterprise-grade remote management and deployment flexibility.
- Server-grade thermal architecture: Utilizing a dedicated CPU/GPU airflow design, equipped with a 6-pipe dual-fan cooler, it maintains stable performance even under sustained loads, delivering up to 140W Turbo power while maintaining a 100W TDP, and operating with noise levels as low as 36 dB. An integrated 350W power supply ensures stable and reliable output for demanding computing tasks and fully loaded extended configurations.
| Use case | Documented route | Important constraints |
|---|---|---|
| Local experimentation, CPU-only inference, limited GPU memory, or Apple Silicon | llama.cpp through Docker Model Runner | Uses GGUF. Docker documents CPU-only Linux support and paths involving NVIDIA, AMD, Vulkan, Metal, Apple Silicon, and other platforms; confirm compatibility for your system and model. |
| Concurrent requests or higher-throughput serving | vLLM through Docker Model Runner | Uses Safetensors in Docker’s comparison. The documented setup requires an NVIDIA CUDA GPU and lists Linux x86_64 and Windows with WSL2 as supported. |
| An agent in Docker Sandboxes using a host-local model | Docker Sandboxes with a local model or Ollama provider | Inference runs on the host. The model’s compute and memory are outside the sandbox’s resource limits, and the shared service and model store continue beyond an individual sandbox session. |
Docker recommends llama.cpp for single-user local development, CPU-only systems, limited GPU memory, and Apple Silicon; it recommends vLLM for concurrent requests and higher throughput where the hardware supports it. Those are use-case recommendations, not measured performance guarantees. Docker Model Runner Docker: inference engines
Account for quantization without treating it as a capacity calculator
Docker’s quantization table lists Q4_K_M at approximately 4.5 bits per weight, with “Low” memory usage and “Good” quality; it lists Q8_0 at 8 bits per weight, with “High” memory usage and “Near-original” quality. These are format characteristics, not a complete VRAM-sizing formula. Actual requirements depend on the model and workload, including context length and throughput target. Docker: inference engines
Put the model API behind an access boundary
Docker states that the Model Runner API is not authenticated. Any client with network reachability—including other containers on the same Docker network—can pull, load, and run models or submit inference requests. Restrict the API to intended clients; do not expose it to untrusted networks without an appropriate access-control layer. Docker Model Runner: networking
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Docker describes the isolation differently by platform: on Linux, inference engines run inside a container; on macOS and Windows, they run in sandboxed environments rather than containers. That boundary applies to the inference engine, not automatically to every agent or tool that communicates with it. Docker Model Runner: security and isolation
Rank #2
- EVOLUTION RYZEN AI MAX+ 395 MINI PC - GMKtec EVO-X2 is the next evolution in AI mini PC Ryzen Strix Halo series. Thanks to AMD Simultaneous Multithreading (SMT) the core-count is effectively doubled, to 32 threads. Ryzen AI Max+ 395 has 64 MB of L3 cache and can boost up to 5.1 GHz, depending on the workload. The Ryzen AI Max+ 395 is currently rated as the "most powerful x86 APU" on the market for AI computing.
- AI NPU with XDNA 2 ARCHITECTURE - Powered by 16 “Zen 5” CPU cores, 50+ peak AI TOPS XDNA 2 NPU and a truly massive integrated GPU driven by 40 AMD RDNA 3.5 CUs, the Ryzen AI MAX+ 395 is a transformative upgrade and delivers a significant performance boost over the competition. The Ryzen AI Max+ 395 excels in consumer AI workloads like the llama.cpp-powered application: LM Studio. Shaping up to be the must-have app for client LLM workloads, LM Studio allows users to locally run the latest language model without any technical knowledge required and unleash their creativity and productivity.
- AMD RADEON 8090S iGPU GAMING PC - The AMD Radeon RX 8060S offers all 40 CUs with up to 2.9 GHz graphics clock and uses the new RDNA 3.5 architecture. The powerful iGPU is positioned between an RTX 4060 and 4070 laptop GPU and therefore enables gaming in FHD at maximum details in most demanding games. The 8060S can also utilize the full 128GB pool, which is perfect for running LLMs such as Deepseek 70B Q8, which runs comfortably on this machine.
- EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
- QUAD SCREEN 8K DISPLAY SUPPORT - EVO-X2 AI Mini PC support 4-screen 4K/8K output via HDMI 2.1 (8K@60Hz), DisplayPort 1.4 (4K@60Hz), and dual USB 4 40Gbps Transfer speed (supporting PD3.0/DP1.4/DATA). Ideal for gaming, video editing, and multitasking, it provides expansive and crisp multi-display support.
Do not assume an agent sandbox limits model resources
Docker Sandboxes can run an agent that connects to a local model on the host. In that arrangement, the sandbox’s resource limits do not constrain the model’s host-side CPU, memory, or GPU use. Budget and monitor those resources separately, and treat the model service’s network access as its own boundary. Docker Sandboxes
Isolate model-generated code separately
Generated code is a distinct, higher-risk workload: it may attempt to access files, network destinations, or other resources available to the execution environment. vLLM’s security guide says its reference Python tool runs generated code in a Docker container, but the container has no network isolation by default and inherits the host Docker networking configuration. For production, vLLM recommends a custom code-execution sandbox with stricter isolation guarantees. vLLM: security
Disable tools you do not need. vLLM documents controls for built-in tool availability, including an allowlist for MCP tool labels; when the relevant variable is unset or empty, built-in tools requested through that mechanism remain disabled. Check the names and behavior against the version you deploy. vLLM: security
Set up the deployment in a deliberate order
- Select the model first. Check its current license, artifact format, runtime compatibility, and hardware guidance. Requirements vary by model and intended workload, so the title alone cannot determine a universal GPU, RAM, or storage recommendation.
- Choose the runtime. Consider llama.cpp for broad local hardware and GGUF use, or vLLM when its NVIDIA CUDA requirement and throughput profile fit the workload. Verify the current platform matrix. Docker Model Runner Docker: inference engines
- Obtain and cache the artifact. Docker Model Runner documents pulling models from Docker Hub, OCI registries, or Hugging Face and storing them locally. Use a source you trust and follow the model publisher’s current security and license guidance. Docker Model Runner
- Restrict API reachability. Allow only intended clients to reach the unauthenticated Model Runner API. Docker Model Runner: networking
- Budget host resources separately. If an agent sandbox connects to a host model, the sandbox’s resource limits do not apply to that model process. Docker Sandboxes
- Harden code execution. If you enable generated-code tools, disable unnecessary ones and use a stricter, network-isolated execution design for production rather than assuming the reference interpreter container provides that isolation. vLLM: security
- Verify the deployed versions and settings. Check the runtime version, operating system, drivers, model settings, and current compatibility matrix. These details can be version-sensitive. Docker Model Runner Docker: inference engines
What self-hosting does—and does not—say about data privacy
OpenAI says its open-weight models are designed for infrastructure controlled by the operator, and that OpenAI does not receive data sent to self-hosted models unless the operator explicitly shares it or uses a managed hosting partner. This describes OpenAI’s receipt of data; it does not establish that a particular host, runtime, API, log, or network is secure. OpenAI: open-weight models
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
