Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Yes—you can run a useful, private LLM entirely inside a Proxmox home lab. The most dependable design keeps Proxmox as the hypervisor, passes a whole GPU through to an Ubuntu virtual machine, and runs Ollama (with Open WebUI if you want a browser) in that guest. A Linux container (LXC) with a shared GPU is lighter, but requires tighter control of host drivers and device permissions.
In practice, your GPU’s VRAM determines which models fit. A GTX 1050 Ti has been used with Qwen2.5 3B in an Ubuntu 24.04 VM; a Qwen2.5-7B-Instruct Q4_K_M build needs about 4.7 GB for the model files. Larger 14B models are commonly sized at 10–12 GB of VRAM, while an RTX 3090’s 24 GB leaves room for longer context and multiple users. These are sizing examples, not performance benchmarks.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
What does a local LLM on Proxmox actually look like?
Proxmox remains responsible for the physical server and virtual machines. One Linux guest owns the inference stack: Ollama provides a local model API, and an optional Open WebUI container supplies a chat interface. Your applications can call Ollama over the private network using its API, so prompts and model responses stay on hardware you control.
The cleanest boundary is a dedicated VM with the entire GPU passed through by PCIe. Proxmox’s documentation describes whole-card passthrough as the common approach for PCI(e) devices and recommends the q35 machine type, OVMF firmware and PCIe mode for compatibility. A passed-through card’s framebuffer cannot be shown in Proxmox’s NoVNC or SPICE console; that matters for desktop graphics, but not for an inference service that you reach through an API, SSH or remote desktop.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Should I use a VM or an LXC container?
Choose a VM when isolation and predictable upgrades matter; choose an LXC when lower overhead and shared access to an existing GPU matter more than a clean boundary.
| Design | Best for | Advantages | Costs and risks |
|---|---|---|---|
| Dedicated VM with whole-GPU passthrough | One inference service, production-like isolation, easier troubleshooting | Guest owns the GPU; host and guest driver stacks are separated; hardware reset and permissions are easier to reason about | Consumes reserved CPU and memory; the GPU cannot be used by other guests while attached |
| LXC with shared GPU devices | Several lightweight services or a lab where the GPU must be shared | Very little virtualization overhead; fast startup; convenient for a local API | Host kernel, driver versions, device nodes and container permissions all have to line up; upgrades can break the setup |
A documented VM pattern uses Ubuntu 24.04, four vCPUs, 12 GB of RAM and a GTX 1050 Ti passed through by PCIe. After a restart, Ollama detected CUDA and served Qwen2.5 3B. A separate homelab build uses a GTX 1080 Ti in its own VM, Ollama models such as Llama 3.1, Mistral, Gemma 2 and Qwen 2.5, and Open WebUI in Docker.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →How do I pass a GPU through to an Ubuntu VM?
Plan the passthrough before installing the model runtime. The GPU must be available to the guest, and the host must retain enough resources to run Proxmox and the remaining services.
- Confirm hardware and firmware. Enable the platform’s IOMMU or virtualization setting in firmware. Check that the GPU and, when applicable, its HDMI-audio function appear as separate PCI devices in Proxmox. Keep a console or SSH path to the host because the passed-through card will not provide a picture in NoVNC or SPICE.
- Create the guest. Install Ubuntu 24.04 (or another supported Linux release), allocate at least four vCPUs and 12 GB RAM for the 3B-class example, and use a q35 machine type with OVMF (UEFI) firmware.
- Add the PCI device. In the VM hardware view, add the GPU’s PCI device, enable PCIe, and include its audio function if the card exposes one. Do not assign the same physical function to another guest.
- Install the guest driver. Inside Ubuntu, install the vendor’s matching GPU driver and reboot. Verify that the guest can see the card before installing an inference runtime; for CUDA hardware, a successful
nvidia-smiresponse is a useful check. - Install Ollama and test locally. Install the current Linux release in the guest, start its service, and pull a model that fits your VRAM. For the documented low-memory example, Qwen2.5 3B is a sensible first test. Restart the service after the driver is ready, then check the Ollama log to confirm that the GPU backend was selected rather than CPU-only execution.
- Add the interface only after the API works. Run Open WebUI in Docker in the same VM or on another trusted host, configure it to use the guest’s Ollama endpoint, and test a short prompt. Keeping the API test separate makes it clear whether a failure is in passthrough, the runtime or the web layer.
Allocate CPU and RAM with headroom for the guest operating system and context processing. A model that fits exactly in VRAM can still fail when a longer prompt, several simultaneous requests or the web interface consumes additional memory.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
How does an LXC shared-GPU setup work?
An LXC setup exposes the host’s GPU device nodes inside a container instead of assigning the card to one VM. It can be efficient, but the host kernel and driver become part of the container’s runtime contract.
- Prepare the host driver first. Install and verify the vendor driver on Proxmox, then identify the render and compute device nodes under
/dev. Keep the host kernel and driver versions fixed until the container is working. - Grant only required devices. Add the GPU device nodes and matching cgroup permissions to the LXC configuration. Map the relevant group IDs into the container rather than running the service as an unrestricted root process.
- Install a user-space runtime in the container. Use a llama.cpp build that matches the host’s GPU stack. A documented example runs Qwen2.5-7B-Instruct Q4_K_M, whose model files are approximately 4.7 GB, and exposes an OpenAI-compatible local API.
- Test from inside and outside. Confirm that the container can enumerate the GPU, then send a request to the API from a separate machine on the private network. If the container starts but falls back to CPU, compare device permissions, library versions and kernel changes before changing the model.
Use LXC when you accept that a Proxmox upgrade or host-driver change can require container maintenance. For a single household assistant, a whole-GPU VM usually offers the clearer recovery path.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHow much GPU VRAM do I need for a home-lab LLM?
Use the model file size as the floor, then reserve VRAM for the context window, runtime buffers, the operating system and concurrent requests. Product names alone are poor sizing guides.
| Workload example | Published memory guidance | What to plan for |
|---|---|---|
| Qwen2.5-7B-Instruct Q4_K_M | Approximately 4.7 GB for the quantized model | A GPU with more than that figure; extra capacity is needed for context and runtime overhead |
| 14B-class models | Indicative guidance of 10–12 GB | At least that much usable VRAM, with additional headroom for long prompts and more than one user |
| RTX 3090 example | 24 GB; an indicative 2026 guide places cards in a $600–800 range | Suitable capacity for larger quantized models, but the price is a market snapshot, not a guaranteed current cost |
Quantization reduces memory at the cost of some quality or capability. If you need a long context window, retrieval documents, tool calls or multiple chats, buy for the workload’s peak rather than the model’s download size. No controlled measurement establishes a particular tokens-per-second rate, power draw or cloud-cost saving for these configurations, so test your own prompts and electricity tariff.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Which GPU vendors work with Ollama and Proxmox?
Support depends on the exact card, kernel, driver and runtime combination—not just the logo on the cooler.
NVIDIA CUDA
The documented GTX 1050 Ti and GTX 1080 Ti VM examples use CUDA successfully after the guest driver is installed and the VM is restarted. Confirm GPU visibility in the guest and watch the Ollama service log for CUDA initialization.
Intel Arc
Intel’s June 2024 guide uses IPEX-LLM and documents environment variables for GPU serving. An Ollama maintainer issue dated September 24, 2024 said Intel GPU support was not officially part of Ollama at that time. If you choose Arc, follow the current Intel-supported runtime path and verify compatibility for your exact Proxmox kernel before purchasing.
AMD Radeon
An AMD article dated October 19, 2026 documents ROCm-based passthrough on current AMD hardware. ROCm versions, supported architectures and Proxmox kernels change, so validate the matrix for your card and distribution rather than assuming that a generic Linux driver is sufficient.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
How do I expose the local model safely on my network?
Keep Ollama bound to a private interface by default. Remote access is a separate security decision from running the model locally.
- Give the VM or container a private LAN address or a restricted virtual network.
- Allow API traffic only from trusted subnets with the Proxmox firewall or the guest firewall.
- Put Open WebUI behind authentication and HTTPS if other household users will connect.
- For access while travelling, use a private overlay such as Tailscale or an authenticated gateway instead of forwarding Ollama’s unauthenticated API directly to the internet.
- Review model and chat logs, because prompts can contain credentials, documents or personal data even when they never leave your property.
A walkthrough published April 2, 2025 combines Proxmox PCIe passthrough, Ollama, Open WebUI and Tailscale. Treat that pattern as a network design example, not a reason to expose the raw model port publicly.
Recommended Free Tools
What should I troubleshoot first when inference is slow or fails?
- GPU is missing in the guest: recheck IOMMU grouping, the q35/OVMF settings, the PCIe flag and whether both the graphics and audio functions were assigned.
- Ollama runs on CPU: verify the guest driver independently, restart Ollama after driver installation, and inspect logs for CUDA, ROCm or other backend initialization errors.
- Model will not load: compare its quantized file size with usable VRAM, then reduce context length or choose a smaller quantization. Leave capacity for runtime buffers.
- LXC broke after a host update: compare kernel, driver, device-node ownership and cgroup rules with the last working configuration. A VM avoids many of these shared-kernel dependencies.
- No picture in the Proxmox console: this is expected for a passed-through GPU. Use SSH, the guest API, Open WebUI or a remote-desktop service for administration.
- Remote requests fail: check the bind address, guest firewall and overlay-network ACLs before changing model settings.
How should I choose between the practical build options?
Start with the service you want, then select the smallest architecture that leaves room for growth.
| Your priority | Recommended starting point | Reason |
|---|---|---|
| Simple, isolated household assistant | Ubuntu 24.04 VM, four vCPUs, 12 GB RAM, whole-GPU passthrough, Ollama and optional Open WebUI | Clear ownership of drivers and devices; easy to snapshot and restore |
| Several lightweight GPU services | LXC with carefully mapped GPU devices and llama.cpp | Lower overhead and shared access, with more maintenance responsibility |
| 7B-class quantized model | GPU with comfortably more than 4.7 GB usable VRAM | Leaves space for context and runtime overhead |
| 14B-class experimentation | 10–12 GB as a minimum planning range; more for long context or concurrency | Matches indicative sizing while preserving headroom |
Snapshot the VM before driver or runtime changes, keep model files on storage with adequate endurance, and document the exact Proxmox, kernel, driver and Ollama versions that work together. That small amount of lab hygiene is often more valuable than chasing a marginally faster model.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Frequently Asked Questions
Can I run a local LLM on Proxmox without a GPU?
Yes, Ollama and llama.cpp can fall back to CPU execution, but interactive speed and practical model size depend heavily on your processor and memory. Treat CPU-only mode as a validation or low-concurrency option rather than assuming it will feel like GPU inference.
Do I need Open WebUI to use Ollama?
No. Ollama exposes an API that scripts and applications can call directly. Open WebUI is an optional browser front end for users who prefer chat history, model selection and shared access controls.
Can two Proxmox VMs use the same passed-through GPU at once?
A whole-card passthrough assignment gives ownership to one guest at a time. Sharing requires a different device-virtualization feature or an LXC-style shared-device design, each with its own hardware and driver limitations.
How do I back up models and conversations?
Back up the VM or container configuration separately from large model files, and export any chat data stored by the web interface. Test a restore on another guest; a hypervisor snapshot is not a substitute for an independently stored backup.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
