How to Run Ollama with Docker Compose on a VPS (CPU, NVIDIA, AMD, HTTPS, and Persistence)

Run Ollama as a single Docker Compose service, publish its API only on the VPS loopback interface, and mount a named volume at /root/.ollama. That arrangement gives you a reproducible container, keeps downloaded models when the container is replaced, and leaves room to add a GPU or an authenticated HTTPS proxy later.

Need a server to try this on? Antihost offers Linux VPS plans with NVMe storage, full root access and a 14-day free trial.

The commands below cover a CPU-only VPS, NVIDIA and AMD GPU variants, Vulkan, model storage, proxy downloads, secure remote access, upgrades, backups, and the failure modes that most often interrupt a deployment. Ollama performance depends on the exact model, CPU or GPU, memory, storage, and workload, so measure those on your chosen host instead of relying on generic speed claims.

#	Preview	Product	Price
1		GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | Gaming PC AI...	$3,649.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

What the deployment looks like

Ollama exposes an HTTP API on port 11434 inside the container. Docker Compose starts the service, maps that port to the VPS, and mounts persistent storage for model blobs, manifests, and related state. A reverse proxy, VPN, or SSH tunnel can then reach the API without making the inference endpoint public.

• Container image: ollama/ollama for CPU, NVIDIA, and the standard Vulkan-capable build.
• Persistent path: /root/.ollama inside the container.
• API port: 11434.
• Default exposure in this guide: 127.0.0.1:11434, so only local processes can connect directly.
• GPU prerequisites: a supported host driver and runtime. The container file alone cannot add GPU support to an incompatible VPS.

Keep the model volume on storage that survives container recreation. A docker compose down followed by docker compose up -d removes the container but does not remove a named volume unless you explicitly delete that volume.

#1 Best Overall

GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | Gaming PC AI Supercomputer, AMD RADEON 8090S iGPU, 4K/8K HDMI 2.1, DP1.4, Dual USB4 Ports, SD Card Reader 4.0, WIFI7, BT5.4

• EVOLUTION RYZEN AI MAX+ 395 MINI PC - GMKtec EVO-X2 is the next evolution in AI mini PC Ryzen Strix Halo series. Thanks to AMD Simultaneous Multithreading (SMT) the core-count is effectively doubled, to 32 threads. Ryzen AI Max+ 395 has 64 MB of L3 cache and can boost up to 5.1 GHz, depending on the workload. The Ryzen AI Max+ 395 is currently rated as the "most powerful x86 APU" on the market for AI computing.
• AI NPU with XDNA 2 ARCHITECTURE - Powered by 16 “Zen 5” CPU cores, 50+ peak AI TOPS XDNA 2 NPU and a truly massive integrated GPU driven by 40 AMD RDNA 3.5 CUs, the Ryzen AI MAX+ 395 is a transformative upgrade and delivers a significant performance boost over the competition. The Ryzen AI Max+ 395 excels in consumer AI workloads like the llama.cpp-powered application: LM Studio. Shaping up to be the must-have app for client LLM workloads, LM Studio allows users to locally run the latest language model without any technical knowledge required and unleash their creativity and productivity.
• AMD RADEON 8090S iGPU GAMING PC - The AMD Radeon RX 8060S offers all 40 CUs with up to 2.9 GHz graphics clock and uses the new RDNA 3.5 architecture. The powerful iGPU is positioned between an RTX 4060 and 4070 laptop GPU and therefore enables gaming in FHD at maximum details in most demanding games. The 8060S can also utilize the full 128GB pool, which is perfect for running LLMs such as Deepseek 70B Q8, which runs comfortably on this machine.
• EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
• QUAD SCREEN 8K DISPLAY SUPPORT - EVO-X2 AI Mini PC support 4-screen 4K/8K output via HDMI 2.1 (8K@60Hz), DisplayPort 1.4 (4K@60Hz), and dual USB 4 40Gbps Transfer speed (supporting PD3.0/DP1.4/DATA). Ideal for gaming, video editing, and multitasking, it provides expansive and crisp multi-display support.

$3,649.99

Buy on Amazon

Plan the VPS before writing Compose

Choose CPU or GPU deliberately

CPU execution requires no GPU stanza and is the simplest way to validate the API. GPU execution can reduce latency for suitable models, but it adds driver, runtime, device-permission, and hardware-compatibility requirements. Check Ollama’s current compatibility information before renting or buying a GPU VPS; support depends on the exact GPU generation, driver, and backend.

Budget memory and disk for the models you will actually use

Model downloads are stored in the persistent volume. Select disk capacity from the models and quantizations you intend to keep, then leave headroom for temporary download space, container layers, logs, and operating-system updates. Do not infer a model’s required memory or speed from its name. Pull one small model first, observe resident memory and disk growth, and only then add larger models.

Decide who can reach the API

A loopback-only bind works when applications run on the same VPS, when an SSH tunnel is sufficient, or when a local reverse proxy terminates HTTPS. For a separate proxy container, put Ollama and the proxy on a private Compose network and publish only the proxy’s ports 80 and 443. Remote clients should use an authenticated HTTPS gateway, a VPN/private network, or a tightly scoped firewall allow-list. An unauthenticated public port 11434 is not an acceptable production boundary.

CPU-only Docker Compose deployment

Create a directory for the service and save this as compose.yaml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama

volumes:
  ollama:

The named volume is the important persistence setting. The loopback address is intentional: Docker still forwards the API to the host, but it does not bind that port to the VPS’s public interfaces.

Validate and start the service

1. Install Docker Engine and the Compose plugin using your VPS distribution’s supported packages. Confirm that your account can run docker commands.
2. From the directory containing compose.yaml, render and validate the file:

   docker compose config

   Fix any YAML or unsupported-key error before starting.

3. Start the container in the background:

   docker compose up -d

4. Watch startup output for image, permission, or filesystem errors:

   docker compose logs -f ollama

   Press Ctrl+C to stop following logs; it does not stop the container.

5. Check the local API version endpoint:

   curl http://127.0.0.1:11434/api/version

   A JSON response confirms that the HTTP API is reachable from the VPS.

6. Download a model into the persistent volume:

   docker compose exec ollama ollama pull llama3.2

7. Run an interactive local session to verify inference:

   docker compose exec ollama ollama run llama3.2

Replace llama3.2 with a model that fits the VPS’s memory and storage. The command is an example of the workflow, not a performance recommendation.

Keep models after container replacement

Compose recreates containers when you change image tags, environment variables, or service definitions. Because /root/.ollama is backed by the named volume, the models remain available after a replacement:

docker compose up -d --force-recreate

Confirm that the volume is still attached by inspecting the service configuration and listing volumes:

docker compose config
docker volume ls

Do not run docker compose down -v unless you intend to delete the named volume and all downloaded models. A volume removal also eliminates the quickest recovery path after an accidental container deletion.

Back up the model volume

Snapshot or back up the disk that contains the named volume when re-downloading models would be costly in time or bandwidth. The exact backup command depends on your storage and backup system; the operational requirement is that the backup captures the volume data while it is mounted and that you periodically test restoring it to a separate Compose project.

Watch disk pressure

Model pulls can fill a small VPS even when the operating system is healthy. Monitor filesystem usage and Docker’s storage directory, set an alert before the disk reaches capacity, and remove unused models deliberately. A full volume can prevent new pulls, upgrades, and sometimes normal application writes.

NVIDIA GPU Compose variant

NVIDIA support requires more than adding a YAML key. Install a supported NVIDIA driver on the VPS, install the NVIDIA Container Toolkit, configure Docker’s runtime, and restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Validate the host’s driver and runtime before debugging Ollama. The host must be able to see the GPU, and Docker must be able to pass it to a test container. If either layer fails, changing the Ollama image will not fix the problem.

Compose service

A commonly supported Compose shape is:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    gpus: all

volumes:
  ollama:

GPU keys have varied across Docker Compose releases. Run docker compose config with the Compose version installed on the VPS and confirm that the resulting service accepts the GPU declaration before pulling a large model. If that release does not support gpus: all, use the GPU syntax documented for that specific Compose release rather than copying an incompatible example.

Verify in layers

1. Verify the physical or virtual GPU and driver on the host with the vendor’s diagnostic tools.
2. Verify that the NVIDIA Container Toolkit can start a GPU-enabled container.
3. Run docker compose config and recreate the Ollama service after changing GPU settings.
4. Inspect docker compose logs ollama while the service starts and while a model is loaded. GPU initialization errors, missing libraries, and permission failures point to the host/runtime layer.

Do not claim that every NVIDIA model runs entirely in VRAM. The usable model size and concurrency depend on available VRAM, system RAM, quantization, and Ollama’s offloading behavior.

AMD ROCm deployment

For supported AMD hardware, use the ROCm image tag and pass the kernel devices documented by Ollama:

services:
  ollama:
    image: ollama/ollama:rocm
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    devices:
      - /dev/kfd
      - /dev/dri
    volumes:
      - ollama:/root/.ollama

volumes:
  ollama:

The VPS still needs a compatible AMD driver and ROCm stack. Check that /dev/kfd and /dev/dri exist on the host, that the Docker service can access them, and that the installed ROCm version is supported by the GPU. Device mapping alone does not install a driver or grant permission.

Using Vulkan devices

Vulkan support is bundled in the standard image and can use compatible devices exposed by the host. It is an option when your hardware and driver support Vulkan but are not using the CUDA or ROCm path.

• Set OLLAMA_VULKAN=0 to disable Vulkan when it causes instability or when you need CPU-only behavior.
• Use GGML_VK_VISIBLE_DEVICES to select which Vulkan devices are visible when the host exposes more than one device.
• Confirm that the host’s Vulkan driver and device nodes are available to Docker before attributing failures to the model.

Because Vulkan behavior is hardware- and driver-dependent, test with a small model and inspect logs before committing the VPS to a larger workload.

Expose Ollama securely over HTTPS

Local reverse proxy on the same VPS

Keep the Compose mapping at 127.0.0.1:11434:11434. Configure your reverse proxy to connect to that local address, terminate TLS on ports 80 and 443, and enforce authentication before forwarding requests. Use a domain name whose DNS record points to the VPS, obtain and renew a trusted certificate, and allow only the proxy’s public ports through the firewall.

Streaming responses need proxy settings that do not buffer indefinitely. Set read and idle timeouts long enough for generation, preserve streaming response headers, and log authentication failures and upstream errors without logging sensitive prompts or generated content by default.

Proxy container on a private Compose network

If the proxy runs in Compose, define a private network and attach both services to it. Do not publish Ollama’s 11434 port at all; publish only the proxy’s 80 and 443 ports. The proxy should use the service name ollama and port 11434 as its upstream. Authentication, TLS policy, rate limits, and request-size limits belong at this boundary.

SSH tunnel for administration

For occasional personal access, an SSH tunnel avoids exposing a public API. Forward a local port to the VPS’s loopback endpoint, then point your local client at the forwarded address. Restrict SSH access with keys, a firewall allow-list, and the account policy you already use for server administration.

Why direct public port forwarding is risky

Port 11434 is an API endpoint, not an authentication gateway. Publishing it on 0.0.0.0 without an access-control layer allows anyone who can reach the port to submit requests and consume your CPU, RAM, GPU, disk, and network. It can also expose prompts and generated data to parties who should not see them.

Proxy outbound model downloads

When the VPS must use a corporate or hosting proxy for model downloads, pass HTTPS_PROXY into the container:

services:
  ollama:
    image: ollama/ollama
    environment:
      HTTPS_PROXY: https://proxy.example.com

If the proxy performs HTTPS interception, install its certificate as a trusted system certificate in the container image or through the deployment method required by your organization. Ollama’s guidance specifically warns against setting HTTP_PROXY; model pulls use HTTPS, and that variable can interfere with client connections.

Operations after the first successful pull

Control image updates

The floating ollama/ollama tag is convenient but can change when you pull a newer image. For reproducible deployments, pin a tested image digest or document an explicit refresh policy. Before an upgrade, back up or snapshot the model volume, pull the new image, recreate the service, inspect logs, and query /api/version.

Use a deliberate restart policy

restart: unless-stopped brings the service back after a daemon restart or transient failure while still respecting an intentional stop. It does not replace monitoring: alert on repeated restarts, failed health checks, disk pressure, and sustained high memory use.

Tune model residency

OLLAMA_KEEP_ALIVE controls how long loaded models remain resident, trading memory consumption for lower latency on repeated requests. The API’s keep_alive parameter can override the server setting for an individual request. Choose a value based on available memory and request patterns, then observe whether models are being evicted or the host is swapping.

Measure before scaling

Record model load time, first-token latency, generation latency, memory use, GPU utilization, disk use, and concurrent-request behavior on the actual VPS. There is no universal token-per-second or monthly-cost figure that applies to every model and host. Keep the test model and workload documented so later image, driver, or model changes can be compared fairly.

Troubleshooting guide

How to choose between CPU and GPU VPS offers

Compare offers using the workload rather than a provider label. The relevant questions are:

Quick Recap

Bestseller No. 1

GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | Gaming PC AI Supercomputer, AMD RADEON 8090S iGPU, 4K/8K HDMI 2.1, DP1.4, Dual USB4 Ports, SD Card Reader 4.0, WIFI7, BT5.4

$3,649.99

A repeatable deployment checklist

1. Choose a VPS with enough RAM, disk headroom, and a supported CPU or GPU backend.
2. Install Docker, the Compose plugin, and—if applicable—the NVIDIA Container Toolkit or AMD driver/ROCm stack.
3. Save the CPU Compose file first and run docker compose config.
4. Start the service, query /api/version, and inspect logs.
5. Pull one small model and measure memory, disk, latency, and accelerator use.
6. Add larger models only after confirming storage and memory headroom.
7. Keep 11434 on loopback or a private Compose network.
8. For remote clients, place an authenticated HTTPS proxy, VPN, or SSH tunnel in front of Ollama.
9. Back up the named volume and test restoration.
10. Document the image version or digest, driver versions, Compose version, model names, and operational limits.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AMD ROCmDocker ComposeHTTPSNVIDIA GPUOllamaself-hostingVPSVulkan