To run vLLM for online inference, launch it with vllm serve, choose a deployment model that fits your infrastructure, and monitor its Prometheus metrics. For customer-level token billing, enable per-request usage reporting and build the account attribution, pricing, and durable usage records in your application: vLLM provides metering inputs, not a complete invoicing or financial ledger system.
How an online vLLM request moves through the system
vLLM’s online serving entrypoint is vllm serve <model>. This differs from using the Python LLM class for offline inference. In the documented V1 online architecture, an HTTP request passes through API server process(es) and engine core process(es) before execution across GPU workers.
As an Amazon Associate I earn from qualifying purchases.
- The API server receives and prepares input. API server processes accept HTTP requests, handle input processing such as tokenization and multimodal loading, and return or stream results to the caller.
- The API server communicates with engine core. The processes use ZMQ sockets to pass work between the request-facing layer and the inference engine.
- Engine core schedules and manages inference. Engine core process(es) run the scheduler, manage the KV cache, and coordinate model execution across GPU workers.
- The response returns to the client. Depending on the endpoint and request mode, the API server returns a response or streams output.
Do not assume that every installation has exactly one API server process. The architecture documentation says the count is normally one, but it scales with data parallelism by default and can be configured manually. The relevant process layout therefore depends on the deployment.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose an API surface that matches the model and task
The online-serving reference documents OpenAI-compatible interfaces for tasks including completions, chat completions, responses, embeddings, audio transcription, and translation. It also documents Anthropic messages and token-count endpoints, among other compatible interfaces. Availability depends on the model and task, and compatibility can change by release; check the documentation for the vLLM version you deploy rather than promising an endpoint based on a different version.
#1 Best Overall
- [ Maximum AI Compute Power ] Dominate complex workloads with the ASUS ESC8000A-E13. This 4U rack server is a powerhouse engineered for mass-scale AI, machine learning, and deep training. Featuring support for dual AMD EPYC 9005/9004 processors and up to eight dual-slot GPUs, it delivers the raw computational muscle required to train LLMs and run complex simulations effortlessly. Accelerate your data science pipeline and transform raw data into actionable intelligence faster than ever.
- [ Advanced Thermal Efficiency ] High performance demands elite cooling. The ESC8000A-E13 features a cutting-edge aerodynamic design with independent CPU and GPU airflow tunnels. Equipped with redundant hot-swap fans and optimized for liquid cooling integrations, this 4U server ensures maximum uptime under heavy, sustained workloads. Keep your data center running cool, quiet, and highly efficient while preventing thermal throttling during mission-critical enterprise operations.
- [ Scale with Flexible Storage ] Future-proof your infrastructure with unmatched storage and expansion flexibility. This offers comprehensive front-panel drive bays supporting Gen5 NVMe, SAS, or SATA drives alongside multiple PCIe 5.0 slots. Designed as a high-density 4U server capable of housing eight dual-slot GPUs: NVD H200, RTX PRO 6000 Blackwell, RTX PRO 4500 Blackwell or AMD Instinct MI350P PCIe Card, each supporting up to 600 watts.
- [ Enterprise-Grade Reliability ] Minimize downtime and secure your ecosystem with server-grade redundancy. The ESC8000A-E13 is built for 24/7 continuous operation, boasting 2+2 redundant (3200W total) 80 PLUS Titanium power supplies and integrated ASUS ASMB11-iKVM for comprehensive out-of-band management. Ideal for cloud service providers, rendering farms, and large enterprise infrastructure, it combines robust physical hardware with smart remote monitoring to safeguard your digital assets.
- [Reliability Guaranteed] Shop with total peace of mind knowing that every new computer component we sell is backed by our EPC 3-year warranty. Whether you are investing in high-speed DDR5 RAM or a powerhouse GPU, we protect your build against defects and performance failures. We stand firmly behind the quality of our hardware, ensuring that your setup remains fast, stable, and secure for years to come.
Operational endpoints documented by vLLM include /health, /load, /v1/models, and /metrics. Keep the serving API and operational interfaces within the authentication and network controls appropriate to your environment.
Choose a production deployment approach
The vLLM Production Stack describes three Kubernetes-oriented approaches. It does not rank them by performance or identify one universally best option; choose based on how much infrastructure control, operator integration, and routing complexity your team needs.
| Approach | What it provides | When it may fit |
|---|---|---|
| Helm chart | The documented standard Kubernetes deployment method, with configuration for models, resources, and routing. | Teams that want to deploy and configure the serving stack through Helm and manage its settings directly. |
| Kubernetes CRDs | Kubernetes-native custom resources for more advanced configuration and operator workflows. | Teams whose infrastructure already relies on Kubernetes operators or needs configuration expressed through custom resources. |
| Gateway API inference extension | An advanced routing approach using agentgateway, the Gateway API Inference Extension, and the llm-d Router to direct requests among pools of vLLM model servers. | Teams that need routing across model-server pools and are prepared to operate the additional routing components. |
Compare the options against your existing Kubernetes capabilities, required routing across pools, scaling needs, and appetite for managing serving infrastructure. The project overview is descriptive; it does not establish quantitative performance differences between these approaches.
Monitor fleet health and capacity with Prometheus metrics
vLLM exposes a Prometheus-compatible /metrics endpoint. Its documented V1 metrics include signals for running requests, KV-cache usage, prefix-cache queries and hits, prompt and generation token counts, request success, request prompt and generation token histograms, time to first token (TTFT), inter-token latency, per-request time per output token, end-to-end latency, prefill time, and decode time.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
Use aggregate counters and engine-state metrics to see how the service is being used and how it is operating. Use request-level histograms to understand request sizes and latency distributions, including SLO-relevant behavior. The vLLM documentation also provides an example using Prometheus for collection and storage with a Grafana dashboard for visualization. These fleet-level metrics describe service activity; they do not, by themselves, attribute usage to individual customers.
Keep latency metrics distinct
Metric names that sound similar can describe different observations. Inter-token latency is recorded per streamed output event. Request-level time per output token (TPOT) is recorded once per completed request and is calculated from end-to-end latency, TTFT, and output-token count. Requests that generate no more than one token are recorded with TPOT equal to zero. The vllm bench serve benchmark excludes those requests, so its TPOT statistics may not match a graph built from the request-level metric.
When building dashboards or comparing results, state which metric is shown and how it is aggregated. A per-event latency distribution is not interchangeable with one observation per finished request, and benchmark calculations may apply different inclusion rules.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteControl histogram cardinality
Custom histogram bucket boundaries increase Prometheus time-series cardinality: each bucket adds a series for each metric and label combination, and additional deployments multiply those combinations. Keep custom bucket lists short and apply them only to metric families you actively monitor to limit storage, scrape size, and query cost.
Rank #3
- AI-Optimized: Designed to support up to 4 GPUs, it is perfect for handling intensive AI and machine learning tasks, ensuring high performance and scalability for advanced computational needs.
- Intelligent Storage: Equipped with 8 hot-swappable 3.5" SATA/SAS drives (12Gbps), featuring SGPIO and temperature control, it ensures efficient data management and reliable storage performance.
- Robust Cooling: The system includes 3x 12038 hot-swap PWM fans and 2x 8038 rear fans, providing advanced thermal management to maintain optimal temperatures and ensure stable operation under heavy workloads.
- Rack-Ready: Comes with a pre-installed rail kit, allowing for quick and easy installation in standard 19-inch server racks, making it ideal for data center environments and enterprise setups.
- Versatile Connectivity: Offers USB 3.0 and the latest USB 3.2 Type-C ports, ensuring high-speed data transfer and compatibility with a wide range of peripherals and devices for enhanced connectivity options.
Return request-level usage for token accounting
For supported API responses, vLLM can return usage counts and timing data on a per-request basis. The Per-Request Metrics documentation, versioned v0.30.0 and dated August 20, 2026, describes this capability as useful for billing, SLA monitoring, and latency analysis. Enable it with --enable-per-request-metrics.
Usage fields include usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens. Responses can also contain timing fields such as TTFT, generation time, queue time, mean inter-token latency, and output tokens per second; timing values may be null when unavailable.
Streaming and multi-sequence conditions
- Streaming: usage and metrics appear on the final usage chunk. A streaming client must request
stream_options.include_usage: true, unless the server is configured to force usage inclusion with--enable-force-include-usage. - More than one generated sequence: timing metrics are suppressed when
n > 1, because they cannot be accurately attributed across multiple sequences. Usage token counts remain accurate in this case. - Completion requests with multiple prompts: timing metrics are omitted because they cannot be attributed to one prompt.
- High concurrency: calculating per-request statistics can add non-negligible CPU overhead. Benchmark the actual workload before enabling the feature in production.
These response fields complement server-level Prometheus aggregates. They provide a request-level usage input, while aggregate metrics remain useful for fleet monitoring and capacity planning.
Design billing rules in the application layer
A token count is not yet a charge. The vLLM documentation does not prescribe prices, customer or tenant attribution, treatment of cached prompt tokens, retries, cancellations or errors, durable financial records, invoice generation, or retention rules. Those are application and business-policy decisions. A fleet-wide aggregate counter cannot produce reliable per-customer records when customer-level invoices are required.
Rank #4
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
Define the policy before calculating charges
Decide explicitly which usage categories are billable and how each request is assigned to an account. For example, the application must determine whether prompt and completion tokens have separate rates, how cached prompt tokens are treated, and which outcome applies when a request is retried, cancelled, or fails. vLLM’s usage fields supply counts; they do not settle those policy questions.
Persist attributable records and version rates
At the point where your application associates a response with a customer, retain the usage data needed by your billing rules alongside the account or tenant identity and the model or service context required to interpret it. Version the applicable rates so a later rate change does not make historical usage ambiguous. Define how records are retained, reconciled, and corrected, and how duplicate handling works for retries. These are implementation recommendations, not features guaranteed by vLLM’s response format.
Do not treat response timing fields as billing units unless your own pricing policy explicitly uses them. Likewise, do not infer customer attribution from Prometheus totals: those metrics are aggregates for service monitoring, not a customer ledger.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Keep development endpoints out of production exposure
The online-serving documentation warns against using development-mode endpoints in production. Its listed operations include cache resets that can disrupt service, pause and resume controls, weight updates that can change model behavior, and collective RPC operations capable of executing arbitrary methods. Keep development mode disabled in production and expose only the endpoints your service requires behind the deployment’s authentication and network controls.
Check release-specific behavior before rollout
Endpoint availability, flags, response behavior, and metrics can change between vLLM releases. The stable online-serving documentation is live, while the cited per-request metrics page is specifically for v0.30.0. Verify the API and configuration against the version actually deployed, then test streaming usage, supported tasks, metric interpretation, and overhead under a representative workload before relying on them in production.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
