Tokens per second (TPS) can mean the total output rate of a system serving many requests or the generation speed experienced by one request. The number alone does not tell you which one a benchmark measured—or how long a user waited for the first response. To interpret a TPS result, check what tokens were counted, the time window used, and the workload and concurrency behind it; then look at time to first token (TTFT) and inter-token latency (ITL) alongside throughput.
What does tokens per second measure?
At its simplest, TPS is a token count divided by a time interval. But the phrase does not specify which tokens, whose experience, or which interval. A benchmark may report output tokens generated across a serving system, output speed for an individual request, input tokens processed during prompt prefill, or a combined input-and-output rate.
As an Amazon Associate I earn from qualifying purchases.
Those are different measurements. NVIDIA defines system TPS as output tokens per second across concurrent requests. AIPerf’s per-user output throughput describes the steady cadence of one request and is the inverse of ITL. A system handling more requests at once may produce more total tokens each second even while each person waits longer between tokens.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
System throughput versus individual-request speed
- Aggregate output TPS: total output tokens divided by a benchmark’s duration across all requests. It measures serving capacity under the tested workload, not the speed seen by a single user.
- Per-user output throughput: the rate at which one request receives generated tokens after generation begins. For a long response, it is approximately 1 divided by ITL, when ITL is expressed in seconds per token.
- Prefill throughput: input tokens processed per second while the system processes a prompt. AIPerf derives a per-user streaming version as input sequence length divided by TTFT. It is not decode speed.
- Input-plus-output throughput: a combined count of prompt and generated tokens per second. It cannot be directly compared with output-only TPS unless the token scope is aligned.
Which metrics explain the experience behind a TPS number?
Throughput does not describe the full wait a user experiences. Three companion measures help distinguish responsiveness from capacity.
#1 Best Overall
- Legend perfected: Modern design with a matte "basalt black" finish in an optimized chassis with customizable AlienFX lighting zones, including the striking stadium lighting.
- Game changing graphics: Step into the future of gaming and creation with the NVIDIA GeForce RTX 5080 graphics, powered by NVIDIA Blackwell architecture.
- Marathon gaming unlocked: This high-performance technology ensures clean energy is consistently available, unleashing the top-level power of Intel Core Ultra 9 processor as you game, livestream, and multi-task for hours on end.
- 240mm heat exchanger: The 240mm heat exchanger, available on the optional CPU liquid cooling, heightens thermal resistance, ensuring temperatures stay consistently low during longer gaming sessions.
- Total command: Alienware Command Center software allows you to create and edit AlienFX lighting across the ecosystem, choose and monitor your performance mode across distinct power states, and create custom gaming profiles for your whole library.
Time to first token (TTFT)
TTFT is the time from submitting a request until receiving its first output token. It includes prompt processing and can also include queueing and network latency. Longer prompts commonly increase it. NVIDIA’s versioned NIM guide also includes tokenization and detokenization of the first token in its description of TTFT.
Inter-token latency (ITL) and time per output token (TPOT)
ITL or TPOT describes the average interval between generated tokens after the first token arrives. In NVIDIA GenAI-Perf’s formula, ITL is calculated as (end-to-end latency − TTFT) / (output tokens − 1). This formula excludes the initial TTFT. For a long response, the individual request’s generation rate approaches the inverse of ITL; that is a per-request decode view, not aggregate system TPS.
Rank #2
- NVIDIA Ampere Streaming Multiprocessors: The all-new Ampere SM brings 2X the FP32 throughput and improved power efficiency.
- 2nd Generation RT Cores: Experience 2X the throughput of 1st gen RT Cores, plus concurrent RT and shading for a whole new level of ray-tracing performance.
- 3rd Generation Tensor Cores: Get up to 2X the throughput with structural sparsity and advanced AI algorithms such as DLSS. These cores deliver a massive boost in game performance and all-new AI capabilities.
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure.
- OC Mode : 1500 MHz (Boost Clock)/Default Mode : 1470 MHz (Boost Clock)
End-to-end request latency
End-to-end latency is the time from sending a request until receiving its final token. It includes queueing, batching, and network overhead. NVIDIA expresses it as TTFT + generation_time. Two systems with similar TPS can therefore feel quite different if their TTFT or end-to-end latency differs.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhy can two TPS benchmarks disagree?
A result depends on both the workload and the benchmark’s measurement rules. NVIDIA’s explainer gives different TPS timing windows for GenAI-Perf and LLMPerf: GenAI-Perf measures from the first request to the final response, while LLMPerf divides by the full benchmark duration, which can include prompt generation, request preparation, and saving responses. In a single-concurrency scenario, NVIDIA reports those LLMPerf overheads can sometimes account for 33% of the benchmark duration. That is an observation in the article, not a universal overhead rate.
Rank #3
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
Workload, concurrency, and serving limits
Prompt length, output length, request rate or concurrency, streaming behavior, serving setup, and measurement window all affect the result. More simultaneous requests can raise aggregate TPS while increasing latency or lowering per-user speed. Throughput may reach a saturation point as available compute is used; beyond that, it can level off or possibly decrease. If requests exceed an engine’s maximum batch size, queueing can increase latency even when throughput has saturated. A concurrency sweep is more informative than a single result when the goal is to understand that trade-off.
Token counting and streamed chunks
Token counts are not always interchangeable. AIPerf distinguishes output tokens from output sequence length, which can include reasoning tokens. Its documentation also cautions that a streamed first chunk can contain multiple tokens; treating that chunk as one token can inflate a derived per-user throughput figure. MLCommons Endpoints specifies that its official input and output sequence counts use a reference tokenizer and chat template.
Rank #4
- System Compatibility Note: 2‑slot ITX card, 169.9x123.5x39.2mm, single 8‑pin power, recommended 500W PSU. Verify chassis clearance before purchase.
- Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
- Intel Arc A380 GPU: Powered by Intel Xe architecture with 6GB GDDR6 on 96‑bit bus – ideal for compact gaming, HTPC, and media builds.
- 2250MHz GPU Clock: Factory overclocked core delivers solid performance for esports titles and everyday creative tasks.
- Small Form Factor ITX Design: Compact 2‑slot card fits easily into mini‑ITX and small form factor cases without sacrificing performance.
Tool-specific conventions
Benchmark implementations may handle warmup, steady state, empty responses, and timing boundaries differently. MLCommons Endpoints documents its own v1.0 conventions: its official result uses a steady-state window where applicable and presents P90 rather than P95. Those are MLPerf Endpoints rules, not universal LLM benchmark requirements.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11How to compare two TPS claims fairly
Before treating one result as faster, make sure the measurements answer the same question. Record or match:
Best Value
- ▶ FLAGSHIP AMD RYZEN AI MAX+ 395 MINI PC – Packing 16 Zen 5 cores, 32 threads (via SMT), 64MB L3 cache, and a 5.1GHz boost clock. Delivers 126 TOPS total AI compute – including a 50 TOPS XDNA 2 NPU, 25% above Microsoft Copilot+ standard. Run 70B+ LLMs locally, keep data private, and tackle 8K editing, compiling, and rendering simultaneously. Recognized as the "most powerful x86 APU" for AI – a true game‑changer for creators, researchers, and power users.
- ▶ AMD RADEON 8060S iGPU – DESKTOP‑GRADE GAMING & CREATION – No discrete GPU needed. With 40 RDNA 3.5 compute units and dynamic memory allocation (up to 96GB), play AAA titles at 1440p high settings, accelerate 8K video exports in DaVinci Resolve, or generate AI art locally. Outperforms RTX 4060 laptop GPUs in benchmarks – all in a silent, compact chassis that fits anywhere.
- ▶ 128GB LPDDR5X‑8000MHz + 2TB SSD + DUAL M.2 SLOTS – Onboard 128GB memory at 8000MHz offers 45% more bandwidth than LPDDR5 for blazing‑fast AI loading and seamless multitasking. GPU shares this pool to run 70B+ LLMs with ease. Pre‑installed 2TB PCIe 4.0 SSD, plus a second M.2 slot for expansion up to 8TB or RAID. Store massive datasets, 8K footage, and game libraries – scale as your needs grow.
- ▶2.5GbE + Wi-Fi 7 + BT 5.4 — The mini computers come with 2.5GbE LAN ports enable firewall, link aggregation, soft routing, and NAS applications. Built-in Wi-Fi 7 and Bluetooth 5.4 offer stable, high-speed wireless connections for projectors, printers, monitors, speakers, and more—ideal for a versatile, clutter-free workspace.
- ▶QUAD 8K DISPLAY OUTPUT & DUAL USB4 – M5 Mini PC drives four 8K@60Hz monitors via HDMI 2.1, DP 1.4, and dual USB4 (40Gbps, Thunderbolt 4 compatible, PD & DP Alt Mode). HDMI and DP each support 8K@60Hz; USB4 handles both video and high‑speed data. Perfect for immersive gaming, professional video walls, or complex multitasking – plus charge devices directly from USB4 ports.
- Model and version, tokenizer, and serving framework.
- Hardware count and type, plus relevant precision or quantization configuration.
- Prompt and output lengths, including whether reasoning tokens are counted.
- Whether the rate is aggregate system throughput or individual-user speed, and whether it counts output tokens, prefill tokens, or both.
- Concurrency or request-arrival pattern, batch limits, and streaming behavior—including how the first chunk is counted.
- Warmup, ramp-up, steady-state, cooldown, and measurement window.
- TTFT and latency distribution alongside throughput when responsiveness matters.
- Whether the number is a reproducible measurement, a vendor-reported result, or a theoretical peak.
For serving many requests, compare aggregate capacity under a comparable load. For an interactive assistant, prioritize TTFT and per-user ITL or TPOT as well as throughput. For long-prompt workloads, examine input prefill throughput. Peak aggregate TPS is not a proxy for one person’s typing-like experience.
What to include when publishing a TPS result
A useful benchmark report makes the number reproducible and interpretable. State the model, tokenizer, serving stack, hardware and configuration, prompt and output lengths, token-counting rule, concurrency or arrival rate, streaming behavior, timing window, and treatment of warmup and overhead. Report whether the figure is output-only, prefill-only, combined, aggregate, or per-user. Pair throughput with TTFT and a latency distribution so readers can see the capacity-versus-responsiveness trade-off.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
