Recommended Free Tools
In Bhushan Kinge’s 2026 independent benchmark, Laya sustained 175 typed decisions per second on an NVIDIA H100 NVL at a measured p99 latency of about 91 ms, within a 130 ms p99 target. At that same target, an RTX PRO 6000 reached 146 decisions per second and an RTX PRO 5000 reached 42. These are results for one specific workload and serving setup—not universal GPU ratings or token-generation speeds.
What the benchmark measured
Laya takes a state and typed questions, then returns structured decisions rather than generating prose. The English checkpoint uses ModernBERT-large, has 421 million parameters, and lists a 512-token context limit. Accordingly, “decisions per second” here means answers to typed questions, not tokens per second. Laya model card
As an Amazon Associate I earn from qualifying purchases.
The benchmark used a frozen sample of 1,000 public SAM.gov contract-opportunity notices. Each request asked three questions: a choice, a score, and a yes/no decision. A result counted as sustainable capacity only if it achieved at least 90% of the offered request rate, stayed within the p99 latency target, returned no errors, and did not build a growing queue. Requests arrived using open-loop Poisson traffic and were served by a small dynamic-batching HTTP server. Benchmark workload and measurement definition
How much capacity each GPU sustained
The table gives the reported capacity at two p99 latency budgets. The figures are workload-specific results using TensorRT FP16 except for the H100 MIG row.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
| GPU and backend | Capacity at p99 ≤ 50 ms | Capacity at p99 ≤ 130 ms |
|---|---|---|
| RTX PRO 5000, TensorRT FP16 | 15 decisions/s | 42 decisions/s |
| RTX PRO 6000, TensorRT FP16 | Not measured | 146 decisions/s |
| H100 NVL, TensorRT FP16 | 105 decisions/s | 175 decisions/s |
| H100 NVL split into seven 1g.12gb MIG instances, eager FP16 | Target not met | Not met reliably |
The H100 NVL’s selected operating point for 175 decisions/s had a measured p99 latency of about 91 ms. The benchmark also translates the 130 ms-target rates into 3.6 million decisions per day for the RTX PRO 5000, 12.6 million for the RTX PRO 6000, and 15.1 million for the H100 NVL. Those daily totals are arithmetic extrapolations from sustained rates, not full-day tests. Reported capacity results
What the GPU comparisons do—and do not—show
Compare at the same latency budget
The 130 ms target is the fairest direct comparison among the three cards because all three have reported measurements at that budget. The RTX PRO 6000’s capacity at the 50 ms target is unknown: the sweep did not test below 50 requests per second. The benchmark therefore does not establish whether it would match or exceed another card at that stricter latency limit. Benchmark method and results
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
MIG isolation was not a win for this document mix
Splitting the H100 into seven MIG 1g.12gb instances provided isolation, but did not reliably meet the benchmark’s achieved-rate requirement on documents longer than 400 tokens. At the lowest tested aggregate load, the slices reached about 49 decisions/s at p99 127 ms, but missed the achieved-rate gate. That finding applies to this setup and workload; it does not establish that MIG is generally inferior or unsuitable for shorter prompts. Benchmark method and results
Backend rankings depend on serving conditions
TensorRT was not consistently the fastest option. In fixed-shape, larger-batch throughput tests, torch.compile with max-autotune FP16 was about 1.3–1.7 times faster than eager FP16 on each card. But those fixed-shape tests do not predict dynamic-serving capacity by themselves. Under dynamic serving at the 130 ms target, TensorRT raised H100 capacity from 93 to 175 decisions/s; on the RTX PRO 6000, eager FP16 and TensorRT both reached 146 decisions/s. On the Blackwell laptop’s multilingual checkpoint, eager FP16 beat TensorRT under the same SLO. The benchmark tested PyTorch eager at FP32, FP16 and BF16, ONNX Runtime CUDA, and TensorRT FP16; torch.compile was tested for fixed shapes, not as a serving backend. Backend and serving results
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Correctness checks are not task-accuracy results
Before timing, the author checked whether backend outputs reproduced upstream FP32 answers on a parity suite of 16 cases and 63 typed questions. Across four GPUs, three checkpoints, multiple precisions, and five backends, all 74 backend-and-device rows passed that suite. Checks also included public JSON equality, finite outputs, and steady-state allocator stability. This is evidence of output fidelity to upstream FP32—not evidence that Laya’s answers are accurate for real procurement decisions. Benchmark fidelity and limitations
How to read the cost estimates
The benchmark’s self-hosting model estimates costs per million decisions as follows. These are scenario calculations, not current quotes or guaranteed operating costs.
Rank #4
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
| GPU | Estimated cost per million decisions |
|---|---|
| RTX PRO 5000 | $0.67 |
| RTX PRO 6000 | $0.66 |
| H100 NVL | $1.86 |
The estimates assume a three-year card amortization, 100% utilization, and electricity at $0.12/kWh. The benchmark’s Jev API estimate was $6.80–$8.20 per million decisions at list token pricing for this workload. That comparison includes Jev’s public-internet path from Arizona but does not add a comparable network path to self-hosted Laya; an API also avoids owning hardware and operating the serving stack. Under the benchmark’s cost model, hosted service may still make economic sense below roughly one million decisions per day. The figure is a scenario threshold, not a universal break-even point. Cost model and assumptions
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Limits of the result
This was one English federal-procurement workload, with one run per configuration for each server sweep and replay. The serving implementation was a compact asyncio dynamic batcher over loopback, not Triton. Jev latency included public-internet access, while self-hosted Laya latency did not. The results should not be generalized to other prompt distributions, languages, models, production stacks, or task accuracy without additional evidence. The benchmark is independent, not an official Laya or Convai result. Benchmark limitations
Quick Recap
Best Value
- Powered by the NVIDIA Blackwell architecture and DLSS 4 OC mode: 2640MHz/Default mode: 2610MHz (Boost Clock)
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.125-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
