Recommended Free Tools
There is no universal price for an AI request. Cost depends on the model, the amount of input and output, how the service is metered, and—when you run the model yourself—how much capacity is reserved but not busy. To compare an API with self-hosting, match the same model quality, request pattern, and latency target, then compare the API bill with the fully allocated cost of keeping your own service available.
What drives the cost of serving each request?
Inference is the work of producing a model response. A request’s cost is shaped not just by its token count, but by the model and hardware, the split between prompt processing and generation, context length, batching, concurrency, latency expectations, and how much of the serving capacity is actually used.
As an Amazon Associate I earn from qualifying purchases.
- Model and hardware: More demanding models may need more accelerator memory or compute. Hardware price alone does not tell you how many useful tokens it can serve at the required quality and latency.
- Request shape: Input length, output length, and context size affect different parts of inference.
- Serving policy: Batch size, concurrent requests, and latency targets influence throughput and utilization.
- Cost accounting: A hosted API charge and the fully allocated cost of operating a self-hosted service use different meters.
Energy is similarly workload-dependent. Microsoft Research notes that peak power draw affects data-center cost, while a 2026 study measures energy across model, inference phase, batch size, context, and output length. Neither establishes a single electricity cost for an AI request. [Microsoft Research’s Splitwise paper] [2026 energy study]
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11How much does AI inference cost per request?
For a hosted API
A useful starting estimate is:
Request charge ≈ input tokens × input rate + output tokens × output rate + applicable cache, tool, or service-tier charges.
#1 Best Overall
- A USB accessory that brings machine learning inferencing to existing systems. Works with Raspberry Pi and other Linux systems
- Performs high-speed ML inferencing: the on-board edge TPU Coprocessor is capable of performing 4 trillion operations (tera-operations) per second (tops), using 0.5 watts for each tops (2 tops per watt). For example, it can execute state-of-the-art mobile vision models such as mobilenet V2 AT 400 FPS, in a power efficient manner
- Works with Debian Linux: connects to any debian-based Linux system with an included USB 3.0 Type-C cable
- Supports tensorflow Lite: no need to build models from the ground up. Tensorflow Lite models can be compiled to run on the edge TPE
- Supports automl vision edge: easily build and deploy fast, high-accuracy custom image classification models to your device with automl vision edge
This is only a starting point: a provider may meter cache reads and writes, long-context requests, batch or fast modes, images, audio, or tools differently. The current pricing table for the specific model and service is the authority for a real estimate.
DigitalOcean’s pricing page illustrates how different meters can coexist: it lists model-specific rates per million tokens, GPU-hour prices for dedicated inference, and says batch inference can be discounted by up to 50% for OpenAI and Anthropic models. These are DigitalOcean-specific terms, not standard market prices; rates and availability can change. [DigitalOcean pricing]
Rank #2
- High-Performance AI Processing: The MX3 is designed to handle the most demanding AI computer vision workloads, delivering exceptional performance and efficiency.
- Flexible Integration: The MX3 can be easily integrated into your existing systems via its M.2 M-key form factor and support for Linux operating systems.
- Energy Efficient: The MX3 is designed to provide high performance while minimizing power consumption.
- Comprehensive Software Development Kit (SDK): The MX3 is supported by a comprehensive SDK that simplifies development and deployment.
- Hardware compatability: The MX3 is compatible with the PCI-SIG M.2 M-key 2280 Specification. It can be used with the Raspberry Pi 5 with a M-key 2280 HAT.
For self-hosted inference
Self-hosting does not usually produce a simple charge for each API call. To estimate a per-request or per-token cost, decide which infrastructure costs belong to the service, then divide by the work it actually serves. Include capacity kept ready for traffic as well as the compute used while requests are active.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why are input and output tokens priced differently?
Providers may charge separate rates for input and output tokens because prompt processing and token generation put different demands on a serving system. A long prompt requires processing the input and retaining its attention state as the response is generated; a long answer requires more generation work. Equal total token counts therefore do not necessarily mean equal compute, latency, or energy.
The KV cache holds attention state used to generate later tokens. In the models and systems studied in Microsoft Research’s Splitwise paper, each active generated token accesses the KV cache for the context so far. The paper characterizes prompt-phase batching as compute-bound and token generation as limited by memory capacity in its experimental setup. Long contexts and many concurrent sequences can consequently make memory capacity and bandwidth important to throughput. These findings describe the studied systems, not every model or serving stack. [Microsoft Research’s Splitwise paper]
How do batching, utilization, and latency change the cost?
Batching lets a system serve more inference work with a hardware allocation, potentially spreading fixed serving work over more tokens. But larger batches are not free: they compete for memory, may contain requests with very different context lengths, and can increase waiting time. A service with a strict response-time target may have less freedom to wait for a large batch.
Rank #4
Utilization matters especially when capacity must remain available between bursts. CNCF’s OpenCost discussion distinguishes two ways of expressing self-hosted cost per million tokens:
- Usage-based cost: Infrastructure attributed to active inference work divided by the tokens processed.
- Allocation-based cost: The cost of capacity and supporting infrastructure kept available divided across the work actually served.
In CNCF/OpenCost’s illustrative example, a usage-based cost of $1.00 and allocation-based cost of $4.00 per million tokens imply 25% utilization. This is an explanatory example, not an industry benchmark; allocation-based cost is the more relevant measure for a build-versus-buy decision. [CNCF on OpenCost inference cost tracking]
Best Value
- ✅Powered by 26 Tera-Operations Per Second (TOPS) Hailo-8 AI Processor. 2.5W typical power consumption
- ✅Scalable, enabling simultaneous processing of multi-streams & multi-models
- ✅Enabling real-time, low latency and high-efficiency AI inferencing on the edge devices
- ✅Supports TensorFlow, TensorFlow Lite, ONNX, Keras, Pytorch frameworks
- ✅Supports Linux and Windows. Supports the temperature range of -40°C to 85°C
A 2026 study shows why per-token energy needs its workload conditions attached. For Llama-3.2-1B on an H200, at batch 16 and context 4K, the authors report that increasing output length from 10 to 512 tokens reduced token energy from 7.46 to 0.72 J/token, while total energy for the batched inference window rose from 1.19 to 5.93 kJ. In that experiment, more output tokens spread energy across more tokens even as total energy increased; it does not establish that longer outputs generally make a request cheaper. [2026 energy study]
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Is self-hosted inference cheaper than an API?
It can be, but compute-only comparisons often leave out costs that determine the real break-even point. Compare the fully allocated self-hosting cost with the API price for the same model quality, request mix, output length, and latency requirement. A cheaper GPU-hour does not prove a cheaper response: you also need to know the throughput achieved on your workload and how much capacity sits idle or reserved.
| Approach | Typical cost basis | What to compare |
|---|---|---|
| Managed inference API | Often input and output usage rates, with possible cache, tier, or batch distinctions | Model and rate tier; input and output tokens; billable features and service modes |
| Dedicated cloud inference | GPU-hour or instance time, plus platform costs | Capacity that must stay available; realistic utilization; latency and scaling behavior |
| Self-hosted infrastructure | Amortized or rented hardware, operations, and idle or allocated capacity | Fully allocated cost per token at observed traffic, including staffing, networking, storage, and resilience |
DigitalOcean lists both serverless model rates and dedicated GPU-hour prices, but those figures cannot be compared directly without measuring throughput and utilization for the target model and request pattern. [DigitalOcean pricing]
- Fix the workload: Specify the model, representative input and output lengths, context sizes, traffic pattern, concurrency, and latency target.
- Estimate the API bill: Apply the provider’s current rates and include any cache, tool, tier, or modality charges that apply.
- Measure self-hosted throughput: Test the target model and workload on the intended hardware and serving configuration; use observed throughput rather than a GPU-hour price alone.
- Allocate the full service cost: Include reserved and idle capacity plus the shared infrastructure and operations needed to keep the service available.
- Compare like with like: Divide by the same unit of useful work and check that quality and latency requirements are met by both options.
How should you read performance-per-dollar claims?
Performance-per-dollar figures are only meaningful with their benchmark, hardware, price assumptions, and measurement method attached. Google Cloud’s 2023 post reports historical results from MLPerf Inference 3.1, including 1.7x–3.9x relative performance improvement for specified H100/A3 workloads over A2 and up to 1.8x performance-per-dollar for a specified L4 comparison. Google explicitly says its derived performance-per-dollar metric is not an official MLPerf metric and is not verified by MLCommons; the results and prices are historical, not current purchasing guidance. [Google Cloud MLPerf Inference 3.1 discussion]
NVIDIA presents cost per token as an end-to-end view that spans GPUs, CPUs, networking, software, and ecosystem, and cautions against looking only at compute pricing or FLOPs per dollar. That is useful framing for a total-cost comparison, but it is vendor positioning—not independent proof that a particular accelerator or platform is cheapest. [NVIDIA AI inference materials]
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
