How to Estimate the Cost of Running Generative AI at Scale (2026 Guide)

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

There is no universal cost per generative-AI query. A defensible estimate starts with your actual workload—requests, input and output tokens, cache hits, peaks, retries and model mix—then adds the capacity and operating costs required to deliver your quality, latency, availability and geographic targets. Report a low, base and high range, keep managed usage, provisioned capacity and self-hosted infrastructure separate, and replace assumptions with billing and utilization measurements as traffic grows. Published figures can anchor an estimate, but they are not transferable prices: NVIDIA’s approximately $0.09 per million-token H100 result applies to one model and serving configuration, while Microsoft Research’s 0.34 Wh and 4.32 Wh figures measure energy, not dollars.

How much does it cost to run generative AI at scale?

The answer is the sum of metered model usage, capacity reserved for peaks, and the infrastructure and operations needed to meet your service-level objectives. A practical model is:

As an Amazon Associate I earn from qualifying purchases.

Total monthly cost = managed usage charges
                    + provisioned or reserved capacity
                    + infrastructure and operations
                    + networking, storage and security
                    + energy and facility overhead
                    + expected retries, errors and idle capacity

For each workload, calculate billable input tokens, cached input tokens and output tokens by model and region. Then test whether that spend still delivers the required first-token latency, end-to-end latency, concurrency, availability, data residency and quality. A low per-token rate is not a low cost if it forces over-provisioning, throttling or a more expensive recovery path.

Use three scenarios rather than one point estimate. The low case uses conservative traffic and short outputs; the base case uses measured traces; the high case includes peak bursts, longer generations, more tool calls and retries. Label every assumption, date, model version, region and meter so another engineer can reproduce the calculation.

How do I calculate LLM inference cost from a workload?

Start with a call-level inventory. Token volume and model mix drive metered cost, while the shape of that volume determines capacity cost.

1. Segment traffic by feature and model

Separate chat, summarization, retrieval-augmented generation, extraction, agents and batch jobs. Record which model serves each feature, whether a fallback model is used, and whether reasoning or tool calls add hidden turns. A single blended average can conceal an expensive agent loop or a small, latency-sensitive endpoint that dictates capacity.

2. Measure request and token distributions

• Requests per day and month for every feature.
• Input tokens per request, including retrieved documents and tool results.
• Output tokens per request and the maximum output setting.
• Cached-token share and the provider’s separate cached-input meter, if any.
• Peak requests per second, concurrent requests and the duration of bursts.
• Reasoning, tool-use and fallback calls per user request.
• Retry, timeout and error rates, including retries caused by throttling.

Use production traces when available. If you are pre-launch, build low, base and high distributions instead of multiplying one average request by a monthly count. Keep the percentile values that affect capacity: p95 and p99 input length, output length, concurrency and burst rate.

3. Convert traffic into billable meters

For each model, multiply the expected quantity in each billing meter by the current rate for the intended region and deployment. Keep input, cached input and output as separate lines; providers can price them differently. Add paid retrieval, tool execution, storage or other services only when your architecture actually invokes them.

Monthly model charge = (input tokens × input rate)
                    + (cached input tokens × cached-input rate)
                    + (output tokens × output rate)
                    + tool/retrieval charges
                    + expected retry tokens

Rates, meter names, model availability and regional terms change. Verify them against the provider’s current documentation when you prepare the estimate. Microsoft’s cost guidance also warns that a fine-tuned deployment can incur hosting charges while it is deployed, even during low-usage periods; read the Microsoft Learn cost-planning documentation for the applicable deployment terms.

4. Separate usage from commitments and minimums

A managed service may combine token usage with a minimum capacity, a reservation or a commitment term. Put recurring commitment charges, overage, deployment hosting and one-time setup on separate rows. Otherwise a lightly used committed deployment can appear cheaper than an on-demand API even though the unused capacity is still a real cost.

What service envelope should the estimate satisfy?

Cost is meaningful only at a defined service level. Write the envelope before comparing models or hosting choices.

• Quality: the evaluation score, task success rate or human-acceptance threshold the model must meet.
• Latency: first-token and end-to-end targets, with percentile objectives rather than only an average.
• Concurrency: maximum simultaneous requests and the expected queueing policy.
• Availability: uptime target, failover design and the behavior allowed during a regional outage.
• Geography and residency: regions in which prompts, outputs and logs may be processed or stored.
• Response mode: whether asynchronous or batch responses are acceptable for part of the workload.
• Safety and retention: filtering, audit logs, encryption and retention requirements that add services or storage.

These constraints can change the economically best model. A slower model may be inexpensive for an overnight batch, while an interactive endpoint may need spare capacity to keep first-token latency within target during a burst.

How do managed token APIs, provisioned throughput and self-hosting differ?

Compare the options at the same quality, latency, traffic distribution and availability target. The following table summarizes the cost components and operational trade-offs established in current provider guidance.

Option	Primary billing basis	Capacity and burst behavior	Cost components to model	Operating burden
Managed token API	Input, cached-input and output usage; possible tool or retrieval meters	Provider capacity and rate limits; usage rises with tokens and retries	Per-meter usage, regional rates, overage and any deployment or hosting fees	Provider operates accelerators; you still design retries, routing, observability and data controls
Committed or provisioned throughput	Reserved capacity or provider-specific throughput units, plus usage or overage terms where applicable	Capacity is held for your deployment; size for peaks and maintain headroom to avoid throttling	Commitment duration, minimum capacity, unused allocation, overage and deployment hosting	You manage sizing and utilization; the provider manages the serving fleet
Self-hosted inference	Infrastructure ownership or rental rather than a token meter	Throughput depends on accelerators, batching, concurrency, model and serving stack; idle capacity is your cost	Accelerators and servers, serving software, networking, storage, security, power, cooling, facilities and operations	You own deployment, upgrades, reliability, scaling, security and capacity planning

Use “cost per useful request” or “cost per useful token” as the comparison unit. A useful request is one that meets the quality and latency envelope, not merely one that completed. A raw accelerator hourly rate or a list price without utilization is not a normalized comparison.

How do I estimate managed API charges?

Managed APIs are easiest to model when every request is translated into the provider’s actual meters and then stress-tested against limits.

Build a meter worksheet

1. List each model, deployment type and region.
2. Enter monthly request counts for low, base and high scenarios.
3. Enter input, cached-input and output tokens separately for each scenario.
4. Multiply each meter by the current regional rate and keep the rate’s effective date.
5. Add expected retries, fallback calls, tool calls and retrieval charges.
6. Add deployment hosting, minimums or commitments as fixed monthly lines.
7. Divide the total by successful, quality-qualified requests to obtain cost per useful request.

Do not assume that a cache hit eliminates all input cost; use the provider’s definition of cached tokens and its meter. Likewise, do not count a failed request as free if your client retries and consumes tokens. Track timeout and throttling behavior separately because a burst can create more billable attempts than the user-visible request count.

Account for model routing

If a router sends simple requests to a smaller model and difficult cases to a larger one, estimate the share and token distribution of each route. Include evaluator, moderation, embedding or reranking calls when they are part of the production path. Report the blended figure only after showing the component lines; routing changes can otherwise hide a cost increase in one feature.

How should I size provisioned throughput for peak traffic?

Provisioned capacity must cover peak and burst demand at the required latency, not merely the average daily token count. Capacity cost is driven by workload shape, maximum output settings, cache treatment and utilization.

Use tokens and concurrency, not averages alone

For every deployment, model the highest sustained and burst request rates, the input and output lengths at those rates, and the concurrency created by slow generations. A long maximum-output setting can reserve or consume capacity even when typical answers are short. Include headroom for traffic variance and failover; otherwise a deployment can look fully utilized on paper and throttle in production.

Understand utilization and throttling

Microsoft’s provisioned-deployment documentation estimates utilization from prompt tokens after cached tokens are removed and the configured max_tokens, then corrects that estimate using actual token counts. At 100% utilization, requests can receive immediate HTTP 429 responses, so a plan that targets full utilization leaves no room for bursts. Follow the formulas and production guidance in Microsoft Learn’s provisioned-throughput documentation, and compare estimated with observed utilization after launch.

Translate provider units correctly

Provisioned-throughput units are not interchangeable across providers. Google documents model-specific burndown rates for its provisioned throughput in GSUs; the required units depend on the model and measured token demand. Use the procedure in Google Cloud’s throughput-measurement documentation rather than applying a unit conversion from another model or vendor.

Price the unused portion

For a committed deployment, calculate the cost of capacity during quiet hours and during planned seasonal peaks. If traffic is highly variable, compare a smaller provisioned base plus overflow usage with a fully provisioned peak. Include commitment duration and the financial impact of scaling down slowly or being unable to transfer capacity between regions.

How do I estimate the total cost of self-hosting an LLM?

Self-hosting replaces a token invoice with a complete operating envelope. Measure throughput on the actual model and serving stack, then allocate every recurring and one-time cost to useful output.

Hardware and serving stack

• Accelerators and servers, whether purchased or rented.
• Inference software, model weights, quantization and upgrade work.
• Batching, parallelism and scheduling needed to meet latency targets.
• Spare capacity for failures, rolling upgrades and regional redundancy.

An H100 server or accelerator purchase price alone is not a self-hosting estimate. The hardware must be paired with usable throughput under your prompt lengths, output lengths, concurrency and latency objectives.

Data, network and security

• Model and dataset storage, checkpoints, logs and backups.
• Low-latency networking between application, retrieval and inference tiers.
• Data preparation, residency controls, encryption, identity and audit systems.
• Ingress and egress charges and cross-region replication where used.

These categories are highlighted in the AWS guide Understanding the costs of generative AI infrastructure. Its advice is concise: “Pick the right type and size of model for your use case. (You may not need the largest model.)” The guide was published in 2024, so use current infrastructure prices for your region and date.

Power, facilities and people

Include electricity, cooling, rack or facility overhead, hardware support, on-call coverage, security response, capacity planning and deployment engineering. Allocate staff and shared-platform costs by accelerator-hour, request or useful token. If an accelerator is idle while waiting for traffic, that idle time belongs in the denominator; otherwise the calculated unit cost is understated.

Measure usable throughput

Run representative prompts with the production model, output limits, batching, concurrency and streaming settings. Record tokens per second, queue time, first-token latency, error rate and accelerator utilization. Convert the resulting usable throughput—not a vendor peak—into cost per useful token or request:

Cost per useful token = monthly self-hosted total cost
                       ÷ successful tokens meeting quality and latency targets

How much does AI cost per query when energy is included?

Energy and dollar cost are related but distinct. You can multiply measured watt-hours by an electricity price for an energy-only estimate, but that does not include hardware, facilities, staff, networking or capital recovery.

Microsoft Research reports a median 0.34 Wh per query for a specified frontier-model scenario: models above 200 billion parameters running on an H100 node under stated workload, utilization and power-usage-effectiveness assumptions. In a test-time-scaling scenario with 15 times more tokens per typical query, the median rises to 4.32 Wh. Both figures are scenario measurements, not universal per-query values or monetary prices. Read the methodology in Microsoft Research’s energy study before using either as an input.

For your own estimate, measure total facility energy for the inference fleet, subtract no “free” capacity, and divide by successful requests under the same quality and latency envelope. Keep this energy metric beside, not inside, the dollar model unless you explicitly convert it using a documented electricity rate.

What published benchmark can anchor an estimate?

NVIDIA reports approximately $0.09 per million tokens for GPT-OSS-120B on an H100 using vLLM at 66 tokens per second per user, attributing the benchmark to SemiAnalysis InferenceX data from April 2026. The figure appears on NVIDIA’s H100 product page.

Treat that number as a scenario-specific benchmark estimate, not a cloud list price, hardware quote or provider-neutral comparison. It does not establish your cost unless your model, H100 configuration, vLLM version, batching, concurrency, token lengths, utilization, electricity and facility assumptions match. Re-run the measurement on your stack and use the result as an anchor in a range, not as a promise.

Can a worked example show the estimation method?

Yes. The following is an illustrative worksheet, not a market quote. Replace the assumptions with your traces and current rates.

Illustrative workload assumptions

• Two million user requests per month.
• Feature A uses a smaller model for 70% of requests; Feature B uses a larger model for 30%.
• Each feature has its own measured input and output-token distributions, cache share, peak requests per second and retry rate.
• Interactive traffic requires a first-token target and a defined p95 end-to-end target; a separate batch queue may run asynchronously.

Calculation sequence

1. Multiply each feature’s request count by its input, cached-input and output-token averages for the low, base and high cases.
2. Apply the current regional rate for each meter and model; add retry and fallback tokens.
3. Test the peak token rate and concurrency against on-demand limits. If the base deployment cannot meet the p95 target, price provisioned capacity with headroom.
4. For self-hosting, benchmark both models with the actual prompt mix, then add servers, spare capacity, storage, network, security, power, cooling and operations.
5. Divide each option’s monthly total by successful requests that pass quality and latency checks.
6. Run sensitivity cases: output tokens 25% higher, cache share lower, peak concurrency doubled, or a 10% retry rate after a dependency failure.

The output should be a range for each architecture, with assumptions visible beside every line. Do not collapse managed usage and fixed capacity into one “per-token” number until you have accounted for idle time.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

How do I compare API, provisioned and self-hosted costs fairly?

Use the same workload replay and acceptance tests for every option. The comparison should answer whether each architecture can deliver the same useful work, not which has the lowest advertised unit price.

• Cost unit: cost per quality-qualified request and per useful output token.
• Quality: identical evaluation set, safety checks and fallback policy.
• Latency: identical first-token and end-to-end percentile targets.
• Throughput: identical peak, burst duration, concurrency and output limits.
• Utilization: include queueing, headroom, idle committed capacity and failed attempts.
• Geography: include region-specific pricing, residency and cross-region transfer.
• Operations: include engineering, on-call, upgrades, security and incident response.
• Energy: report watt-hours separately and document the measurement boundary.

A managed API often wins for uncertain or spiky demand because you avoid owning idle accelerators. Provisioned throughput can be justified when traffic is predictable and latency guarantees matter. Self-hosting can make sense when utilization is high, model control or residency is essential, and you can operate the full stack; the estimate must prove those conditions rather than assume them.

What are the most common estimation mistakes?

Using daily averages for peak capacity

Average daily tokens hide bursts and long generations. Size against peak concurrency and burst shape, then reserve headroom below the utilization level at which throttling begins.

Multiplying one average request by a blended price

Different features and models have different token distributions and meters. Keep model mix, cached input, output and retries separate until the final roll-up.

Ignoring maximum output settings

Provisioned-capacity calculations can use configured maximum output as well as actual output. An unnecessarily high maximum can consume capacity or force a larger deployment even when typical answers are short.

Comparing a GPU hourly quote with an API token rate

The GPU quote omits serving software, storage, network, security, power, cooling, staff, spare capacity and idle time. Convert both options to cost per useful request under the same workload.

Treating benchmarks as guarantees

Hardware, model, serving stack, batching, prompt lengths and concurrency change throughput. Reproduce the benchmark conditions or label the figure as an anchor with a confidence range.

Counting energy as a dollar cost

Watt-hours are an environmental or energy metric. Convert them to dollars only with a stated electricity price, and keep capital and operating costs separate.

Failing to update the model

Compare forecast token counts and utilization with production billing and telemetry every reporting period. Re-estimate after a model change, prompt change, traffic shift, new region or altered cache policy.

How should I maintain the estimate after launch?

Make the cost model an operational dashboard rather than a one-time spreadsheet.

1. Export provider billing by model, meter, deployment and region.
2. Join billing with request logs containing feature, token counts, cache status, latency, retries and outcome quality.
3. Track provisioned utilization, throttling responses, queue time and unused capacity at short intervals so bursts are visible.
4. For self-hosting, record accelerator utilization, power, throughput, error rate, spare capacity and facility overhead.
5. Recalculate low, base and high scenarios when the observed distribution moves outside the assumptions.
6. Review the service envelope with product and compliance owners before changing model, region or response mode.

This feedback loop turns an estimate into a defensible forecast. It also reveals which lever matters: reducing output tokens, increasing cache hits, routing easy requests to a smaller model, changing batch timing or adding capacity for a specific peak.

Which architecture should I choose?

Choose the option whose total cost remains acceptable after it meets your quality, latency, peak-throughput, residency and availability requirements.

• Choose managed usage when demand is uncertain, traffic is bursty or minimizing infrastructure operations is the priority.
• Choose provisioned throughput when demand is predictable enough to keep reserved capacity utilized and the application needs consistent latency or throughput.
• Choose self-hosting when sustained utilization, control, residency or model customization justify the infrastructure and operational envelope.

In every case, publish the assumptions, the low/base/high range, the cost per useful request, the energy metric and the date on which rates and model availability were checked. That is a more reliable decision than selecting a model from a single per-token figure.

Frequently Asked Questions

How often should a generative-AI cost estimate be updated?

Review it at least monthly for active production systems and immediately after a model, prompt, routing, region, cache or pricing change. Trigger a new capacity test when peak concurrency or output-length percentiles move materially.

Should retries be included in token-cost calculations?

Yes. Count every retry that reaches the model, including retries caused by timeouts, throttling or downstream failures. Keep retry tokens as a separate line so reliability work does not disappear inside the average request volume.

How can I reduce inference cost without changing models?

Measure output-length and cache distributions first. Then test shorter maximum outputs, prompt and retrieval reduction, higher cache-hit rates, batching for asynchronous work, and routing that prevents duplicate tool or fallback calls; verify that quality and latency targets still hold.

What data should I log to make the estimate auditable?

Record model and deployment, region, feature, input/output/cached tokens, maximum output setting, request and retry identifiers, latency percentiles, queue time, status codes, concurrency and quality outcome. For self-hosting, add accelerator utilization, power and usable throughput.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

AI infrastructurecloud costscost estimationgenerative AIGPU economicsLLM inference