What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
An LLM telemetry table has no single natural denominator. A row might count a model inference call, a broader GenAI operation, or a top-level application request—and one application request can contain several child operations. To interpret a count, token total, average, or latency figure, first identify what was counted and how its scope was set.
What does one row or count represent?
OpenTelemetry distinguishes a broad GenAI operation from a model inference call. An operation can be an LLM request, a function call, or another distinct action in a larger workflow. An inference span, by contrast, represents a client call to a GenAI model or service that generates a response or requests a tool call. See OpenTelemetry’s GenAI metrics conventions and GenAI spans conventions.
As an Amazon Associate I earn from qualifying purchases.
Those units are not interchangeable. The OpenTelemetry walkthrough illustrates a top-level invoke_agent span with child chat spans for model calls and execute_tool spans for tool calls. A chart counting child model calls therefore need not show the same number as a chart counting top-level application requests. The walkthrough is an example of trace structure, not a universal aggregation rule: combining child operations into a request-level measure is an implementation choice that needs to be stated. Read the OpenTelemetry walkthrough.
How can one request produce several counts?
Consider an illustrative agent interaction: one user request starts an agent workflow, which makes two model calls and invokes a tool once. A call-level count could record two model inferences and one tool operation; a request-level count could record one application request. The numbers describe different units, not competing measurements. If the workflow retries a call or makes further model calls, a call count may rise while the top-level request count remains one.
#1 Best Overall
For this reason, “tokens per request” is ambiguous unless “request” is defined. It might mean tokens divided by HTTP or application requests, agent turns, or model inference calls. A workflow can also send repeated or expanded prompts across its child calls, so a token sum across those calls is not necessarily a count of unique user-provided text.
What to check before comparing telemetry tables
- Unit and scope: Is the denominator a top-level application request, an inference call, or any GenAI operation?
- Aggregation and time window: Is the figure a count of observations, a sum, a rate, or an average per request or call? Over what interval?
- Call composition: Are retries, tool calls, embeddings, and multiple model calls included? Does the table aggregate child spans to a parent request?
- Token basis: Are input and output reported separately? Are cached, image, reasoning, or other token categories included, and are values billed or model-consumed?
- Grouping: Which provider and exact requested model are shown? OpenTelemetry recommends provider and model attributes where applicable; its provider attribute may identify the configured client or proxy rather than the ultimate upstream provider.
- Latency scope: Is the value the duration of an inference call or of a broader workflow? OpenTelemetry defines inference duration from issuing the request until the response is fully received, or until the operation ends in error or cancellation. A workflow duration should not be labelled model latency.
Why token totals need their own definition
Input and output are distinct observations, not one undifferentiated token count. The OpenTelemetry walkthrough’s token metric separates those types. Its conventions also distinguish usage categories such as input, output, cache-read input, cache-write input, and reasoning output. The definitions say input-token totals should include all input types, including cached tokens, while detailed usage attributes are subsets of total counts. When a provider exposes both billed and consumed counts, OpenTelemetry recommends reporting billed counts so the measurement aligns with charged units. Consult the current GenAI metrics guidance for the conventions in use.
Cost per request likewise requires a declared request boundary and a billing-aligned token or provider-cost basis. Token and duration metrics can help estimate cost, but the conventions do not define one universal cost formula. State what the numerator includes and whether the denominator is application requests, agent turns, or model calls before presenting a per-request figure.
How to interpret missing usage and content data
Missing streaming usage is not zero
In NVIDIA’s NeMo Guardrails metric reference, streaming token usage is emitted only when the upstream provider returns a usage field. If the field is absent, the implementation records no observation; that is deliberately different from recording zero tokens. Its metrics are recorded once per downstream LLM call, not once per IORails request. This is an implementation-specific example, not a universal behavior across providers. See NVIDIA’s metric reference.
Content capture is optional
Prompt and tool-argument text is not required to measure token counts or durations. In the OpenTelemetry walkthrough, content capture is off by default because prompts and tool arguments can contain sensitive data; enabling it adds message and tool details to spans. Metadata such as model names, token counts, and durations can support operational measurement without recording that content. Decide deliberately whether content-level debugging is necessary and appropriate for the data being handled.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to put in a useful table label
A compact label can remove much of the ambiguity: name the unit, scope, aggregation, time window, and token basis. For example, “Input tokens summed across inference calls, including cached input; billed usage; daily” says more than “tokens per request.” If the table presents an average, specify whether it is per top-level request or per inference call and how calls with missing usage are handled. Check the OpenTelemetry convention version used by your instrumentation, because its GenAI documentation is actively developed and names or stability status can change.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
