An AI agent is working when it reliably completes its assigned task at an acceptable quality and speed for a cost that makes sense for that result. A convincing final answer is not enough: check what the agent did, whether the task actually passed your criteria, and what the entire run cost.
What counts as an AI agent working?
Define success in terms you can observe, not whether the response sounds polished. Depending on the job, success might mean the requested action was completed, the result passed a domain-specific check, the agent chose the required tool, or it followed applicable instructions.
As an Amazon Associate I earn from qualifying purchases.
There is no universal pass threshold or ROI formula established by the cited documentation. Set criteria that fit your application, then evaluate a representative set of tasks rather than relying on a handful of memorable runs. OpenAI recommends progressing from inspecting and grading traces to building datasets and running repeatable evaluations when comparing workflow changes (Evaluate agent workflows).
- Use deterministic checks where the result can be verified reliably.
- Use structured graders for workflow or response criteria that need judgment.
- Review a sample of results with a person, especially when an incorrect outcome has meaningful consequences.
A model-judge score can be useful, but it should not be treated as the sole definition of truth. The right combination of checks depends on the task.
#1 Best Overall
Inspect the full execution trace
Look beyond the final response. A trace can show the steps in a run, including model responses, tool calls, and delegated work. OpenAI’s tracing documentation describes recorded step status, duration, and data, which can help pinpoint where a workflow failed or spent time (Tracing).
Follow the run from start to finish and look for:
- A tool that failed, was unavailable, or returned an unusable result.
- Repeated calls that did not make progress toward the task.
- An incorrect tool choice or an unnecessary handoff to another agent.
- A slow step that dominates the run’s latency.
- A final answer that claims completion even though the required action did not succeed.
Trace grading can evaluate workflow-level questions, such as whether the agent selected the right tool, handed off when appropriate, followed instructions, or improved after a prompt or routing change. OpenAI describes trace grading as “the fastest way to identify workflow-level issues” in its agent evaluation guidance.
Count the cost of the whole run
The length of the final response is a poor proxy for spend. A run may involve several model calls, cached input, output tokens, reasoning tokens, delegated agent work, retries, and charges for tools, sandboxes, or third-party services. OpenAI’s observability and usage documentation identifies input, cached-input, and output tokens as cost-bearing usage and says reasoning tokens are billed as output.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →For an operational view, calculate cost per task that passes your agreed success criteria:
total attributable run costs ÷ number of tasks that passed the success criteria
This is a practical metric, not a published industry standard. Report it alongside pass rate and latency: a cheaper setup can look efficient while failing more often. Attribute costs consistently across model calls, retries, delegated work, and applicable non-model services.
Rank #3
Check whether your cost data is complete
Monitoring totals are estimates unless the required usage and pricing data are present and reconciled. Different tools handle missing usage and non-LLM charges differently, so understand what the number includes before treating it as spend.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteLangfuse: ingested versus inferred cost
Langfuse documents both ingesting usage and cost from a model response and inferring cost from usage plus a model definition. Its Token & Cost Tracking documentation says inference cannot determine the correct cost for reasoning models such as OpenAI o1 when token usage is missing, because the reasoning-token count is not visible. For those generations, it instructs users to provide token usage.
LangSmith: automatic LLM tracking and manual entries
LangSmith documents automatic cost calculation for supported LLM calls using token counts and model prices. Its cost tracking guide also describes manual cost entry for other run types, including tools and retrieval. Custom calculations require the relevant token counts, model and provider information, and price.
OpenAI usage: unknown does not mean zero
For OpenAI agent usage, counts may arrive after a turn ends; a blank or null value means unknown, not zero, and recorded values may change. OpenAI cautions that these usage counts are not necessarily a final bill in its observability and usage documentation and tracing documentation. Reconcile operational telemetry against provider billing data before using it as a financial ledger.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Compare configurations by outcomes, cost, and speed
When changing a prompt, model, routing rule, or tool setup, evaluate the old and new configurations on the same dataset with the same criteria. Compare these measures together:
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11- Pass rate: the share of tasks that meet the defined success criteria.
- Cost per passing task: the attributable cost of completed runs that pass, with cost components and estimation method stated.
- Latency: total completion time, plus slow steps visible in traces where available.
- Workflow behavior: failure, retry, tool-call, and handoff patterns.
OpenAI recommends datasets and evaluation runs for repeatable comparisons over time; its tracing and usage documentation describes the step and cost information that can support that analysis (agent evaluations, tracing, and observability and usage). The right trade-off depends on the job: a high-stakes task may justify higher cost for greater reliability, while a low-value task may not.
Quick Recap
Warning signs that an agent may be wasting money
- Good-sounding answer, failed task: the trace shows an unsuccessful action or unfinished work. Grade the outcome, not just the wording.
- More calls without better results: repeated model or tool calls and retries do not improve the pass rate. Inspect traces to find recurring workflow failures.
- Suspiciously low cost totals: the accounting excludes retries, delegated agents, tools, or other applicable charges.
- Inferred costs without adequate usage data: the tracker lacks required token counts or matching model-price information. For example, Langfuse documents that it cannot infer reasoning-model cost without the relevant usage counts.
- Blank usage shown as zero: an unknown or null value is being mistaken for no usage, even though OpenAI says those values may be delayed or change.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
