A healthy dashboard does not prove an AI feature is doing its job. A request can return successfully while the answer is irrelevant, unsupported, incorrect, unsafe, or incomplete. The 26% in this title describes the reported incident—not a general failure rate—and its denominator, observation window, and definition of failure have not been established. The practical fix is to define success around the user’s task, trace the request end to end, and monitor output quality separately from infrastructure health.
What does “failing” mean for this feature?
Start by turning “it failed” into a measurable user outcome. A timeout is different from a fluent answer that does not solve the task, and both differ from a tool call that silently returns the wrong data. Those cases can have different causes and require different alerts.
As an Amazon Associate I earn from qualifying purchases.
For any reported failure rate, record the numerator, denominator, time range, and affected request or user segment. For example, distinguish “26 of 100 requests in this workflow over a specified week did not complete the task” from “26% of all model calls failed.” Without those details, the percentage is not interpretable or comparable.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Task failure: the user did not get the intended result, even if the response was delivered.
- Quality failure: the answer was wrong, irrelevant, unsupported by its sources, incomplete, or unsafe.
- Operational failure: a timeout, provider error, tool failure, excessive latency, or another interruption prevented the result.
- Behavioral signal: abandonment, repeated prompts, corrections, or negative feedback may reveal friction, but each is an imperfect proxy for failure.
Choose the outcome that matters for the feature and document how it is judged. A user asking a factual question may need a grounded answer; a workflow assistant may need to complete a sequence of actions. The measure should reflect that task rather than treating every response as equally successful.
#1 Best Overall
Why can monitoring say “healthy” while users get bad answers?
Monitoring has several layers. AWS separates application and system health, business and user interaction, and model or AI quality. Microsoft likewise describes operational observability alongside evaluation of AI behavior. A service can pass the first layer—requests are completing and infrastructure is available—while failing at the task or answer-quality layer.
| Signal layer | What to measure | What it can reveal |
|---|---|---|
| Application and system | Request errors, latency, throughput, dependency availability | Whether the service and its components are responding reliably |
| User and business outcome | Task completion, abandonment, retries, corrections, or other feature-specific outcomes | Whether people can accomplish the intended job |
| Model and output quality | Relevance, groundedness, correctness, completeness, and safety, using suitable evaluators | Whether a delivered answer is useful and acceptable |
Keep these signals visible together rather than collapsing them into one “healthy” status. A successful HTTP response is evidence that a request returned; it is not evidence that the answer was correct. A quality measure is also not a substitute for latency and error tracking: the feature needs both.
How do you find where a bad result entered the request path?
Reconstruct representative failures as end-to-end traces. Google recommends monitoring the full application path and retaining lineage for executed components; Microsoft describes traces spanning model calls, tools, agent decisions, and dependencies. The purpose is to identify the boundary where the request stopped meeting expectations, not merely to confirm that a model call occurred.
- Select a confirmed failure. Keep the original input, the observed output, the expected task outcome, and the method used to classify it. Use appropriate access controls when traces contain sensitive user data.
- Follow the request from entry to response. Record the relevant model call, retrieval results or knowledge source, tool invocation, agent decision, and downstream dependency. Include timings and errors at each boundary.
- Capture the versions that shaped the result. Preserve the model identifier, prompt or configuration version, retrieval source or index version, and relevant tool or dependency versions. Without this lineage, a later change can make the failure difficult to reproduce.
- Compare with a nearby successful request. Look for differences in input, retrieved context, tool arguments, model or prompt version, environment, and timing. This can narrow the investigation, though one comparison does not establish a cause.
- Group confirmed failures. Break them down by input type, model or prompt version, retrieval source, tool, environment, and time. Repeated clusters point to a more testable hypothesis than an overall failure percentage.
A trace is a record of what happened, not proof that the answer was good. Pair it with an evaluation of the task outcome so that a complete execution path is not mistaken for a successful one.
How can you tell whether quality changed in production?
Evaluate the feature continuously against representative examples, then check whether those examples still resemble production traffic. A benchmark that no longer reflects actual requests can make quality appear stable while users encounter unfamiliar inputs—or can signal a change that has little effect on the tasks that matter.
Maintain a task-specific evaluation set
Build evaluation examples from realistic feature use, including confirmed failures and important edge cases. Define what counts as a passing result for the task, and use human-reviewed outcomes or established ground truth to check whether automated evaluators agree with the failures people care about. Microsoft documents both sampled production evaluation and scheduled evaluation; the choice depends on how quickly changes need to be detected and how much review capacity is available.
Compare production inputs with the evaluation baseline
Google describes checking for skew and drift using signals such as text size, token counts, vocabulary, topics, and embedding or statistical methods. A change in the kinds of requests users send can alter performance without a code deployment. Compare relevant input and output patterns over time, and inspect whether a shift is concentrated in a particular segment rather than assuming the whole feature has regressed.
Promote confirmed failures into regression checks
When a failure is understood well enough to classify, add an example or scenario to the evaluation suite. Re-run it when prompts, models, retrieval sources, tools, or application behavior change. This creates a record of known failure modes and makes regressions easier to catch; it does not guarantee that future inputs will behave the same way.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Could capacity or rate limits explain the problem?
Yes, but investigate capacity as one hypothesis alongside quality, retrieval, tools, and changing inputs. Check provider rate-limit responses, concurrency, retry behavior, queueing, backpressure, fallback capacity, and whether an agent or tool loop is consuming more calls or tokens than expected. Retries can help with transient errors, but uncontrolled retries can increase load and delay responses; budgets and limits make that behavior easier to contain.
Datadog’s 2026 analysis of its LLM Observability customer traces reported errors on 5% of LLM-call spans in February, with 60% of those errors attributed to exceeded rate limits. For March, it reported errors on 2% of spans, with rate limits accounting for almost one third of errors and nearly 8.4 million rate-limit errors in total. These are vendor-specific span-level figures; Datadog notes its customer base is a large but imperfect sample of the global market and classifies 429 spans as rate-limit or resource-exhausted errors. They do not diagnose this incident, and they should not be compared with the title’s 26% without matching the unit, denominator, and time window.
What should monitoring do once it finds a problem?
Monitoring is useful when evidence leads to a timely, owned response. AWS describes capture, alerting, and response as connected parts of production monitoring. Microsoft documents alerts based on quality thresholds as well as production evaluation. Set thresholds around meaningful user or product outcomes and operational risk, then route each alert to a team or person with a response procedure.
- Alert separately on operational conditions such as elevated errors or latency and on quality or task-success signals.
- Include the affected feature, time range, segment, relevant versions, and trace or evaluation examples in the alert where practical.
- Specify who investigates, which checks to run, and when to roll back, change a prompt or retrieval source, adjust capacity, or escalate to a provider.
- Review whether the alert detected a user-relevant problem and whether the response reduced it; tune thresholds that produce noise or miss meaningful failures.
For a newly reported failure rate, the first useful sequence is therefore: define the failed task and denominator, separate quality from operations, inspect representative traces, compare production with evaluation data, test the most plausible causes, and convert confirmed failures into regression cases and actionable alerts.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
