Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteA green AI evaluation means the configured grader passed on the sample it evaluated. It does not, by itself, prove that the test made the intended call to the target model. Check the run’s status, output item, grader results and per-model usage—especially the invocation_count—then make your test fail if the expected call is absent.
Why is my AI eval green when the model was never called?
An evaluation has criteria and a data-source configuration; a run also uses a model configuration. A passing result answers a narrow question: did the configured grader accept the evaluated sample? It does not independently prove that your application reached the target-model call you meant to test. OpenAI’s Evals API reference exposes run status, output items and usage information; its Graders reference describes the criteria graders can apply.
As an Amazon Associate I earn from qualifying purchases.
A test can therefore appear green if its configured checks pass on an available sample even though the particular target invocation you care about was skipped. The run’s status and the execution evidence are separate things to inspect.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHow do I verify that my eval actually invoked the model?
- Find the run and check its status. Confirm the run reached a terminal state. A completed or successful status does not establish that the desired application behavior occurred.
- Inspect an output item. Review its sample or input, output and grader results. This shows what the evaluation actually assessed, rather than only its overall pass/fail status.
- Check per-model usage. Look for the expected target model and its
invocation_countin the run’s usage information. If the expected model has no recorded invocation, that is evidence the intended call may not have happened; confirm with instrumentation in your own application or test, since the API reference does not describe every application-side execution path. - Identify grader-model activity separately. If a model-based grader is enabled, determine which model’s usage belongs to the grader and which belongs to the target behavior under test.
- Assert the call in the tested path. Add a spy or mock assertion, or use provider-side telemetry appropriate to your stack, so the test fails when the expected target invocation is absent. Treat this as a test-design safeguard, not as an automatic guarantee supplied by the evaluation result.
Can a mock or cached response make an LLM test pass without a model call?
Yes, if the sample evaluated by the grader can come from a mock, fixture, cached value or another path that avoids the target-model call. The grader judges the sample against its configured criterion; a passing judgment on that sample is not proof of where the sample came from. The API reference describes run outputs and usage, but does not establish which application-side path produced a particular value. Pair output-quality assertions with an explicit invocation assertion or suitable telemetry.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What does each grader establish?
Grader types answer different evaluation questions. None, on its own, verifies that a separate target-model invocation occurred.
| Grader type | What it evaluates | Does it prove the target model was called? |
|---|---|---|
| String check | A configured text relation or condition. | No. It assesses text, not the call path. |
| Text similarity | Similarity according to the configured metric. | No. It assesses similarity, not the call path. |
| Python | Supplied code applied to the evaluation data. | No, unless your code separately verifies invocation evidence. |
| Score model | A model-based score against the configured criterion. | No. The grader may itself use a model; distinguish its usage from the target invocation. |
| Label model | A model-based label against the configured criterion. | No. The grader may itself use a model; distinguish its usage from the target invocation. |
These grader categories and their behavior are described in OpenAI’s Graders API reference. For run status, output items and per-model usage, consult the Evals API reference.
Quick Recap
How to make a green result meaningful
- Use grader results to check output quality against the criterion you configured.
- Use per-model usage and
invocation_countto inspect recorded model activity for the run. - Use an assertion or telemetry in your own execution path to prove the specific target call is exercised.
- When a model-based grader is involved, label target and grader activity distinctly in your diagnostics.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
