PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteBuild the suite around frozen, versioned cases and the model responses recorded for them. Run that replay suite on every pull request with Vitest, validate output structure with Zod, and grade only what a rule can actually establish. Keep live model calls in a separate job. In this setup, “deterministic” describes your test inputs and replay path. It does not promise that a hosted model will return identical text on every call.
What “deterministic” can and cannot mean here
A hosted model is not a pure function. Even when you pin a model name and fix request parameters, the provider can change the underlying system and the output can vary. OpenAI’s Cookbook recommends keeping the seed and other request parameters fixed and comparing system_fingerprint across calls. It says that outputs will be mostly identical when the seed, parameters, and fingerprint all match, and it states plainly that determinism is not guaranteed (OpenAI Cookbook: Reproducible outputs with seed). That page dates from 2023, so confirm that your provider still accepts a seed parameter and returns a fingerprint before you rely on either.
As an Amazon Associate I earn from qualifying purchases.
The practical consequence is simple. Do not try to make live inference reproducible. Make the test input fixed. Once the model’s answer is saved as a fixture, the pull-request suite is testing your parser, your schema, your grading logic, and your application behavior against a known answer. A green run tells you that your code handles that answer correctly. It does not tell you that a new live call would produce the same answer.
Split the suite into two lanes
Most teams need two different questions answered, and they should not share one job. The first is whether a change broke something that was already working. The second is whether the model’s current behavior has moved. The first belongs in the fast gate that blocks merges. The second belongs in a controlled schedule.
| Property | Replay of recorded responses (pull-request CI) | Live-provider evaluation |
|---|---|---|
| Repeatability | High for the same committed fixtures and code | Best effort; provider guidance does not guarantee identical output |
| What it catches | Parser, schema, application logic, and grader regressions against known cases | Changes in current model or provider behavior |
| External dependencies | None when the harness is isolated from the network | Network, provider availability, and credentials |
| Inference calls | None during replay | One or more per case; cost and latency scale with suite size |
| Role in CI | Blocking gate on every pull request | Scheduled or manual job; treat failures as signals to investigate |
If your team accepts some run-to-run variability, you can let the live lane block releases too. Make that a decision you write down, not a side effect of wiring a live API call into the test file.
Define “good” before you write the harness
OpenAI’s evaluation best-practices guide describes a workflow that starts with defining an objective, collecting a dataset, defining metrics, comparing runs, and evaluating continuously (OpenAI: Evaluation best practices). For a deterministic suite, the objective should be concrete. “Summaries are good” is not testable. “Ticket triage returns a valid category, a priority from 1 to 5, and a summary under 280 characters” is.
Build the dataset to cover three kinds of cases: typical inputs, edge cases such as empty or very long text, and adversarial inputs that try to break the output contract. A suite made only of happy paths will pass while the code fails on the first unusual ticket in production.
Free tools Windows power users keep installed
One-click scans. No signup required.
Version the case material
Store each case as a reviewable file. A case should contain the input, the expected labels or rubric, the recorded response, and metadata. The SitePoint tutorial that covers this Vitest and Zod pattern describes recording fixture metadata such as the model version and a hash of the prompt (SitePoint: Testing LLM Output in CI with Vitest and Schema Validation, published 2026-09-23). The following fields are a practical minimum:
model: the exact model identifier used during capturepromptVersionor a prompt hash, so a prompt edit is visible in reviewcapturedAt: the date the response was recorded- Request parameters, plus
system_fingerprintwhen the provider returns it - The raw response exactly as received
Give the whole dataset a version or rely on the commit that contains it, so that a changed score can be traced to a changed example as well as to changed code. Do not refresh a fixture just to turn CI green. A fixture update changes the evaluation data, so it needs the same review as a change to the expected behavior or to a grading rule.
Write contract tests with Vitest
Vitest’s guide recommends organizing tests around inputs, outputs, side effects, and errors, with focused tests that each check one behavior (Vitest: Testing in Practice). Use describe to group cases and test or it for individual checks. An assertion that fails causes the test to fail (Vitest: Writing Tests).
Start with checks that code can decide exactly: required keys, expected types, allowed enum values, numeric ranges, explicit refusal markers, and correct handling of malformed model output. Here is a minimal fixture-backed test. The exact import paths depend on your project layout.
Recommended Free Tools
import { describe, test, expect } from 'vitest';
import { readFile } from 'node:fs/promises';
import { TicketTriage } from '../src/ticket-triage';
describe('ticket triage, replayed responses', () => {
test('ticket-017 recorded output satisfies the output contract', async () => {
const raw = await readFile('evals/fixtures/ticket-017.json', 'utf8');
const fixture = JSON.parse(raw);
const result = TicketTriage.safeParse(fixture.response);
expect(result.success).toBe(true);
});
test('malformed output is rejected, not silently accepted', () => {
const result = TicketTriage.safeParse({ category: 'unknown' });
expect(result.success).toBe(false);
});
});Vitest awaits promises returned by async tests and fails the test when the promise rejects. Its Test API reference documents a five-second default timeout that can be changed globally (Vitest: Test API). Replay tests should finish well within that limit. If a test does real network work, set its timeout explicitly, and keep such tests out of the pull-request lane so that a slow provider cannot make the ordinary suite flaky.
Validate structure with Zod
Zod schemas are a good way to state the output contract once and reuse it in the application and in the tests. The SitePoint tutorial uses schemas for structure, types, enumerations, and numeric bounds, then parses fixture content against them and reports the issues that come back (SitePoint tutorial). The schema from the example above could look like this:
Rank #4
import { z } from 'zod';
export const TicketTriage = z.object({
category: z.enum(['billing', 'bug', 'feature', 'other']),
priority: z.number().int().min(1).max(5),
summary: z.string().min(1).max(280),
});Check the method names against the Zod version you install. The schema proves conformance to the contract you wrote and nothing more. A response can pass every field check and still be wrong about the facts. Truth, completeness, and safety need their own rules and their own graders.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Grade only what each check can establish
Different grading methods support different claims. Keep the claims matched to the method.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Check type | What it can establish | Limit to state in the suite |
|---|---|---|
| Exact code assertion | Required fields, enum values, ranges, refusal markers, format rules | Covers only the rules you encoded |
| Similarity measure (lexical or embedding) | Textual drift against a recorded answer | No universal threshold exists; choose one per task, justify it, and do not present it as a quality score |
| LLM judge with a written rubric | Open-ended criteria such as tone or completeness | Validate the judge against human-reviewed cases before letting it block a merge |
An unvalidated judge is worse than no judge, because it produces a binary gate that nobody can explain. If you add one, record its rubric, its model, and how well it agreed with human reviewers.
Best Value
Wire the suite into CI
Vitest enters run mode automatically in CI and in non-interactive terminals, so a plain test command does not start the watch mode that developers use locally (Vitest: Command Line Interface). That behavior makes the pull-request gate predictable. A suggested sequence for the replay lane is:
- Install dependencies from the lockfile with
npm ci. - Run the type checker and linter your project already uses.
- Run the fixture-backed evaluation tests, for example with
npm run test:evals, which callsvitestagainst theevals/directory. - Upload the test report and a short summary that lists the dataset version and fixture count.
When the suite grows, Vitest’s CLI documentation covers sharding with --shard=<index>/<count> and merging reports from the shards. Use these when the run time becomes a real constraint, not as a default.
Run the live lane as a separate workflow, on a schedule or on demand. It needs its own secrets, and each run should record the model identifier, the configuration, and the returned fingerprint. Compare its results over time to see drift, and decide in advance which kinds of drift require a human to act.
The result is a gate that answers the question it is built for. Application and grading regressions stop merges. Provider variability shows up in its own report, where it can be read and acted on, instead of making ordinary pull requests fail unpredictably.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
