What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
In a first-person postmortem on DEV Community, published September 22, 2026 and updated September 23, developer rizkynandapr reports that his US home-services intake agent passed 22 of 30 scripted test calls on its first full run. He attributes the eight flagged failures to his own prompt, schema, and test scripts. The reported failures are as much about test design as about model behavior, and the higher scores he reports later came from changed criteria and models, so they do not add up to a clean accuracy rate.
What the agent does
The agent takes requests for HVAC, plumbing, and roofing work in the United States over phone, SMS, or a web form. It turns each request into a structured JSON record for a contractor’s system. Everything below comes from the author’s own test runs. It is not an independent benchmark of agent reliability, and the test set is small and author-written.
As an Amazon Associate I earn from qualifying purchases.
How the test was built
The author wrote 30 scripted test calls, ten per trade. Each call was scored against nine criteria. Four are deterministic checks: schema validity, urgency, required fields, and emergency type. The other five were meant for a second model acting as a judge.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteA later judge run used gpt-4.1-mini as both agent and judge. That run was therefore not a direct repeat of the first full run, and the scoring setup differed.
The first full run: 22 of 30 passed
The first full run passed 22 calls and flagged eight. The post first described those eight as failures in the deterministic half of the checks. After additional judge scoring, and as the harness and scoring details came to light, the author revised that reading. The problems he lists fall into two groups.
Prompt and schema defects
- The prompt pointed to the schema by file path. A path in a prompt does not give the model the file’s contents, so the model never saw the schema it was meant to follow.
- The emergency guardrail said to stop intake but did not say when intake could resume.
- Residential property classification was not specified.
- Urgency rules did not cover repeat failures or commercial tenants.
- The mapping from caller description to emergency type was undefined.
Test-input defects
Some scripted caller lines never supplied the address or callback number that the required-fields check demanded. A call whose script never gave the agent that information could not pass a check that required it. The fix was to make the script supply the data or to make the expected outcome account for its absence.
Two defects the original checks missed
Raw JSON in the middle of a call
In one call, the caller said “Okay hang on,” and the agent emitted raw JSON. The author reports it happened three times in that call. The final record was still schema-valid, so the checks passed the call. The original criteria looked at the final record, not at everything the agent produced along the way.
A correct record followed by an empty one
In a web-form case, the agent had already produced a correct record. The runner then sent an unconditional disconnect nudge, which led to a second, empty record. The checker preferred the last turn, so it scored the empty record. A commenter suggested the pattern could be caught by a cheap deterministic check, which the author then added as a criterion rather than a prompt change.
A quota error erased finished work
A quota failure stopped the runner. Because it wrote its report only at the end of the run, six cases that had already been scored were lost.
Fixes and the scores that followed
The author reports the counts below in roughly this order. The criteria changed between stages, so read them as checkpoints rather than one accuracy rate.
| Stage, as the author reports it | Reported result | What changed |
|---|---|---|
| First full run | 22 of 30 passed; 8 flagged | Original nine criteria, before fixes |
| Later full run after fixes | 29 of 30 passed | Structural and runner fixes applied |
| Retained transcripts rescored with the record-counting criterion | 25 of 30 passed | Whole-transcript record count added |
| Card-number case rerun after a redactions fix | 26 of 30 passed | One case rerun after its fix |
| Judge run, gpt-4.1-mini as agent and judge | Four remaining model failures identified | Not a direct repeat; the last four roofing cases had not been rerun after the structural and runner-nudge fixes at the time of the initial update |
The new record-counting check, which the commenter described as “exactly one record in the transcript, and no caller turn after it,” flagged seven of the 72 stored transcripts. Six of those seven had previously passed.
Why the author will not claim 30/30
He writes, “So I’m not going to tell you it’s 30/30.” The test set is small, scripted, and written by him, and it changed as defects were found. The models and scoring configurations also differed across runs. A single headline number would hide all of that, so none of the counts above should be merged or quoted as the agent’s score.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to check in your own agent evals
Each lesson below maps to one of the failures above.
- Give the model the schema text. Load it from the same file the validator uses and place the contents in the prompt. A file path is not the schema.
- Make every scripted call passable. Each required field must appear in the caller’s lines, or the expected outcome must account for its absence.
- Define what follows a stop rule. In the author’s emergency flow, once a caller confirms they are outside, the minimum follow-up is the address and callback number, asked one at a time. This is the author’s rule for his intake workflow. It is not medical or emergency-response guidance.
- Read transcripts of calls that passed. A valid final record can hide raw JSON emitted mid-call.
- Make the runner durable. Save partial progress after each case, identify quota errors specifically, and resume completed work rather than discarding it.
- Test the checker itself. Commenters on the post described successive edge cases in counting JSON records, including fenced output and nested envelopes. The author reports shipping a fix after checking it against the real runner. That is an iterative lesson, not proof that the final scanner handles every output format.
The emergency guard and the author’s other materials
For emergency records, the author built a free n8n workflow, discussed in an n8n Community post dated September 23, 2026 and published as the GitHub repository rizkynandapr/n8n-intake-record-guard. It pages a human when an emergency record lacks an address or callback number, flags placeholder fields, and routes records onward. The n8n description adds an optional comparison against caller ID when the platform supplies that number. The repository is an implementation and may change over time.
The author also sells Dispatch Kit, a paid digital package that contains the prompt, the guardrails, all 30 eval cases, and the runner.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
