Recommended Free Tools
Debashish Ghosal reports that he reduced a planned 2,490 live agent-tool test runs to 206 by splitting coverage into two goals: exercise every scenario at least once, then exercise every gate decision type in each framework. He says the smaller plan retained the coverage he wanted—but this is one practitioner’s account, not an independently replicated benchmark, and it depends on assumptions that may not hold in other systems.
What the 206-run plan was designed to prove
In a September 22, 2026 DEV Community post, Ghosal describes field-testing agent-tooltrust, an open-source gate for AI agent tool calls. The test space contained 83 agents and 30 scenarios, producing 2,490 possible agent-scenario combinations. Instead of running every combination against a live model, he divided the live test into breadth and depth.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
- Breadth: Did every scenario run at least once across the agent set?
- Depth: Could every framework surface each of the gate’s four decision types?
His project already had 2,490 deterministic assertions, which he says exercised every engine decision path without LLM calls. The live tests therefore supplemented those assertions; they did not replace them. Ghosal’s account and figures are in his DEV Community post.
How the two live-test plans worked
Plan A: scenario breadth
Ghosal reports 83 runs, assigning one scenario to each agent. Across those runs, he says, the plan exercised all 30 scenarios across 10 frameworks and 5 agent classes, with a result for each of the 83 runs.
#1 Best Overall
Plan B: decision-type depth
A further 123 runs aimed to exercise all four decision types—allow, audit, escalate, and deny—within each framework. Ghosal reports 116 of 123 results, or 94%, and seven not-available outcomes.
Together, the plans used 206 live runs rather than all 2,490 possible combinations. Ghosal describes that as roughly a 12-fold reduction for the coverage goals he defined. The claim is about fewer live model calls after deterministic engine-path testing, not fewer tests overall or proof that every agent-framework combination behaves correctly.
What the full matrix adds—and what the smaller plan trades away
The comparison below reflects the goals and estimates in Ghosal’s account, not a controlled comparison across projects.
| Dimension | Full cross-product | Breadth-and-depth design |
|---|---|---|
| Scenario breadth | Every scenario is paired with every agent. | Ghosal reports all 30 scenarios exercised at least once across the agent set. |
| Decision-type depth | Every agent-scenario pairing is tested, so the matrix includes many opportunities to observe decisions. | Designed to surface all four decision types within each framework; Ghosal reports 116 of 123 results and seven not-available outcomes. |
| Agent-framework interactions | Can expose behavior specific to a particular agent-framework combination. | Drops many combinations, so cross-cutting interactions may go undetected. |
| Live model calls | 2,490 possible runs for 83 agents by 30 scenarios. | 206 reported runs across Plans A and B. |
| Time and review burden | Ghosal says each real LLM call took 30–80 seconds and estimates about 2.7 hours for the full set with 10 workers, before debugging overhead. | Fewer calls reduce live-run volume, but the post does not quantify equivalent runtime or debugging effort for the 206-run design. |
Why the seven not-available results are not wrong decisions
Ghosal says all seven Plan B failures occurred because the model did not call the guarded tool. He labels these outcomes “not-available,” distinguishing them from an unexpected-decision error, in which the engine returns the wrong verdict. He reports no unexpected-decision errors among the seven.
That distinction matters when interpreting a run: a model that never invokes a tool has not demonstrated that the gate made the wrong decision. Ghosal gives an example of a small local 4B model that sometimes answered in prose when prompted with five tools. This is an example from his test, not a general performance finding about 4B models.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When this reduction is—and is not—safe to borrow
The key assumption: engine and adapter independence
Ghosal says the covering design assumes independence between the gate engine and its framework adapter. He considered that assumption valid for this project because the engine is framework-agnostic. If your application has behavior that emerges from a specific agent working with a specific framework, the omitted combinations could contain important failures. In that case, run the full cross-product before reducing the live matrix.
Keep deterministic tests as the first line of defense
The smaller field test only makes sense alongside deterministic checks of engine behavior. Ghosal describes the live field test as a second line of defense: the $0 assertion-failure result is trustworthy only if code review has already caught actual bugs. The field-test sampling strategy cannot establish that the engine is correct on its own.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Treat the coverage claim as project-specific
The reported design shows one way to allocate live calls against two stated coverage goals; it does not establish a universal sampling rule or prove that “same coverage” transfers to another test suite. Ghosal acknowledges that he cannot fully answer where deterministic testing should stop and real-agent testing should begin. The boundary remains a judgment call that depends on what the live model and framework integrations can reveal in a particular system.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
