A model that returns a yes/no decision—or a structured choice with probabilities—instead of prose can make an agent’s next step easier to handle in code. But a clean output format does not guarantee a reliable decision. I built jev-tools, a Claude Code plugin using hosted OpenJev, and found that small samples, inconsistent scores, and the data sent to an API matter as much as the branching logic.
The measurements below are my own September 2026 smoke tests, not an independent benchmark. As I wrote in my DEV Community article, “The samples are small and run-to-run noise is real, so read these as smoke tests, not benchmarks.”
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
What I built
jev-tools is an independent Claude Code plugin written for Python 3.10+ using the standard library. It uses OpenJev, served by Codiv. I am not affiliated with Codiv, OpenJev, or TypeSafe AI. The model does not generate prose for the plugin: a request supplies state, such as text or JSON, plus typed questions; a response can contain a yes probability, a pick with probabilities, or an expected score with probabilities for each level.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →That structure lets ordinary code apply thresholds and decide what to do. It does not, by itself, show that the returned probabilities are calibrated or that a decision is correct. Codiv’s characterization of calibration has not been independently verified here. The model responses were described as taking tens to hundreds of milliseconds; the complete plugin operations took longer.
#1 Best Overall
Features in the plugin
- Rule hook: A
PreToolUsehook for Edit and Write checks a proposed change against rules inCLAUDE.mdorAGENTS.md. When it suspects a violation, it asks a stricter second question; in active mode, a confirmed violation can block the edit. - Skill picker: An opt-in feature selects an installed skill for a prompt and confirms the candidate before injecting it.
- Review precheck: Seven yes/no checks on a Git diff cover secrets, dependencies, authentication, schema changes, weakened tests, swallowed errors, and risky logic. The result routes the change to a fast or full review.
- Rule calibration: Replays recent commits against the project’s rules and labels them decisive, noisy, weak, or quiet before enforcement.
- File discovery: Uses two stages to find relevant files.
- Browser navigation: Chooses a next click from interactive page elements.
- Status: Reports whether installation wiring is alive.
Design choices and failure behavior
I kept thresholds in code, set failure behavior per feature, and used a shared standard-library client. The rule and skill hooks fail open if the API is unavailable; review precheck fails safe by routing to a full review. In shadow mode, hooks log what they would do without blocking or injecting. Those choices matter because a structured prediction still needs an explicit policy for uncertainty and outages.
What broke in live use
The first version passed offline tests. Live API calls and Claude Code sessions exposed problems that mocks had missed:
- 403 responses: Codiv’s edge rejected Python’s default
User-Agent. I added a custom one to the client. - A false rule alarm: An edit that read its API host from configuration scored 0.91 against a “do not hardcode API hosts” rule. A stricter second-look question vetoed the same edit.
- Unstable browser decision: The navigator called a goal-met page blocked. On identical calls, the goal-met probability varied around the decision bar—0.93 on one call and below 0.8 on another. I changed the rule to combine a likely-met goal with the absence of remaining clicks.
- Shallow skill match: The picker could inject a skill based on a weak match. I added a second-stage confirmation that includes the full skill description.
- Inconsistent file discovery: It scored 3 of 4 and then 0 of 4 after a rewrite. Rather than trust one run, I built an evaluation with six variants.
These are my reported observations and fixes; they have not been independently reproduced.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsWhat I measured
I ran the tests below against hosted OpenJev in September 2026. They were small smoke tests, not benchmarks, and the measurements should be read in that context.
| Feature or measure | Result | What the result does—and does not—show |
|---|---|---|
| Rule enforcer | 4 of 4 planted violations blocked and 4 of 4 clean edits allowed after the second-look check. I also reported correct blocking and allowing in real headless Claude Code sessions. | I wrote both the rules and planted edits, which may flatter this result; the sample is too small to establish general accuracy. |
| Skill picker | 3 of 3 correct on my roster: two matches and one correct “none.” | A tiny, author-selected roster is not evidence of broad skill-selection performance. |
| Review precheck | A rename-only diff went to fast review. A diff containing a hardcoded key, swallowed exception, and emptied test file went to full review with the right flags. | These examples exercise intended routing cases; they are not a representative review set. |
| Browser navigator | 5 of 5 steps on a synthetic login page. | I never tested it in a real browser session. |
| File discovery | Top-three hits were 4 to 6 of 8, compared with 4 of 8 for plain keyword counting; identical reruns differed by up to 2. | In this eight-query evaluation, discovery did no better than the keyword baseline, and reruns were inconsistent. |
| Latency and input | About 1 second per prompt or edit, about 2 seconds when confirming a violation, and about 5,000 input tokens per edit at 20 rules. | These are measurements of this implementation, not general latency or token-use claims about the model. |
| Rule calibration replay | On a different project, 20 rules over 24 real hunks produced no fires above 0.35. | That could mean the rulebook fit clean history—or that it was blind to relevant issues. The replay alone cannot distinguish those explanations. |
I also cited a one-week field report on a similar skill router in which about 5% of suggestions were followed by the agent. That figure is not from jev-tools; the source article’s displayed date does not state a year.
What leaves your machine
The plugin sends project context to api.codiv.ai. What is sent depends on the feature: it may include a filename and diff, project rules, a prompt and installed-skill descriptions, excerpts from candidate files, a Git diff, or a browser goal, URL, and interactive element names. Shadow mode still sends data because the plugin needs the API response to log a decision. Only turning the plugin off prevents these API calls.
Before a request, local pattern-based redaction attempts to remove common key and credential patterns, and files with secret-like names are excluded. This is not a privacy guarantee: unusual token formats can evade patterns, and redaction does not catch names, email addresses, or customer and employee information. The README also warns that internal business data may remain. I could not establish storage or retention practices from the public API documentation; the README says retention is unknown and advises checking the provider’s terms before sending anything you would not paste publicly.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHow to try it without starting in enforcement mode
The setup instructions require Python 3.10+ on your PATH and a free OpenJev key. Hooks call python; on systems that only provide python3, an alias may be needed. The README recommends storing OPENJEV_API_KEY at user level rather than committing it to a repository. It documents JEV_MODE as defaulting to shadow.
Best Value
- Install and check prerequisites. Confirm Python 3.10+ is available as
python, then configure the API key outside the repository. - Begin in shadow mode. Let hooks record their proposed decisions without blocking or injecting. Remember that this mode still sends the feature’s project context to the hosted API.
- Review the logs and calibrate rules. Look for false alarms and missed cases; replay recent commits with the calibration feature before enforcement.
- Only then consider active mode. The README recommends a week in shadow mode followed by log review and calibration before switching. Its documented defaults are a 0.80 rule-flag threshold and a 0.70 second-look confirmation threshold; these are defaults, not values tuned or validated for every project.
The README describes a free tier of 100 million input tokens. Quotas and service terms can change, so verify current terms before relying on that figure.
What the experiment supports—and what it does not
The useful idea is not that a yes/no model makes agent decisions trustworthy. It is that typed outputs can make a decision boundary explicit: the model returns a score or choice, and ordinary code owns the threshold, confirmation step, and action. The live false alarm, probability variation, and file-discovery reruns show why that boundary still needs testing.
The strongest-looking rule result came from four planted violations and four clean edits written by the same person who designed the rules. The browser result came from a synthetic page, and several features were exercised by script rather than through the actual skill loader. The file search result did not beat keyword counting in its small evaluation. Those limits make the project an early engineering report, not evidence that this architecture is generally more accurate than conventional alternatives.
The practical rollout lesson is narrower: make thresholds and outage behavior explicit, observe decisions before enforcing them, and judge any benefit against the data sent to a third party. Whether that trade-off is acceptable depends on the contents of the repository and the provider’s current terms.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
