PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchA larger score in a “with skill” evaluation does not prove the skill caused the improvement. The first question is literal: Was the skill actually invoked? In an exploratory report by Driftproofhq, the biggest pass-rate difference appeared in plugin runs where the traces showed no skill invocation. A second question matters just as much: Did both arms sit at the ceiling? When every run passes, a binary score may hide differences in quality.
What did the evaluation compare?
Driftproofhq’s September 20, 2026 article describes tests of three skills from the public addyosmani/agent-skills pack: code review and quality, git workflow and versioning, and documentation and ADRs. The author used one case for each skill and compared with-skill and without-skill conditions. The report is exploratory: these cases do not establish how the skills perform across other tasks.
As an Amazon Associate I earn from qualifying purchases.
The article compares two evaluation methods, but they measure different stages of skill use. One asks whether an agent discovers and invokes a skill; the other supplies the skill text and scores what happens after exposure. Their numbers should not be treated as directly comparable.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Plugin evaluation: discovery plus use
In Anthropic’s built-in plugin evaluation, the skill is installed as a plugin. The model must discover and invoke it, then apply it while working with tools in a workspace. Runs are graded pass or fail. This method therefore includes activation behavior in the evaluation.
#1 Best Overall
Runner evaluation: application after guaranteed exposure
The author’s runner inserts the skill text directly into the model’s context, guaranteeing exposure, and assigns a continuous score from zero to one over multiple draws. It measures application given exposure, not whether the model would find or invoke the skill on its own. Driftproofhq reports using claude-opus-5 as both target model and judge for both approaches.
Was the skill actually invoked?
For documentation and ADRs, Driftproofhq reports three passes in three plugin runs with the skill, versus one pass in three runs without it. But the author also reports that the Skill tool was not called in any of the three plugin runs, or in a supplementary run. Based on those reported traces, the apparent difference cannot be attributed to skill activation: the skill was not invoked in the runs showing the improvement.
As Driftproofhq puts it, “If activation is not recorded anywhere, the difference is not attributable to the skill, whatever its size.” The key lesson is not that the skill had no possible effect; it is that a with-versus-without label is insufficient evidence of a causal effect when invocation is absent or unverified.
What to inspect in an invocation trace
- Whether the skill was available to the agent in that condition.
- Whether a tool call or other trace entry shows the skill was actually invoked.
- Whether the invocation happened before the behavior it is supposed to explain.
- Whether the control condition differed only in the intended way, rather than through another change in setup.
Without an activation record, the plugin test may still show a difference between two conditions, but it does not show that applying the skill produced that difference.
Did both arms sit at the ceiling?
In the code-review case, Driftproofhq reports three passes out of three in each plugin arm, so the pass-rate difference was zero. The same case received continuous-runner scores of 0.918 with the skill and 0.783 without it. Those are different measures: the binary result says both arms cleared the pass threshold, while the continuous scores distinguish performance within the passing range. The author summarizes the threshold problem: “Both arms had cleared the pass threshold, so pass/fail had nothing left to report.”
A zero pass-rate delta does not by itself show that a skill had no effect. It can mean the measure was too coarse to reveal a difference once both conditions passed. Conversely, a continuous score difference is not automatically proof of a meaningful or generalizable benefit, especially with few draws and a single case.
How much confidence do the reported numbers support?
The native evaluation used three runs per arm. With so few runs, one changed result moves the pass rate by 33 percentage points. That makes apparent improvements or declines especially sensitive to individual outcomes. Across the three tool-using tasks, Driftproofhq reports that all 18 sessions—nine with the plugin and nine without—passed both native grading and post-session verification. These figures are the author’s reported results, not independently verified benchmarks or broad estimates of skill effectiveness.
The continuous runner’s plus/minus figures are described by the author as sample standard deviations across draws. They indicate observed spread in those draws, not a confidence interval or a stated probability that the true effect falls within a range. The report is exploratory rather than a significance claim.
Best Value
- One case per skill: results describe those cases, not the skill’s performance in general.
- Few native runs: with three runs per arm, each outcome substantially changes the pass rate.
- Different measurement designs: discovery-and-invocation pass/fail and guaranteed-exposure continuous scoring answer different questions.
- Same model generated and judged: the author notes this creates a risk of self-preference in scoring.
Did passing checks mean the work was correct?
Not necessarily. Driftproofhq describes one ADR example that passed structural checks but asserted repository history that the fixture had not supplied. The article does not say this error occurred in all 18 sessions; it identifies one example. It illustrates why structural compliance and successful tool use are not substitutes for checking whether factual claims are grounded in available evidence.
For an agent-skill evaluation, a useful verification plan should therefore distinguish at least two questions: did the output meet the required format or workflow, and are its substantive claims supported by the task’s evidence? A pass on the first does not guarantee a pass on the second.
What a stronger skill evaluation should report
To interpret a with-skill result, readers need enough detail to identify what was tested and where the outcome came from. A more informative report would include:
- Invocation records for each run, not just the condition label.
- Separate results for discovery/activation and performance after guaranteed exposure.
- The scoring rule and threshold, alongside a finer-grained measure when pass/fail leaves both arms at the ceiling.
- Run counts and outcome variability, with uncertainty described accurately rather than implied by a small sample.
- Checks for both structural requirements and factual grounding.
- More than one case per skill before making claims about general effectiveness.
- Independent or otherwise carefully designed judging to reduce the risk that the model favors its own outputs.
These distinctions keep the claim aligned with the evidence: a plugin can fail to activate, a skill can be applied after forced exposure, and an evaluation can fail to detect a difference because both conditions pass. Those are separate findings, not interchangeable measures of skill quality.
Source and scope
The figures and observations in this article come from Driftproofhq, “The biggest improvement in my skill evaluation came from a skill that was never invoked,” published on DEV Community on September 20, 2026. The underlying report artifacts were not independently inspected, so the numerical results and trace descriptions are attributed to that account.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
