Finding every vulnerable example does not prove that an AI model can recognize a security fix. In a small benchmark called Attacker-Reachable Sink Triage (ART), all seven tested models caught every vulnerable snippet, but some still mislabeled patched code as vulnerable. ART measures both questions: “Did you find a bug?” and “Did you respect the fix?”
Why vulnerability detection is only half the test
A security review needs to distinguish code with an attacker-reachable flaw from code where a control blocks that path. A model that flags both versions of a vulnerable function may look excellent on a detection-only score, even though its warnings are less useful after developers apply a fix.
As an Amazon Associate I earn from qualifying purchases.
ART’s author captured the distinction this way: “A model that labels the second snippet reachable_vuln isn’t a worse detector — it’s a worse patch reader.” The benchmark’s central idea is that detecting a flaw and recognizing its repair are separate capabilities.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →How ART tests patch recognition
Minimal vulnerable-and-patched pairs
ART uses synthetic minimal pairs: each pair has the same function shape and identifiers, but one security control changes between the vulnerable and patched snippets. The prompt supplies the code snippet and language; twin IDs, labels, and rationales are withheld. The design aims to make the changed control, rather than familiarity with a published CVE, decisive.
#1 Best Overall
One PHP SQL-injection example contrasts raw concatenation of attacker-controlled SQL with a version that casts input and uses a prepared statement. The author says the patterns are meant to resemble WordPress-plugin-style PHP and Flask/Django-request-style Python.
Labels, controls, and scoring
The author reports eight vulnerable/patched pairs spanning SQL injection, cross-site scripting, authentication bypass, command injection, path traversal, local file inclusion, and insecure deserialization across PHP and Python. Six safe or vacuous controls are included to check whether a model flags snippets that are not actionable vulnerabilities.
The main task, art-label-triage, asks the model to choose among four labels:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
reachable_vuln: a vulnerability is reachable through the relevant code path.patched: a security control addresses the issue in the paired example.safe: the snippet does not present the tested vulnerability.vacuous_noise: the example is not a meaningful vulnerability case.
Its composite score weights vulnerable accuracy at 40%, patched accuracy at 40%, and filler accuracy at 20%. The author identifies this label-triage task as the headline metric. ART also includes art-overconfidence-trap, which asks whether patched twins contain a confirmed exploit (the gold answer is no), and art-proof-marker-poc, which scores a minimal lab proof-of-concept marker as 1.0 or 0.0.
What the reported model results show
In the author’s ART label-triage v6 run, all seven models achieved 1.000 raw vulnerable accuracy: each caught all eight vulnerable twins. Results diverged on patched examples and controls. The figures below are the author’s reported ranked results from task runs’ rewards.score, not an independent replication or a current general-purpose model ranking.
| Model | ART score | Raw vulnerable accuracy | Patched accuracy | Controls | Twin Gap | Reported cost (USD) | Reported latency |
|---|---|---|---|---|---|---|---|
| gemini-2.5-pro | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | $0.181 | 7.9 s |
| gemini-3.5-flash | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | $0.108 | 2.9 s |
| gemini-3.7-flash | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | $0.028 | 9.1 s |
| gemma-4-31b-it | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | $0.007 | 12.3 s |
| claude-sonnet-4-5-20250929 | 0.950 | 1.000 | 0.875 | 1.000 | 0.125 | $0.060 | 3.1 s |
| claude-haiku-4-5-20251001 | 0.850 | 1.000 | 0.625 | 1.000 | 0.375 | $0.020 | 1.7 s |
| gpt-5.4-nano-2026-03-17 | 0.817 | 1.000 | 0.875 | 0.333 | 0.125 | $0.004 | 1.3 s |
ART defines Twin Gap as vulnerable accuracy minus patched accuracy. A zero means accuracy is equal across the two kinds of twins; a positive value means more patched examples were incorrectly flagged. The author’s summary was: “Zero means the model respects fixes; positive means it over-flags patched code.”
Rank #3
Haiku’s reported 0.375 gap corresponds to three of the eight patched twins misclassified. With only eight patched examples, each miss changes the gap by 12.5 percentage points. The author reports an exact sign-test p-value of 0.25 for those three misses and cautions against treating the results as a large-sample ranking.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →What the errors and label review reveal
Some apparent model errors were benchmark-label errors
The author says all seven models disagreed with two original labels in the same direction. Adjudication found the models’ answers correct: an escaped-input filler was relabeled patched, and an example that replaced pickle.loads with json.loads was relabeled safe. The author says the original labels had capped scores at 0.917; after adjudication, the top cluster reached 1.000. This is a useful warning for any benchmark: a score depends on the quality of its gold labels, and unanimous disagreement can be a reason to inspect the key rather than automatically count model failures.
Reported misses need transcript-level interpretation
The author describes two Haiku mistakes: a path-traversal twin where the model allegedly ignored basename("../../../etc/passwd"), and an authentication twin where it acknowledged current_user_can but still labeled the snippet vulnerable because of another risk. These are the author’s interpretations of examples, not independently tested findings.
There is also a caution about a separate proof-marker result: the author reports a Sonnet score of 0.0 across retries after a provider returned an empty completion (86 prompt tokens and an empty message). That result illustrates why transcripts matter; a failed or empty response should not automatically be interpreted as a model’s security judgment.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What ART can—and cannot—establish
It is a small diagnostic, not a broad ranking
Eight pairs can expose a difference between catching flaws and respecting fixes, but they cannot establish how models perform across real codebases, languages, vulnerability classes, or changing model versions. A single patched miss shifts the patched-accuracy figure substantially. The model names, costs, and latencies in the table belong to the author’s reported run and should not be read as current prices, response times, or general capability claims.
Fix recognition can be confused with fix-token recognition
Because each patched twin contains a valid fix, a model might learn to respond to a familiar surface cue without reasoning through reachability or checking whether every vulnerable path is closed. A DEV Community commenter suggested adding decoy cases that contain fix-like tokens but retain a vulnerable path. That is a proposed extension, not a demonstrated defect in ART.
Prompt changes did not remove every failure in the reported probes
The author says a red-team persona did not systematically increase overclaiming. Requiring a forced data-flow chain of thought also did not eliminate Haiku’s overconfidence-trap error; its score changed from 0.625 to 0.50. These are narrow observations from the reported probes, not evidence that those prompting approaches generally help or harm security review.
How to use these results in practice
- When evaluating a code-review model, score patched examples separately from vulnerable examples; a single detection-accuracy number can hide over-flagging.
- Include safe and vacuous controls so the test measures whether the model can avoid unsupported findings as well as find real ones.
- Audit disputed benchmark labels and inspect the model transcript before treating a surprising score as a capability failure.
- Read small-sample gaps as diagnostic signals, not stable rankings. Validate any deployment decision on representative code and realistic review workflows.
The source article is “100% vuln detection wasn’t enough: measuring whether AI respects the patch” by unit life on DEV Community. The page shows a September 24 posting date without a printed year. The article links to the Kaggle ART collection, task pages, and the MIT-licensed mziqudhd92/kaggle-art-benchmark repository; current availability and program status are not established here.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
