A 30-record test can show how a particular model handled a defined set of security advisories—but it cannot establish a model’s general cybersecurity accuracy without a named model, preserved records, prompts, scoring rules, and results. Those details are not identified here, so no accuracy percentage or winner can be stated. The useful way to make this test credible is to publish enough information for someone else to repeat it and check every answer against its source.
What this test can—and cannot—tell you
An evaluation of 30 records measures performance on those records, under the tested conditions. It does not, by itself, measure how well a model handles all vulnerability questions, other advisory formats, or real-world security work. The result is especially hard to generalize if the records were hand-picked, chosen to expose particular failure modes, or drawn from a narrow time period.
As an Amazon Associate I earn from qualifying purchases.
The test’s title does not identify the model, its exact version, the 30 records, or any outputs or scores. Without those particulars, its measured accuracy is unknown. Do not fill that gap with results from unrelated model evaluations: a compliance score or a factuality result from another test answers a different question.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchEvaluation-set composition matters. Google’s guidance distinguishes adversarial testing from standard evaluation and recommends building sets suited to the system and risks being assessed. State whether the records were ordinary examples, adversarial selections, or chosen to test specific mistakes; explain the selection rule. Google’s evaluation guidance offers a framework, not evidence about this model’s performance.
#1 Best Overall
Make the 30 records reproducible
For each record, publish its CVE identifier, the retrieval date and time, and the exact text or data payload the model received. Preserve the payload or provide a stable archived copy. An identifier alone is insufficient to reproduce an earlier run because NVD records can change.
NIST’s National Vulnerability Database (NVD) provides vulnerability APIs and JSON 2.0 data feeds. NVD describes its 2.0 APIs as the preferred way to stay current and its feeds as a way to import data and synchronize later changes. For a repeatable test, document whether records came from an API, a feed, or another source, along with the query, selection criteria, retrieval timestamp, and extraction method. NVD’s vulnerability API documentation describes available fields and retrieval; its feed guidance explains JSON 2.0 feeds and synchronization.
Say which fields the model saw and which fields served as the answer key. Depending on the record and NVD’s analysis, available information may include a description, references, CVSS scores, and product applicability statements. Published and last-modified dates are also record fields; not every record necessarily contains every analyzed field. A test that gives the model a description alone is not equivalent to one that also supplies references or applicability data.
Be precise about the description used as ground truth. NVD says descriptions are maintained by the responsible CVE Numbering Authority (CNA), not controlled by NVD. NVD detail pages may show current descriptions and analysis descriptions, and they can differ. Identify which version you scored against and explain any discrepancy that materially affects a judgment. NVD’s detail-page guidance explains these distinctions.
Identify the model and preserve the run conditions
“Free model” is not a reproducible model identifier. Report the provider, product or model name, exact version if exposed, date and route used to access it, and any relevant free-tier constraints. If the interface does not reveal a stable version, say so; a rerun through the same product name may not use identical underlying model behavior.
Publish the full system and user prompts, the model’s complete outputs, and the settings that could affect them. Include whether browsing or other tools were enabled, the temperature or other available generation settings, and any context or input limits that changed what the model could see. If there were multiple runs, report each rather than selecting only the most favorable answer.
Rank #3
A reader should be able to distinguish a model’s knowledge from information supplied in the prompt. If the model could browse, follow links, or call tools, say what it could access; otherwise, describe the input as a closed-record task. These are different tests and should not be combined under a single accuracy figure.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesScore claims against the record, not just the wording
Set the scoring rubric before inspecting responses. A practical method is to break each answer into factual claims and compare each claim with the source record and the fields the test designated as authoritative. Semantic correctness matters: different wording can convey the same fact, while a fluent answer can introduce an unsupported detail.
- Correct claims: supported by the designated record or other explicitly permitted source.
- Omissions: required or material details from the rubric that the response leaves out.
- Unsupported additions: claims that the permitted source does not substantiate.
- Abstentions: refusals or statements that the model cannot determine the answer.
- Material errors: incorrect claims likely to change a reader’s understanding of the vulnerability or affected product.
Report at least two complementary error measures: the share of claims containing factual errors and the share of responses with at least one material error. Also report omissions, unsupported additions, and abstentions separately. Keep exact-match grading separate from semantic correctness; an answer need not reproduce the record verbatim to be accurate.
OpenAI’s published factuality evaluation uses both claim-level errors and responses containing at least one factual error as measures. It also cautions that deliberately difficult prompt sets are research signals, not estimates of error prevalence in ordinary production use. That is a reporting precedent, not a result for this 30-record test. OpenAI’s factuality evaluation notes describe the distinction.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep CVSS severity separate from factual accuracy
A model can correctly repeat a CVSS score and still misstate the affected product, vulnerability description, or conditions. Conversely, a factual summary can be accurate without discussing CVSS. Score those dimensions independently.
Recommended Free Tools
NVD’s wording is direct: “CVSS is not a measure of risk.” A CVSS score is a qualitative severity measure, not a complete assessment of risk to a particular organization. Do not treat a correct severity label as proof that the model accurately understood the advisory, or use severity as a substitute for checking claims against the record. NVD’s vulnerability detail-page guidance provides this context.
Best Value
What a complete report should publish
A rerunnable report should include the following materials in one place:
- Provider, model name and version—or a clear statement that the interface does not expose a stable version—plus the access route and run date.
- All 30 record identifiers, inclusion criteria, retrieval timestamps, exact snapshots, and extraction method.
- The exact system and user prompts, complete model outputs, enabled tools, and available generation settings.
- The answer-key source and fields, including which description was used where NVD and CNA text differs.
- A rubric defined before scoring, the annotator procedure, per-record scores, aggregate measures, and any disagreements and how they were resolved.
- Limitations, including selection bias, records or fields omitted, interface changes, and any unequal input or context constraints.
If comparing multiple models, keep the records and prompts constant and state any differences in access, tools, or context limits. Report factual correctness, coverage of material details, unsupported claims, source attribution, abstention behavior, and stability across reruns as separate axes rather than collapsing them into a single unexplained score.
How to interpret a 30-record score
Even with complete materials, treat a score from 30 records as a result for that sample and configuration. Unless a documented sampling method supports broader inference, it is a small, selected test—not a reliable estimate of accuracy across all cybersecurity questions or users. A test intentionally weighted toward tricky cases may be useful for finding failure modes, but it should not be presented as typical everyday performance.
Reproducibility also has limits when a provider changes a model behind an unchanged product name. Preserved inputs and outputs make the original run auditable; they cannot guarantee that a later run will use the same model. State what the interface exposed and what a future tester can actually hold constant.
For this specific 30-record test, no model identity, record list, prompt set, scored outputs, or measured result is established. The defensible conclusion is therefore that its accuracy cannot be determined from the available particulars—not that it passed or failed.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
