A small local model can help triage AI-incident reports, but the available evidence does not support a general accuracy promise. One project evaluation reports a Qwen2.5 7B Instruct model achieving an incident macro-F1 of 0.749 on a frozen set of 131 examples. It also documents confident false dismissals of subtle incidents. Treat that score as a result for one model, prompt, label scheme, dataset and local setup—not a forecast for another workflow.
What the local-model evaluation found
The Open-source AI Incident Observatory documents an offline workflow comparing a majority-class baseline, a deterministic keyword baseline and an Ollama-hosted model such as qwen2.5:7b-instruct. The evaluation first separates relevance triage—relevant, not_relevant or insufficient_evidence—from incident-type classification. It scores incident type only for examples judged genuinely relevant, so off-topic reports cannot inflate that score. The project’s evaluation and methodology documentation describes the setup and results.
As an Amazon Associate I earn from qualifying purchases.
The frozen test set contains 131 examples: 93 concrete incidents across nine types, 24 hard negatives and 14 under-evidenced reports. The difficult cases include misleading trigger words in non-incidents, incidents described without expected keywords, and similar categories such as goal persistence and resistance to correction.
Recommended Free Tools
| Measure | Reported result |
|---|---|
| Incident macro-F1 | 0.749 |
| Relevance macro-F1 | 0.747 |
| Overall accuracy | 0.733 |
| Selective accuracy at 0.939 coverage | 0.724 |
| Abstention precision | 0.88 |
| Abstention recall | 0.50 |
| Keyword baseline incident macro-F1 | 0.273 |
| Majority baseline incident macro-F1 | 0.025 |
These are the project’s reported results, accessed in 2026, for its particular test set and evaluation setup. They are not independent replication results or a representative estimate across incident datasets. The project also reports approximately 4.2 seconds per classification on a laptop without a GPU and $0 cost in that local setup; runtime and cost will depend on the device and deployment.
Read the metrics together. Coverage is the share of examples the model classifies rather than abstains on; selective accuracy is correctness among those committed classifications. Abstention precision and recall indicate how well the model uses its insufficient-evidence option. Macro-F1 gives each class equal weight, helping expose weak results on rarer categories that overall accuracy can hide.
Where the model can fail
The project reports high-confidence false dismissals of understated incidents, including a cleanup script that emptied an S3 backup bucket and a system that claimed tests passed when the test suite had not run. It also found that many harmless_malfunction cases were classified as not_relevant, a boundary the project identifies as difficult.
Rank #2
- The 2024 ERG guide helps satisfy 49 CFR 172.602 DOT requirement. This requirement states that hazmat shipments be accompanied by emergency response info.
- Pocketbook aids in emergency preparedness, planning, and training with ERGs numerically indexed and color-coded to help emergency responders find vital information fast.
- 2024 Updates: The Pipeline and Hazardous Materials Safety Administration (PHMSA) released a comprehensive summary of updates. Most significantly a QR code on the back cover that provides access to critical incident reporting information.
- Other changes for 2024 have been made to continue to provide the most accurate emergency response information to help all front-line persons and all first responders stay safe during transportation emergencies.
- Specifications: 4" x 5 1/2" Pocketbook Size, English, Spiralbound. Copyright 2024.
Evaluation plumbing matters too. An early schema bug rejected a null incident type, producing an implausible zero score for one class and a 34% abstention rate. After the schema was corrected, the reported not_relevant F1 rose to 0.69 and abstention fell to 6%. A parser, prompt or label-handling defect can therefore look like a model failure—or mask one.
Why the label and taxonomy change the answer
“Classifying an AI incident” can mean deciding whether a report describes an incident, assigning an incident type, rating harm severity, estimating regulatory risk, or identifying a cause. These are distinct tasks, and a model’s result on one does not establish performance on the others. Define whether labels are single-label or multi-label, what counts as evidence, and who assigned or adjudicated the labels.
Cause labels deserve particular care. An incident report may document a system’s goal or behavior without revealing the technical mechanism that produced it. A failure-cause taxonomy paper proposes moving from system goals, which may be observable, through methods or technologies to technical causes that often require expert analysis. Avoid presenting a model’s inferred cause as a fact established by the report. The taxonomy paper explains this distinction.
RiskNet describes a multilingual, news-derived incident resource with alignment and multidimensional labels including domain, cause and severity. That work can inform benchmark design, but its dataset description is not a performance result for a small local model. RiskNet’s paper outlines the resource and its goals.
What human-reviewed results do—and do not—show
MIT’s AI Incident Tracker update from June 30, 2026, describes a pilot comparing seven candidate models with its existing pipeline across five taxonomies: harm severity, EU AI Act risk level, causal taxonomy, domain and subdomain. The project found EU AI Act risk level the most difficult task, and targeted prompt clarifications improved results. Some frontier models met or exceeded the human baseline on three taxonomies without prompt changes; after targeted revisions, Opus 4.6 matched or exceeded that baseline across all five on the pilot sample. Those findings show that model choice and wording matter, not that small local models match expert judgment.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The human reference was limited: consensus labels came from two reviewers per incident across 10 incidents. The update also reports that 43% of errors were risk-level overestimates and 57% underestimates across its tested model and prompt combinations. That split applies to this pilot, not to other systems or taxonomies. MIT’s June 2026 update gives the pilot’s scope and qualifications.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to test a model for your own triage workflow
Use the exact report population, label definitions and operating constraints you expect in deployment. Keep prompt development separate from a human-reviewed holdout set so tuning does not turn the final evaluation into another training exercise.
- Define the decision. Specify whether the model handles relevance, incident type, severity, cause or regulatory risk, and whether it may assign multiple labels or abstain.
- Build a representative test set. Record class counts and include hard negatives, near-neighbor categories, paraphrased incidents and reports that lack enough evidence. Document label sources and reviewer adjudication.
- Compare on identical examples. Test each model or baseline against the same held-out reports and annotation rules. Report per-class precision and recall as well as macro-F1 and overall accuracy.
- Measure commitment and error costs. Report coverage, selective accuracy at that coverage, abstention precision and recall, and calibration. Review false dismissals separately from false inclusions because their operational costs may differ.
- Test deployment conditions. Measure latency and memory use on the actual device, and assess privacy, data handling, operating cost and reproducibility. Do not transfer a laptop runtime or cost result to other hardware.
- Inspect failures before action. Have reviewers examine confident mistakes, schema errors and ambiguous labels. Keep human review in the loop before classifications trigger consequential decisions.
How strong is the broader evidence?
The directly relevant local-model result is useful as a concrete starting point, but the available evidence does not establish independent, representative performance for small local models across AI-incident datasets and taxonomies. MIT’s pilot concerns a different model set and has a small human-reference sample. Dataset and taxonomy publications help explain how to build better evaluations, but they do not close that performance gap.
Results from another domain should not be substituted. A 2026 cybersecurity incident-classification study reports strong results for particular model and preprocessing combinations, but it evaluates cyber-threat intelligence, not AI incidents. Its scores are not an estimate of AI-incident classification performance. The cybersecurity study is relevant only as an illustration that model choice and preprocessing can matter in a different incident-labeling task.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
