Combining five useful warning signals made one self-hosted LLM corruption guard unusable: its first fusion rule falsely flagged 42.9% of clean production text. In Farid Aghayev’s 2026 case study, the problem was not simply which detectors were present, but how their scores were combined. Replacing noisy-OR with a rule led by the strongest signal, then tuning individual detectors, brought the author-reported false-positive rate down to 0.000% in that evaluation. Those figures describe one system and test set, not corruption guards generally.
What this guard is designed to catch
Aghayev uses “decoding corruption” operationally for a generation stream that stops answering and veers into unrelated or malformed material. That is different from a hallucination in the ordinary sense: a response can remain coherent while making false claims, whereas this guard looks for visible stream derailment. It is an anomaly detector, not a truth verifier.
As an Amazon Associate I earn from qualifying purchases.
That distinction matters because the system looks for unusual patterns in the stream, not whether claims are factually grounded. Fluent output that stays on topic, uses the expected script, and looks statistically ordinary can evade it. The author says subtle factual errors are handled separately with grounding or factuality methods.
How five detectors read one stream
The implementation maintains a shared state object that ingests characters incrementally. At checkpoints, detectors inspect the accumulated state rather than each making a separate pass over the output. The five signals are complementary in name, but some can react to the same benign feature of a text.
#1 Best Overall
| Signal | What it examines | Potential overlap on clean text |
|---|---|---|
| Deterministic rules | Measures such as digit fraction, symbol density, and the longest run of identical characters. | Figures, punctuation, and formatted data can raise individual measures. |
| Character n-gram surprise | Online character-pattern likelihood, learned from the stream’s clean prefix. | Unusual but valid codes or domain-specific text may be surprising. |
| Count-Min Sketch repetition | Frequency of character shingles tracked in a compact sketch. | Repeated terms or structured passages can appear without corruption. |
| Rolling SimHash | Similarity to a fingerprint captured at the end of the clean prefix. | Legitimate changes in style or content can affect similarity. |
| Character entropy | Variation in characters, intended to help identify patterns such as table dumps and degenerate loops. | Structured or repetitive material can change entropy without being an error. |
Aghayev describes the implementation as bounded-memory and reports that it processes about 50,000 characters per second on one CPU core. That is a figure for his implementation, not a general performance benchmark.
Why noisy-OR produced a 42.9% false-positive rate
The guard’s early production text included legitimate economic writing with many figures, percentages, variable codes, and Markdown tables. Several detectors could register mild warnings on the same clean passage. The initial noisy-OR fusion accumulated those weak scores into a strong combined score, so the guard flagged clean text far too often.
In Aghayev’s evaluation, noisy-OR produced a 42.9% false-positive rate on clean production text. For the described workflow, that rate made the configuration unusable. The key failure was the assumption that multiple individually modest warnings should automatically add up to strong evidence of corruption. When signals respond to overlapping characteristics of healthy text, their combined score can exaggerate the risk.
How the revised fusion changed the result
The author changed the combination rule so the strongest individual signal dominated, with a limited bonus when at least two detectors crossed a threshold. He reports that this reduced the false-positive rate to 0.024%. Further detector-level adjustments brought the reported rate to 0.000% in his evaluation.
| Configuration | Author-reported false-positive rate | What changed |
|---|---|---|
| Initial fusion | 42.9% on clean production text | Noisy-OR allowed mild detector scores to accumulate. |
| Revised fusion | 0.024% | The strongest signal dominated, with limited corroboration from a second threshold-crossing detector. |
| Further tuning | 0.000% | SimHash contribution was limited, surprise was gated behind repetition, and sketch repetition was scored by rate rather than maximum count. |
All three rates are reported by Aghayev for his system and evaluation; they are not independently replicated or a promise of equivalent results on other traffic. The case study’s broader engineering point is that a detector roster does not determine performance on its own: the fusion rule can decide whether overlapping weak warnings become useful corroboration or a flood of false alarms.
What the evaluation establishes—and what it does not
Aghayev reports replaying the finished guard across 121 historical messages. It flagged two messages, which he identified as prior corruptions, and did not flag the other 119. The positive examples in the benchmark were synthetic: four corruption classes were injected into clean text at controlled onsets. The author says those examples match known historical failures, but also cautions that production may contain patterns the test set did not model.
This is useful evidence about the system’s behavior on those messages and constructed examples, not proof that it catches all real-world corruption or that its clean-text performance will hold under distribution shift. The 0.000% figure should be read in that limited context.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Calibrating thresholds without overclaiming certainty
The described threshold-selection method uses split-conformal calibration on held-out clean streams with an operator-selected false-alarm budget. A finite-sample guarantee from this approach depends on calibration examples and future examples being exchangeable: broadly, they must be drawn under conditions that make their order irrelevant and their distributions comparable. A changing workload, new formats, or previously unseen corruption patterns can break that assumption. Calibration therefore does not guarantee the same false-alarm behavior after traffic shifts.
Streaming behavior and integration costs
The guard delays forwarding output initially so it can establish a clean baseline. Aghayev’s protocol holds the first 350 characters for inspection and sends none during that phase. If the text is judged clean, the guard flushes the buffer, freezes its self-calibrated baselines, and checks each subsequent 400-character interval. It requires two consecutive qualifying checks before aborting. The author estimates the initial hold adds about one second at typical decoding speeds.
Abort handling also matters beyond detection quality. In the described integration, abort is raised as an exception inside the token sink. The transport initially treated it as an upstream failure, which could trigger its HTTP circuit breaker. The operational lesson is to represent and handle caller-side cancellation correctly in the transport rather than letting an intentional guard abort look like an infrastructure fault.
When this approach is—and is not—a fit
- Useful for: catching conspicuous stream derailments with measurable pattern changes, especially when a service needs to stop forwarding suspect output while it is still being generated.
- Requires care with: domain-specific or highly structured text, where figures, codes, tables, repetition, or unfamiliar character patterns may trigger individual signals.
- Not a substitute for: factuality checks, grounding, or other methods that assess whether a fluent answer is true.
- Needs ongoing validation: clean calibration streams and corruption examples representative of the traffic the system will actually see.
For developers building a multi-signal guard over stylistically varied output, the central lesson from this case is to test the fusion rule against clean, representative traffic—not merely to add detectors. Signals that look useful in isolation can combine badly when their false alarms overlap.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
