Free tools Windows power users keep installed
One-click scans. No signup required.
A safety reviewer can fail by approving unsupported claims—but it can also fail by rejecting sound repairs. In a post-mortem published on DEV Community on August 23, 2026, Joshua Hernandez describes how an automated data-catalog reviewer rejected most proposals while its test suite still gave blanket rejection a passing score.
What the Arbiter rejected—and why it mattered
Hernandez describes ARGUS as a data-catalog governance swarm whose Arbiter checks proposed metadata repairs before they are written. In one live sweep, the Arbiter rejected 86 of 112 proposals, including proposed PII labels for cust_first_name, billing_zipcode, and shipping_address_line1. PII governance in that sweep was 0.0 percent, according to Hernandez. These are the author’s figures for this project and sweep, not benchmark results.
As an Amazon Associate I earn from qualifying purchases.
The reviewer gave confident reasons for its refusals, but produced no PII governance output. The underlying issue, Hernandez says, was not simply that one model behaved differently after a model-family change. The prompt allowed different readings: it treated schema and recorded lineage as evidence, and warned against inventing facts, but did not clearly say when interpreting that evidence was allowed.
One model had applied that distinction as common sense. Another read the instructions literally and wanted corroboration beyond the schema to conclude that a column named cust_first_name held a first name. The model rotation exposed an ambiguity already present in the prompt, rather than creating it.
#1 Best Overall
How did rejecting everything still look like success?
The original adversarial check had four cases: one proposal that should be approved and three that should be rejected. A reviewer that rejected every proposal therefore got three cases right—3 of 4, or 75 percent—even though it failed the one approval case.
That aggregate score blurred two different errors. A false approval can put an unsupported claim into a catalog; a false rejection can leave a correct repair unapplied. When rejection cases dominate a small test, a reviewer can look cautious and score well while refusing useful, evidence-supported work.
What evidence should permit a repair?
Hernandez revised the Arbiter’s instructions to distinguish facts established by the data, interpretations grounded in those facts, and claims about the world that need additional support.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsRank #2
Facts established by schema or lineage
A schema establishes which columns exist and their types. Recorded lineage establishes the upstream and downstream relationships it lists. A proposed description that simply restates those facts can be approved without demanding outside corroboration.
Interpretations grounded in clear evidence
A clear column name can support a reasonable classification. Hernandez’s examples include treating cust_first_name as personal data, billing_zipcode as a postal code associated with a customer record, and order_id as an order identifier. The distinction is between drawing a defensible interpretation from available evidence and inventing an unrelated fact.
Not every plausible-looking name is equally clear. Hernandez cautions against automatically classifying an ambiguous name such as region or account_number on a warehouse table as PII. Those cases need to remain meaningful tests of judgment rather than being treated as obvious positives.
Rank #3
Claims that need separate evidence
A schema does not establish who owns a dataset, how often it refreshes, how many rows it contains, whether it is trustworthy, or whether an unrecorded downstream consumer exists. Such claims require supporting evidence; they should not be inferred from column names or lineage records that do not contain them.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →How Hernandez changed the evaluation
The revised harness expanded the test from four cases to nine: five expected approvals and four expected rejections. It included evidence-grounded descriptions and column restatements, along with clear PII labels for cust_first_name, billing_zipcode, and email. Rejection cases tested unsupported claims such as fabricated refresh cadence, row counts, ownership, data quality, or a downstream consumer, as well as the false-positive trap of tagging generic region as PII.
Instead of rolling every result into one accuracy score, the harness recorded three outcome classes:
Rank #4
- False approval: the reviewer accepts an unsupported proposal.
- False rejection: the reviewer rejects a proposal the evidence supports.
- Error or no judgment: the reviewer fails to provide a usable decision.
This reporting makes it harder for one type of mistake to hide another. A useful test should include both approval and rejection cases, clear examples and ambiguous ones, and a way to see each failure direction separately.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the revised harness showed—and what it did not
Hernandez reports that the revised harness scored 9/9 on two model families that had not seen the revised prompt: gpt-4o-mini through GitHub Models and gemma-4-26b through OpenRouter. In those runs, both models approved grounded PII repairs, rejected fabricated claims about the world, and avoided the generic-region false positive.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →That is a reported result from a nine-case harness, not an independently reproduced measurement. It also does not establish that a later live sweep restored PII governance: Hernandez’s post reports the harness outcome, not a post-change production result.
The engineering lesson
A safety mechanism is not safe merely because it is reluctant to approve. Refusing a supported repair can leave a catalog inaccurate and a sensitive column ungoverned; accepting an invented claim can introduce false information. Those risks need separate visibility in both the instructions and the evaluation.
Hernandez’s most useful debugging question captures the test-design failure: “how did rejecting everything still look like success?” His post-mortem’s answer is specific: the test had three expected rejections for every expected approval, so blanket rejection earned 75 percent. The response was to clarify what evidence permits interpretation and to test approvals, rejections, ambiguity, and no-judgment outcomes explicitly.
Because the incident and results are reported by the project’s author in a contest submission, they should be read as a specific engineering case, not a universal guarantee about model behavior or a general benchmark of safety reviewers.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Read Joshua Hernandez’s post on DEV Community
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
