Free tools Windows power users keep installed
One-click scans. No signup required.
A fraud score can look impressively predictive while measuring the bank’s alerting process rather than fraud itself. In a 2026 project, Syed Darain Qamar reports that the bank’s score had a ROC-AUC of just 0.053 across 5,565 closed investigations—but simply reversing it was not a sound solution. The project’s more useful lesson is how to evaluate selected cases, assemble auditable evidence, and let an investigator express uncertainty.
Why did the bank’s fraud score rank cases backwards?
Qamar’s project began with six months of card transactions, 5,565 closed investigations, a fraud policy, and twenty alerts. The transaction data had no fraud labels, according to the author. That matters: the team was not evaluating every transaction against a known truth. It was working with cases that had already passed through an investigation process.
As an Amazon Associate I earn from qualifying purchases.
Across those 5,565 closed cases, the bank’s detection score had a reported ROC-AUC of 0.053. ROC-AUC measures how well a score ranks positive cases above negative ones; a value near 0.5 indicates little ranking separation, while a value below 0.5 means the observed ranking is often reversed. But this result does not establish that low scores generally mean fraud.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchThe score helped determine which transactions became alerts. Qamar says high-score alerts often proved to be legitimate purchases, while confirmed fraud could arrive through customer reports and have low scores. The investigated cases were therefore selected by a workflow partly shaped by the score itself. The metric describes that selected set, not an unbiased sample of all card transactions.
#1 Best Overall
In the project’s balanced October holdout, simply inverting the bank score reportedly reached 93% accuracy. Qamar rejects that as a useful fraud model: it fit the construction of that benchmark, whose trigger-score range created a different sampling frame. A spectacular metric on selected cases can be a warning to inspect how labels and samples were produced, not a reason to deploy the opposite rule.
What was left after the score was set aside?
The team instead modeled behavior using eleven named findings intended to be checked individually by an analyst. The reported approach fitted log-odds weights on investigations opened before October 2016, then evaluated them on 278 October alerts of the same kind. Qamar reports ROC-AUC 0.849, accuracy 0.791, and Brier score 0.157 for that holdout. Those results apply to the stated alert sample; they do not establish performance on all transactions, another bank, or a later period.
The findings emphasized changes relative to a cardholder’s own patterns and connected activity, rather than treating one conspicuous purchase as decisive. Among high-score alerts, the author reports a 93.4% fraud rate when the device was already known to the account, versus 12.3% when the device was marked new. A purchase far above a customer’s median, with nothing else changed, had a reported fraud rate of 23%. Qamar says velocity relative to each card’s own rhythm and concurrent activity in the cardholder’s home region were more useful signals.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →These figures are conditional on the project’s high-score alert sample. They should not be read as general probabilities for familiar devices, new devices, or unusually large purchases across the wider card population. The practical point is that context and combinations of evidence can matter more than a single intuitive red flag.
How did TigerGraph support an auditable investigation?
TigerGraph served as evidence storage and case memory, not as proof that a transaction was fraudulent. The project represented customers, cards, transactions, device profiles, billing regions, email domains, and closed cases as connected entities. That structure let an investigator follow relationships among a transaction, an account, a device, and relevant prior cases.
Time boundaries were essential. Queries were cutoff-bounded so a case could not draw on information recorded later—including cases that closed after the investigation date. The agent re-derived claims with GSQL and compared aggregate results, sampled transaction fields, and the flagged transaction itself. The exporter blocked cases that failed parity checks, reducing the risk that graph data and the investigator’s account of it diverged.
Investigations were written back as queryable graph entities linked to findings, transactions, implicated cards and devices, and cited prior cases. The project’s GraphRAG corpus contained 503 documents: 37 policy chunks and 466 similar analyst notes. Because the near-identical notes could bury policy material in retrieval, the project ranked policy chunks and case narratives separately.
The design addresses two distinct questions: whether a claim is supported by the underlying records, and whether the claim is being interpreted under the applicable policy. Separating those checks makes the investigation easier to inspect, but does not by itself validate the underlying data or guarantee a correct decision.
How should an agent handle weak evidence?
The described policy called for verification before blocking when there was a single weak signal below 0.70. Rather than treating a score as an automatic verdict, the agent recorded its initial recommendation, requested evidence, simulated a cardholder response, documented that assumption, and revised its assessment while preserving both recommendations and the reason for the change.
Qamar describes two cases where asking the cardholder was not appropriate:
- Customer-reported transaction: A customer report already counts as a denial, so the agent does not ask that same cardholder to validate the reported purchase.
- Shared-origin cluster: When several customers are connected by a common origin, one cardholder’s answer cannot settle the broader pattern. The described response is reporting and monitoring connected cards.
This approach treats uncertainty as part of the record: what evidence was missing, what was requested, what assumption followed, and why the recommendation changed. It is more transparent than silently converting an uncertain score into a block decision.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What did the twenty benchmark cases show—and not show?
Across twenty benchmark cases, Qamar reports eleven fraud assessments, six legitimate assessments, and three uncertain ones. The agent made seven evidence requests, changed its recommendation four times, and produced two suspicious activity reports. These are outcomes on the supplied cases, not a broad evaluation of an independently sampled fraud population.
Rank #4
The earlier hand-tuned heuristic reportedly scored zero correct out of forty on the same holdout and abstained on 31 cases. That comparison is difficult to interpret without more detail on how the forty-case evaluation relates to the later 278-alert holdout and how its labels and coverage were defined. The author’s own account makes clear that coverage and abstention matter alongside accuracy: a system that declines to decide on most cases cannot be compared fairly with one that makes a recommendation on each case without reporting that difference.
Qamar notes that four of the twenty cases did not match the five documented typologies. This is a significant boundary: known patterns can support structured investigations, but a system that only recognizes supplied patterns may miss unfamiliar fraud. The project identifies discovery and investigation beyond the twenty supplied cases as unfinished work.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What would be needed before treating the model as dependable?
Qamar calls calibration the project’s honest gap. The weights were fitted on investigated alerts, not the general transaction population, so a displayed probability is tied to that sampling frame. A Brier score of 0.157 on the stated holdout is informative for that evaluation, but it does not establish reliable probabilities in a different population or later period.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsThe author proposes checking a reliability curve on a held-out period and grounding thresholds in a real cost model. In practice, a responsible evaluation should answer several separate questions:
Best Value
- Population fit: Does the test sample resemble the cases where the system will actually be used?
- Ranking: Does the model rank fraud cases above legitimate ones in that sample?
- Probability quality: Do cases assigned similar probabilities occur at those rates, as tested with calibration analysis?
- Coverage: How often does the system abstain or request more evidence?
- Decision costs: What are the consequences of false positives and false negatives, and how do those costs set thresholds?
- Auditability: Can an analyst inspect the evidence and reproduce the conclusion without later information leaking into the case?
The reported metrics do not answer all of these questions. The article provides no independent replication, broad population study, or evidence of deployment outcomes. Its results are best understood as a project report and a demonstration of evaluation risks, not proof of production readiness.
The larger lesson: measure the process, not just the score
A score can be wrong for the use a bank wants while still being deeply entangled with the process that generated the observed cases. Reversing it may exploit that selection artifact rather than reveal fraud. The stronger response is to examine who was investigated, how outcomes were recorded, what information was available at decision time, and whether the model is tested on a sample matching its intended use.
Qamar’s project makes that lesson concrete: pair behavioral signals with time-bounded graph evidence, keep policy distinct from precedent, expose the arithmetic behind an assessment, and record uncertainty and recommendation changes. Its reported results are promising within a narrow October alert sample, but the author’s stated gaps—calibration, cost-sensitive thresholds, undocumented patterns, and evaluation beyond twenty supplied cases—remain central to any claim that such an investigator is reliable.
As Qamar puts it in the DEV Community article, “An interface that renders beautifully and passes every schema check tells you nothing about whether the investigation is any good.”
Quick Recap
Read Syed Darain Qamar’s DEV Community article.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
