October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run ScanOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk4 min

The Benchmark That’s Half Traps, and Why That’s Brilliant

Finding a source-to-sink path is easy. Judging whether it is exploitable is hard, and only safe near-miss test cases can measure it.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A security scanner benchmark made only of vulnerable code measures the easy half of the job. Finding a path from user input to a dangerous function is a pattern-matching problem. Deciding whether that path can actually be exploited is the hard part, and you can only test it with code that looks dangerous but is safe. That is the argument in a September 24, 2026 DEV Community article by Ali Afana, an AI builder and security researcher, and it holds up well. His summary: “A benchmark that only rewards finding things measures the easy half.”

All counts and scanner scores below are Afana’s own reported figures. They have not been independently verified, so read them as one author’s run, not as rankings.

As an Amazon Associate I earn from qualifying purchases.

Why a labeled test set needs safe cases at all

Afana frames the reader’s obvious question as “Why does anyone need a labelled test set?” Without labels you cannot tell a scanner’s correct alerts from its noise. But labels alone are not enough. If every labeled case is a real vulnerability, the best strategy is to flag everything. That scores perfect recall and tells you nothing about whether the tool can reason.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The distinction at the center is between two questions:

#1 Best Overall
Sale
Azul Mosaic Tile-Placement Strategy Board Game, 2-4 Players, 30-45 Min
  • AWARD-WINNING STRATEGY BOARD GAME: Azul won the 2018 Spiel des Jahres. This draft and place game is inspired by Portuguese mosaic tiles to outscore rivals in this acclaimed game for adults and families
  • TILE PLACEMENT & MOSAIC ART: Select tiles from shared factory displays, complete pattern rows, and build your stunning mosaic wall. Every placement decision shapes your score and your board
  • BOARD GAMES FOR ADULTS & FAMILIES: Easy rules get you playing in minutes, yet deep tile placement strategy and draft & deny mechanics create satisfying complexity for experienced adult board gamers
  • PERFECT BOARD GAME FOR TWO ADULTS: Azul shines in head-to-head 2-player duels and scales brilliantly to 3 or 4 players. New tile combinations each round make every game a fresh, replayable challenge
  • GREAT GIFT ADULT BOARD GAME: Designed for 2-4 players ages 8 and up, 30-45 minutes playtime. Azul is consistently a top-rated mosaic board game worthy of any collection, family game night at home, road trips, vacations or gifts
  • Is there a source-to-sink path? Does untrusted input structurally reach a dangerous operation such as a SQL query, an HTML response, a file path or a shell command?
  • Is that path exploitable? After accounting for constants, sanitizers and unreachable branches, can an attacker actually control what reaches the sink in a harmful way?

The first is detection. The second, which the article calls discrimination, is where scanners earn trust or lose it, because every false alarm costs a human’s time.

What the benchmark contains

The article describes OWASP Benchmark as a generated Java application whose test cases are labeled as either vulnerable or safe. The safe cases are decoys: near-misses that keep the shape of vulnerable code but change one meaningful thing that makes it harmless.

Rank #2
Sale
Carcassonne Tile Placement Strategy Board Game, 2-5 Players, 35 Min
  • CLASSIC TILE PLACEMENT: Draw and place landscape tiles to build cities, roads, fields, and monasteries, then deploy meeples as knights, farmers, and monks to claim features and score points.
  • STRATEGY FOR ADULTS AND FAMILIES: Carcassonne pairs intuitive rules with meaningful decisions, making it accessible for ages 7+ while still engaging experienced adult board gamers.
  • REPLAYABLE MEDIEVAL ADVENTURE: Randomized tile draws create a different landscape every game, bringing fresh puzzles and competitive fun to family game night and casual group play.
  • TWO TO FIVE PLAYERS: Built for 2-5 players with an average 35-minute playtime, Carcassonne fits weeknight sessions at home, family gatherings on vacation, and adult board game evenings.
  • INCLUDES MINI-EXPANSIONS: The base game comes with The Abbot and The River mini-expansions in the box, adding variety to the classic Carcassonne board game experience from the start.

For the four categories the article discusses, the author reports the following. These are article-level figures for these four classes, not the total for every category in the benchmark.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Vulnerability class Real vulnerabilities Decoys
SQL injection 272 232
Cross-site scripting 246 209
Path traversal 133 135
Command injection 126 125
Total 777 701

That is 1,478 cases, with decoys making up roughly 47% (my arithmetic from the reported counts). The set is close to a coin flip between “real” and “trap,” and that balance is the point.

Rank #3
Sale
CATAN Board Game (6th Edition)
  • EXPLORE THE ISLAND OF CATAN: Settle the uninhabited island of Catan by gathering resources, building infrastructure, and nurturing trade relationships.
  • STRATEGY AND COMPETITION: Compete with 2-3 opponents to expand your settlements and cities while managing resources and avoiding the robber.
  • TRADE, BUILD, AND SETTLE: Use brick, wood, wheat, ore, and sheep to construct roads, settlements, and cities in your race to 10 victory points.
  • REPLAYABLE AND ENGAGING: With a modular hexagonal board, no two games are the same, offering endless strategic opportunities and replayability.
  • FOR FAMILIES AND STRATEGY ENTHUSIASTS: Designed for 3-4 players, ages 10 and up, CATAN 6th Edition is perfect for family game nights and friendly competition. Add the CATAN 5-6 Player Extension (sold separately) to expand your game to 5-6 players.

Three kinds of trap, shown by the article’s examples

The helper that returns a constant

In the article’s BenchmarkTest00052 case, request data appears to flow into a SQL statement. But the helper method it passes through returns the literal "bar" and ignores its argument. The user-controlled flow the code seems to have does not exist. A scanner that stops at “input went into a function, output went into SQL” raises a false alarm.

The encoder in the middle

In BenchmarkTest00282, an HTTP Referer header is passed through ESAPI.encoder().encodeForHTML before being written to output. The flow exists structurally, but the encoding neutralizes it for the HTML context. Seeing the path is not enough; the tool has to recognize that a sanitizer sits on it and that it is the right one for that sink.

Rank #4
Sale
Stonemaier Games: Wingspan by Elizabeth Hargrave
  • Bird Collecting: You are bird enthusiasts - researchers, bird watchers, ornithologists, and collectors - seeking to discover and attract a beautiful and diverse network of birds to your wildlife preserve.
  • Build Your Engine: Gain food tokens via custom dice in a birdfeeder dice tower, lay eggs using egg miniatures in a variety of colors, draw from over 170 unique bird cards and play them. Each bird extends a chain of powerful combinations in one of your habitats. Earn points, play new birds, help others, and many other abilities!
  • Award Winning: WIngspan is the 2019 winner of the prestigious Kennerspiel des Jahres award, along with many others!
  • High Replayability: WIth 170 unique bird cards, 26 bonus goal cards, and 8 goal tiles, Wingspan provides endless opportunities for bird combinations and goal achievements, making this a great gift for families, teenagers, students, couples, and anyone who loves birds and nature.
  • For families, solo gamers, and game groups alike: This medium weight, set collection and hand management strategy game for 1-5 players has a 60-90 minute playing time with only a 6 minute setup time. Great game for couples, solo gamers, 2 players, family, and friends. Includes swift-start pack for first time player guidance.

The branch that can never run

The third example uses the condition (7 * 18) + 106 > 200. That evaluates to 126 + 106 = 232, so it is always true, and the conditional always selects a constant instead of the tainted parameter. The author presents this as a limitation of his own scanner and its code-slicing setup, not as a general claim about every taint tracker. It illustrates why reasoning about conditions is a separate skill from tracing data.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

What the decoys exposed

The author reports that decoys revealed many false positives in his scanner, and in his comparisons with CodeQL and Semgrep. His reported false-positive rates on decoys:

Best Value
No Escape Board Game - Strategy Board Game for Adults, Family, Party - Unique Strategic Space Sabotage Traitor Maze Game with Tiles - Fun for Kids, Teenagers, Adults, 2 to 8 Players
  • Quick and Easy Setup: Get the fun started in minutes! No Escape Board Game is suitable for board game party nights with kids, teenagers, and adults. Easy setup ensures more time for an exciting space escape adventure
  • Dynamic Maze Runner Game: Every game feels unique! Experience a thrilling maze runner game with dynamic tile laying and action-packed sequences. Suitable for 2-8 players board games sessions that keeps everyone on their toes
  • Engaging Space Station Games: Dive into the depths of the space station with our board games for 2-8 players. The No Escape Board Game offers a captivating escape board game experience with strategic gameplay and endless fun
  • Party Board Game Night: Bring excitement to your next party board game night! With quick setup and easy-to-learn rules, this escape board game is suitable for kids' birthdays, teen hangouts, or adult gatherings
  • Action-Packed Maze Escape: Combine strategy with luck and navigate through the maze escape. A premium experience that includes high quality piece of dice, meeples, and tiles
Tool (as reported by the author) False-positive rate on decoys
Author’s deterministic layer, SQL injection 86%
Author’s deterministic layer, command injection 89%
Author’s deterministic layer, XSS 90%
Author’s deterministic layer, path traversal 84%
Author’s deterministic layer, summarized 88%
CodeQL 61%
Semgrep 65%

Treat these as results from one run with one configuration and one dataset. They are not a ranking of CodeQL against Semgrep, and they say nothing about current versions or other rule sets. What they do show is the benchmark’s value: on a positives-only set, a tool with an 88% false-positive rate on safe look-alikes could look excellent.

Why “flag everything” must lose

Take the article’s four-class set. A scanner that flags all 1,478 cases catches all 777 vulnerabilities, so recall is 100%. But only about 53% of its alerts are real (777 of 1,478, my arithmetic), and it wrongly flags every decoy. On a positives-only benchmark the same tool would score a perfect result. Enough negatives make precision visible and turn the recall-versus-precision tradeoff into something you can measure rather than assume.

Lessons for building or judging a benchmark

These are the author’s recommendations, not a formal standard.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Use near-miss negatives. Make each safe case differ from its vulnerable twin by one meaningful property. Obviously unrelated safe code is too easy to reject and proves little.
  2. Include enough negatives. There should be enough that indiscriminate flagging scores badly. The roughly even split in the article’s set does this.
  3. Organize decoys into failure families. Group them by why they are safe: constants, sanitizers, unreachable conditions. A cluster of failures then points to a missing capability, such as recognizing sanitizers or evaluating conditions, instead of a single unexplained score.

How to use this when evaluating a scanner

  • Ask for precision on safe look-alikes, not just the number of vulnerabilities found.
  • Check whether a benchmark’s safe cases are close variants of its vulnerable ones.
  • Look at results by failure family. An overall false-positive rate hides whether a tool is blind to encoders or to dead branches.
  • Remember that labels in a generated benchmark are the generator’s judgment, and a synthetic suite may not reflect your own codebase.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.