In one preliminary benchmark, four language models turned a fact-free company brief into job ads where 35% to 50% of the extracted claims were labeled concrete. Because the brief supplied no employer-specific facts, those concrete details were unsupported commitments—not evidence of accuracy or writing quality. The results are a snapshot from one prompt, not a reliable model ranking.
What the benchmark asked models to do
In a post published October 5, 2026, Karlis Gutans described a Kaggle benchmark built around a junior backend engineer job ad. The prompt named the role and seniority and asked the model to address the service, programming languages and data stores, code review, team size, location, pay, and the first six months. It did not identify an employer or supply company-specific facts. Models could fill those gaps with specifics, use placeholders, or stay general. Gutans’s post on DEV Community describes the benchmark; its reported results are the author’s, not independently reproduced measurements.
As an Amazon Associate I earn from qualifying purchases.
The benchmark’s score is a share: concrete claims divided by all extracted claims, from 0 to 1. A fixed judge model, separate from the contestants, extracts exact text spans and classifies them as concrete, general direction, or empty slogan. Code rejects claims not found verbatim in the ad, requires a quoted particular to retain a concrete label, flags template placeholders, and excludes bare skill nouns and “English” as particulars. In a prompt with no employer facts, a concrete claim is therefore an unsupported commitment, even if it sounds plausible.
What the reported runs found
The post’s current leaderboard reports these single-run results. Cost and time are reported for the generated ad and both judge calls; the author says most tokens are used by the judge.
#1 Best Overall
| Model | Concrete-claim score | Reported run cost | Reported time |
|---|---|---|---|
| Claude Haiku 4.5 | 0.50 | $0.037 | 29 seconds |
| Gemini 3.7 Flash | 0.48 | $0.062 | 44 seconds |
| Gemini 2.5 Flash | 0.37 | $0.074 | 60 seconds |
| GPT-5.4 mini | 0.35 | $0.069 | 45 seconds |
| Grok 4.5 | No score | not stated by the post | not stated by the post |
Grok 4.5 returned an error and was excluded because the benchmark does not publish partial scores. The figures above are the author’s reported values, not a controlled comparison across multiple prompts or repeated trials.
What one ad looked like
For Gemini 3.7 Flash, the post gives a 2,528-character ad with 29 extracted claims: 14 concrete, 13 general-direction statements, two empty slogans, and no placeholders. The author says none of the requested topics was left blank. That is an example of how the scoring operates, not a broader estimate of what the model will produce on other briefs.
Rank #2
Why the scores are not a leaderboard verdict
Higher is not better here. In a brief that contains no employer facts, a higher concrete-claim share means the ad contains more specifics that the prompt cannot substantiate. A lower score can indicate more general wording or placeholders; it does not, by itself, prove that an ad is accurate, useful, or suitable for hiring.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Each model’s score comes from one brief and one generated ad, with roughly 30 claims. For Gemini 3.7 Flash, the post reports a 95% interval of about 0.31–0.66 around its 14-of-29 concrete share; every model’s score falls inside that interval. The apparent grouping of the four scores into two pairs is a lead for further testing, not an established result. The benchmark post’s uncertainty discussion cautions against treating the numbers as a ranking.
There is also an unresolved measurement question: the generated-ad judge has not been hand-checked on ads produced by these models. The author cites 86.7% agreement with hand labels for an earlier version of the rules on real job postings. That result used a different judge model and older prompts, so it does not establish the current judge’s reliability on generated ads. A scoring system can make its rules explicit without those rules being validated for this new use.
What the benchmark can—and cannot—tell employers
The benchmark draws attention to a real drafting risk: when an employer leaves facts out of a prompt, a fluent ad can still present specifics about pay, location, team structure, or early responsibilities. Those claims should be checked against approved employer information before publication. The score itself does not identify which claims are false; it identifies claims the fact-free brief did not support.
Rank #4
It also does not establish whether these ads attract qualified applicants, improve hiring quality, or affect bias. The post proposes testing all 20 briefs several times, hand-checking a sample of generated claims, breaking results out by topic—especially pay—and measuring applicant outcomes. That last question would require employer or job-board data that the reported leaderboard does not provide.
Recommended Free Tools
How this fits with earlier job-ad research
A 2022 peer-reviewed study examined bias and realism in zero-shot GPT-3 job advertisements. Borchers, Gala, Gilburt, Oravkin, Bounsi, Asano, and Kirk reported that diversity-encouraging prompt engineering produced no significant improvement in bias or realism, while fine-tuning—especially on unbiased real ads—could improve realism and reduce bias. That work is relevant context, but it tested different methods and questions; it is not a replication of Gutans’s specificity benchmark. Read the paper in the ACL Anthology.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
