In Elio Liberatore’s 2026 benchmark, four LLMs’ generated simulation code matched the author’s Monte Carlo playoff estimates slightly more closely than the models’ direct probability guesses did. That is evidence of agreement with one model across 18 MLB and NFL cases—not proof that the LLMs’ playoff probabilities were calibrated against real postseason outcomes.
What the benchmark compared
Liberatore’s DEV Community post describes a benchmark with 18 cases: five MLB and 13 NFL. It asks whether models can estimate the chance that a team makes the playoffs, and compares two ways of answering. The target in both tasks is the author’s own Monte Carlo probability for that team. Read the benchmark post by Elio Liberatore.
As an Amazon Associate I earn from qualifying purchases.
Task A: Give a probability directly
Each model receives a team’s record, remaining games, season point or run differential, and a short narrative. It must return one playoff-probability number. The benchmark scores that number against the author’s Monte Carlo estimate.
Free tools Windows power users keep installed
One-click scans. No signup required.
Task B: Write and run a simulation
Each model instead writes Python code to simulate the team’s remaining games. The code is executed, and the resulting probability is compared with the same target used for Task A. This tests whether the model can produce executable simulation code and achieve target agreement; code execution is not itself evidence of forecast calibration.
The author says the business behind the reference model runs 10,000–20,000 simulations per team for MLB and NFL playoff odds and cross-checks prices against Kalshi. Those details describe the stated reference-model context; the post does not establish the model’s full methodology or show that its outputs have been independently validated.
What scores did the models report?
The post reports mean scores across all 18 cases, on a 0–100% scale where higher is better. These are author-reported aggregate figures; the post directs readers to the live Kaggle benchmark for per-case results.
Rank #2
| Model | Direct estimate (Task A) | Executed simulation code (Task B) |
|---|---|---|
| GPT-5.4 mini | 98.0% | 99.6% |
| Gemini 3.7 Flash | 97.3% | 99.7% |
| Gemini 3.8 Flash | 97.1% | 99.7% |
| Claude Haiku 4.5 | 95.4% | 99.6% |
For each model shown, the code-generation score is higher than its direct-estimate score, and the four code scores are tightly grouped. The author interprets this pattern as suggesting that, in this benchmark, the models were more reliable at translating a simulation request into code than reasoning directly to a number. It is a result from this small comparison, not a general finding about all LLMs or all forecasting tasks.
The post also names Claude Opus 4.8, GPT-5.5, and Qwen 3 Next 80B Instruct as unable to complete either task because Kaggle returned a 403 PermissionDeniedError before billing. The author attributes those failures to a platform limitation, not to model performance.
Why high agreement is not the same as calibration
Calibration asks whether events assigned a given probability occur at about that frequency across a suitable set of resolved forecasts. For example, among many independent events forecast at 70%, roughly 70% should happen if those forecasts are well calibrated. A single team estimate cannot establish that relationship.
This benchmark instead asks how closely the LLM outputs match a model-generated target. A score near 100% on that comparison means close agreement with the reference probabilities under the benchmark’s scoring setup. By itself, it does not show that teams forecast at 70% actually qualified about 70% of the time, that the reference model is calibrated, or that the result transfers to other seasons, leagues, or forecasting conditions. The post does not provide its exact scoring formula, case-level dataset, confidence intervals, or an independent replication, so the aggregate figures should not be treated as independently audited.
Rank #4
Forecast evaluation also distinguishes calibration from comparative skill. Yeh, Rice, and Dubin’s study of continuously updated NBA game forecasts used calibration surfaces and Brier-score comparisons; it found the forecasts reasonably calibrated and more skillful than some naive models, but did not show significant superiority over simple logistic-regression models based on relative team strength and evolving score difference. That work concerns live NBA game forecasts, not playoff qualification or this benchmark. Read the NBA forecasting study.
More broadly, Turtel and colleagues report that different proper-scoring-rule training objectives produced distinct calibration and error profiles in real-world binary forecasting. Their study is not about playoff odds, and each training condition used a single seed, so some differences may reflect training stochasticity. It is context for why forecasting behavior depends on how models are trained, not direct evidence about the teams or models in Liberatore’s benchmark. Read the forecasting study.
Best Value
What a Monte Carlo playoff estimate represents
A Monte Carlo playoff estimate samples possible outcomes for remaining games, applies the league’s qualification and tiebreak rules, and counts how often the team reaches the postseason. It is a model-based frequency: its meaning depends on the game probabilities, rules, inputs, and assumptions used in the simulations.
One public methodology explains its own process as rating teams from season performance, converting ratings into game probabilities, applying home advantage, simulating the schedule 100,000 times, and reporting the resulting fraction. It also says injuries, trades, suspensions, and roster changes are not directly incorporated, and that its data providers do not validate its forecast model. These are limitations of that publisher’s system, not established details of Liberatore’s reference model. See the publisher’s playoff-odds methodology.
What would establish real playoff calibration?
A stronger test of actual calibration would freeze forecasts at a defined point in time, retain the information available then, and compare them with resolved playoff-qualification outcomes across enough cases. The evaluation should make these elements explicit:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minute- Event: Define whether the forecast concerns making the playoffs, winning a game, or winning a championship.
- Timing and information: Record when each forecast was made and what information the forecaster could use.
- Cases and outcomes: Include the full set of forecasts and their eventual results, not just aggregate agreement with another model.
- Scoring: State the metric and how it treats probabilities; calibration analysis and measures such as Brier score answer related but distinct questions.
- Uncertainty: Report uncertainty around score differences so readers can judge whether a small gap is meaningful.
To answer whether an LLM “actually reasons about it” or echoes a number encountered in sports media, agreement with a Monte Carlo target is not enough. The benchmark compares outputs with that target, but does not isolate where the model’s answer came from. Testing the broader question would require a design that evaluates forecasts against outcomes and controls what information is available.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
