A 77% average pass rate does not mean an AI agent will complete a task reliably every time. In one AppWorld experiment, a GPT-4.1-backed ReAct agent averaged 77% success across five attempts per task, but succeeded on all five attempts for only 53% of tasks. That 53% is a benchmark result—not a measured production success rate. The difference is a reminder to test repeatability, not just average performance.
What the 77% and 53% figures measure
The figures come from the September 8, 2026 arXiv preprint Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course by Evelyn Duesterwald, Benjamin Elder, Lilian Ngweta, Shashanka Ubaru, and Malgorzata Zimon. The authors evaluated ReAct agents on AppWorld’s 168-task test_normal split, running each task five times and using the benchmark’s standard grader.
As an Amazon Associate I earn from qualifying purchases.
- Mean@5: 77%. The average fraction of successful attempts across the five runs per task.
- Pass5: 53%. The fraction of tasks that succeeded on every one of the five runs.
- Consistency gap: 24.4 percentage points. Mean@5 minus Pass5 in this GPT-4.1 setup.
A task that passes three runs and fails two adds successes to the average, but it does not count as reliably completed under the all-five measure. That is how a strong-looking average can coexist with a much lower rate of repeatable task success.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhy an average can hide instability
An average aggregates outcomes. It does not show how successes and failures are distributed across individual tasks. Two agents can have similar average pass rates while differing in whether they consistently solve a subset of tasks or alternate between success and failure on many tasks.
#1 Best Overall
For a user, the distinction matters because a production interaction is usually one attempt. Mean@5 describes average performance across repeated trials; Pass5 asks how often the same task remained successful across all five. The 53% figure is not the chance that any one production attempt will succeed, and it does not assume that the five outcomes are independent.
How to read repeated-run metrics
The preprint distinguishes three metrics that answer different questions:
- Pass@k: the fraction of tasks that succeed at least once in k attempts.
- Mean@k: the average fraction of successful attempts over those attempts.
- Passk: the fraction of tasks that succeed on every attempt.
The consistency gap is Mean@k minus Passk, expressed in percentage points. The paper also defines normalized consistency as Passk divided by Mean@k. That ratio helps distinguish stability from raw capability: when average success is low, the absolute gap has less room to grow.
What the experiment found beyond the headline
Results differed by model
The study also evaluated GPT-OSS-120B. Its baseline was 34% Mean@5 and 10% Pass5, with a reported consistency gap of 23.8 percentage points. The similar gap size does not mean the models had similar task performance: GPT-OSS-120B’s average success was much lower.
Task difficulty changed the pattern
For GPT-4.1, the reported absolute gap increased from 17.5 percentage points on easy tasks to 30.2 points on hard tasks. GPT-OSS-120B behaved differently: its hard-task mean pass rate was only 9.5%, which limits the possible absolute gap, and its normalized consistency on hard tasks was zero in this evaluation. The results therefore do not support a blanket rule that harder tasks always produce the largest consistency gap.
Consistency guidelines improved repeated success in this setup
The authors’ method analyzes variability at agent decision steps, generates targeted natural-language consistency guidelines, stores them as episodic memory, and retrieves them for similar tasks. The analysis and guideline generation are described as offline stages.
For GPT-4.1 on the same tasks, Pass5 rose from 53.0% to 69.0%, an increase of 16 percentage points; Mean@5 rose by 3.6 points rather than declining. On similar-task generalization, Pass5 increased by 13 points. GPT-OSS-120B gained 6 points in same-task Pass5. These are findings from the AppWorld experiments, not guaranteed gains for deployed systems.
Recommended Free Tools
What this means—and does not mean—for production
The paper measures benchmark performance, not a 53% production deployment rate. Its result covers one benchmark split, a ReAct agent pattern, five attempts per task, the benchmark’s grader, and two evaluated model backends. The authors mention informal observations of similar patterns with other architectures, but do not quantify those settings.
The practical warning is narrower: a one-run or average benchmark score can conceal whether an agent repeats its success on identical cases. The study attributes potential run-to-run flips to uncertain decisions during execution and discusses variability even under temperature-zero decoding. It does not establish that temperature alone causes inconsistency, or that setting temperature to zero guarantees deterministic results.
How to test an agent for repeatability
A useful local evaluation can adapt the paper’s metrics without treating its setup as a universal protocol:
- Fix the task set and success rule. Keep the cases and grading criteria stable so comparisons are meaningful.
- Run each task repeatedly. Record each outcome, not only the aggregate score. State the number of attempts per task.
- Report both average and all-runs performance. Mean@k shows average attempt success; Passk shows the share of tasks that passed every attempt.
- Include evaluation context. Identify the model, agent architecture, benchmark or task set, grader, repeat count, and task-difficulty mix.
- Separate baseline from interventions. Label results on the same tasks separately from results on similar or generalized tasks.
These details make a score interpretable: a pass rate without its repeat count and success definition cannot tell a reader whether it represents occasional success, average success, or consistent completion.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
