Before you publish a coding-agent leaderboard, freeze three things and report them alongside the ranking: the task pack, identified by a hashed manifest; the scoring contract, meaning the metric definitions and their version; and the negative controls that show the pack and grader can detect bad work. Without those, a score cannot be checked against what it was measured on.
This framing comes from Avery Wang’s DEV Community article “Freeze the Manifest Before the Agent Leaderboard,” listed with a September 17 date; the listing does not show the year. Wang presents the process as a small experiment with pinned inputs. The code and workflow examples in the article are proposals that the author describes as unexecuted, so treat the method as a protocol to adopt and test, not as a validated tool.
As an Amazon Associate I earn from qualifying purchases.
What to freeze
1. The task pack
Hash every file in the benchmark and record the result in a manifest. Any change to a file should produce a different pack identity, so a score always points to the exact tasks that produced it. For programming tasks, Wang’s setting uses hidden tests and a declared per-item time budget; both belong in the manifest.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
If you discover that a pack is flawed, publish a new pack identifier rather than editing the old one. Silent edits are the failure this step exists to prevent: a leaderboard built on a pack that changed underneath it no longer compares the same work across candidates.
#1 Best Overall
2. The scoring contract
Define each outcome as an artifact or an observable event, such as a patch that compiles, a test run that passes, or a run that hits its time limit. Report the components separately: compile, test, lint, timeout, and assertion deletion. An agent that passes by removing the assertions it was supposed to satisfy should appear as a distinct outcome, not disappear into a total.
If you publish one headline number, Wang’s example uses a strict conjunction: a candidate counts only when every component passes. A conjunction is easy to audit, but it is also harsh, so the component breakdown should stay visible next to it. Record the metric version with the pack identifier, because a changed scoring rule produces a different leaderboard even when the tasks are identical.
Rank #2
3. Negative controls and baselines
Run known-bad inputs through the same harness used for candidates. Wang’s sample battery includes three: an empty patch, a run with shuffled tests, and an echoed prompt. None of them should pass. The proposed publish gate refuses to release a leaderboard if any control passes unexpectedly, because that points to a problem in the task pack or grader that must be understood before ranking anyone.
Treat this battery as a starting point. The article offers it as an example and does not report running it, and it does not claim the three controls catch every failure mode.
Before candidates run
- Confirm that candidate-authored patches run inside an execution sandbox. Wang states that a sandbox is required before running model-written code.
- Keep hidden tests out of the candidates’ reach, and confirm the per-item time budget is declared in the manifest before the first run.
- Run the negative controls first. If they pass, fix the pack or grader and assign a new pack identifier before going further.
What a trustworthy ranking carries
A ranking should travel with the evidence that makes it checkable. Wang’s proposed report includes the following, and a reader can apply the same test to any published agent score:
- The pack identifier, so you can confirm every candidate ran the same tasks.
- The metric version and the definition of each component outcome.
- The control results, showing that each control failed as expected.
- Per-candidate component outcomes, not only a single blended figure.
- A statement of whether the task pack matches the claim being made.
Wang’s position is that a number without its manifest, metric, and controls is not an adequately documented measurement.
What freezing does not guarantee
A frozen pack preserves identity. It does not show that the tasks represent the work you care about, that the tests measure quality, or that the metric is fair. The article is explicit about these limits:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems- It does not measure taste, architecture, or long-horizon refactors.
- Hidden unit tests are a poor oracle for interface work, migrations, and incident response when the fixtures do not encode the real loss. A passing test in those domains may reflect a weak fixture rather than a good patch.
- It warns against reducing the method to a single promotional percentage.
Private pack or public benchmark
Teams have two broad routes. They can build a protocol around a private task pack, or they can rely on an existing public, versioned benchmark that already has hidden tests and documented controls. Wang notes that researchers who already have the second kind of foundation may not need to rebuild the first.
Best Value
Compare the routes on the same axes:
- Task relevance: do the tasks resemble the work behind your claim?
- Benchmark versioning: is there a stable identifier for each release?
- Grader transparency and leakage risk: can you see how outcomes are graded, and could candidates have seen the tests?
- Control coverage: are negative controls documented, and do they fail as expected?
- Quality coverage: does the benchmark capture the quality dimensions your claim depends on?
The article does not compare named commercial benchmarks and reports no head-to-head results, so it cannot tell you which route scores better.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Lessons from other benchmark projects
Two projects outside coding agents show similar choices, though neither validates Wang’s controls or code.
ARC-AGI-3 project plan
A mutable GitHub project plan from the dcw06/ARC-AGI-3 repository (“ARC-AGI-3 Project Plan 7”) makes its protocol explicit. It hashes and archives a fixed evaluation manifest, uses the game as the unit of generalization, aggregates repeated seeds within each game, and predeclares how crashes, timeouts, and missing results are treated. An earlier version of the same plan states that the fixed evaluation set and the aggregation rules matter to the result, and that games untouched in the relevant leaderboard split count as zero. That rule belongs to that project and should not be generalized to other benchmarks.
Autonomous-vehicle policy lab
A decision log in the parvpatodia/av-policy-lab repository (DECISIONS.md, frontier-upgrade branch) shows a freeze delayed after validity concerns, then later frozen scenario sets recorded with hashes and a disjoint selection probe. The example shows that freezing can be conditional: the team resolved design problems before locking inputs. It is unrelated to coding agents and is not a general standard.
Quick Recap
What is and is not established
- Established: the article recommends pinning the pack, specifying component outcomes, running negative controls with a publish gate, and publishing the evidence with the ranking.
- Not established: that the proposed harness has been run. The article describes its scripts as proposed and unexecuted and reports no results from them.
- Not established: that the numeric values in the article’s scripts and gate examples reflect measured performance. They are illustrative and should not be quoted as benchmark results.
- Not established: that the control battery catches every failure, or that the method generalizes to all kinds of agent work.
- Source context: the article discloses that it was prepared as part of product outreach for MonkeyCode, so read its framing with that in mind.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
