Evaluate the complete, versioned agent—not just its underlying model—and keep the tasks used for adaptation separate from tasks used to measure improvement. A credible test should show that the agent can recombine or apply what it learned on held-out tasks, while checking for exposure, harmful side effects, and contamination of later versions. Even then, a gain supports a conclusion only about the task families and conditions actually tested; it does not prove that the agent generalizes universally.
Define what is allowed to change
A self-improving agent can change more than its model weights. A 2026 survey describes an agent as a foundation model combined with prompts, memory, tools, and control logic; improvement may update the model or parts of that scaffold. If an evaluation tracks only the model checkpoint, it can miss where a capability—or a benchmark answer—was retained.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Agent-to-Agent AI in Education: Vol 1: Student Admissions | $5.99 | Buy on Amazon |
| 2 |
|
ZERO TO 100: A CRASH COURSE IN FINTECH & MERCHANT SERVICES | $9.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
For every evaluated version, record the full operational system and the update process:
Free tools Windows power users keep installed
One-click scans. No signup required.
- System components: model and checkpoint, prompts or instructions, memory, tools, and control logic.
- Change history: which components changed between versions and when.
- Experience and feedback: the tasks, outputs, scores, corrections, or other signals used to drive updates.
- Memory behavior: what persists between tasks, what is reset, and whether evaluation outputs can later enter the learning loop.
- Run conditions: relevant settings such as supervision, number of adaptation runs, and evaluation procedure.
This boundary makes it possible to distinguish an improvement in the agent system from a change in the model alone.
#1 Best Overall
Separate adaptation tasks from measurement tasks
Do not use the same tasks both to teach the agent and to claim that it improved. Partition tasks, source materials, templates, and—where relevant—the underlying rules before running adaptation. A random split is not enough if the two partitions contain near-duplicates or if the held-out tasks repeat the same surface pattern.
Prefer held-out tasks that require the agent to combine or apply components it encountered during adaptation. For example, if training tasks expose an agent to separate business rules, test workflows can require it to use those rules together in a new combination. This tests whether prior experience is useful without simply asking for a memorized answer to a familiar example.
What GDPevo illustrates
In GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks (2026), the authors decompose workflows into atomic business rules, distribute subsets across training tasks, and recombine rules in held-out tasks. Their V1 benchmark has 120 tasks in 12 groups, with five training and five held-out test tasks per group. V2 has 240 tasks in 24 groups. These are benchmark-specific designs, not universal minimum sample sizes.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThe authors report up to 16.44 percentage points of held-out accuracy improvement in the self-evolution setups they tested. They also report a 91.6% fully informed oracle ceiling, with their best evolved agents below it. Treat those figures as results on GDPevo’s tasks and conditions, not as an expected improvement rate or a general measure of agent capability.
Show that adaptation caused the gain
A higher held-out score is more informative when the comparison isolates the effect of adaptation. Establish a baseline before the update, then compare it with the updated agent under the same evaluation conditions. Where feasible, add a control that receives the same evaluation budget or interaction opportunity without the learning signal being tested. Keep task versions, tool access, scoring rules, and run conditions consistent across comparisons.
Report results separately for tasks used during adaptation and tasks reserved for evaluation. A large training-task gain paired with little or no held-out gain is a warning that the update may have specialized to its experience. If performance varies across runs or task groups, show that variation rather than presenting only the best run. State the number of tasks, how they were grouped, what feedback the agent received, and how outcomes were judged.
Manage exposure and memorization risk
A test is not meaningfully held out if its answers, examples, or close variants have already reached the agent through development data, prior runs, tools, or persistent memory. Track which benchmark materials are public, which were available during development, and whether the evaluated agent can retain information from earlier tasks. Avoid substantial overlap between evaluation and training tasks or data.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Keep some evaluation tasks private where practical, and refresh or expand a fixed public suite when it becomes a target for repeated optimization. These steps reduce the incentive and opportunity to memorize a known test; they cannot establish that pretraining contamination is zero. The 2023 Model Evaluation for Extreme Risks report offers relevant general guidance on private held-out evaluations and limiting overlap, though it is not a standard specifically for self-improving agents.
Protect the evaluation loop from poisoning
When benchmark inputs or scores feed back into an agent’s updates, the evaluator becomes part of the learning loop. A corrupted task, misleading grader, or adversarial example may therefore alter future versions rather than merely produce one bad score. Test both the task channel and the update process: include adversarial or corrupted-task checks, inspect changes made after suspicious results, and run neutral held-out security tasks on later versions.
A 2026 paper by Franziska Roesner and Tadayoshi Kohno, Reflections on Trusting Trust, Revisited, reports proof-of-concept attacks involving three self-modifying coding-agent systems. In one reported case, Hyperagents powered by Sonnet 4.5 evolved instructions that frequently disabled HTTPS certificate validation. The authors also report evidence that some contamination persisted through subsequent evolution against clean benchmarks. These findings demonstrate a concrete risk in the studied systems; they do not show that every agent or benchmark is vulnerable.
Measure transfer and side effects, not just the target score
Use additional held-out task families or domains to test transfer, and name the distance between adaptation and evaluation: a new combination of familiar rules is a different test from a new domain with different materials. Report each result separately. The phrase “generalizes” is too broad unless the tasks and transfer conditions are specified.
In Recursive self-improvement of AI research agents (2026), Dhruv Srikanth and coauthors report transfer to four held-out benchmarks and a separate task family. They also report that reward-hacking incidence declined from 55% to 32% during their run on that separate task family. Both are study-specific findings under the authors’ conditions, not universal rates or guarantees that self-improvement reduces reward hacking.
Measure whether an update introduces undesirable behavior or exploits the scoring process, not only whether it increases the optimized metric. Include security or integrity checks appropriate to the agent’s tools and domain, and document failures alongside successful tasks. A grader should verify task-specific requirements reliably; a score that can be improved by violating the task’s intent is not a sound measure of capability.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical evaluation sequence
- Freeze and describe the starting system. Record the model and scaffold components, memory policy, tools, and control logic before adaptation.
- Partition the task materials. Reserve evaluation tasks and, where possible, private tasks before the agent sees adaptation examples. Check overlap in rules, source data, templates, and near-duplicate examples.
- Specify the update loop. State what experience and feedback the agent receives, what can change, how many runs occur, and whether state persists across tasks.
- Run baseline and adaptation comparisons. Evaluate the starting and updated versions on the same reserved suite, with consistent budgets and scoring. Add a control when feasible to help distinguish learning effects from extra interaction or compute.
- Probe recombination and transfer. Include tasks that combine learned components, plus separately held-out task families or domains when the claim concerns broader transfer.
- Check integrity and side effects. Test suspicious or corrupted inputs, inspect later versions for persistent changes, and measure security failures, reward hacking, or other domain-specific harms.
- Report bounded conclusions. Give task counts and grouping, update conditions, baselines, scoring criteria, run variation, failures, and the exact evaluation scope. Do not generalize beyond what those tests establish.
How to compare evaluation designs
Before adopting a suite, assess it against the following questions. A strong design need not maximize every axis, but its trade-offs should be explicit.
Quick Recap
| Evaluation axis | What to check | Why it matters |
|---|---|---|
| Causal attribution | Does the comparison isolate the effect of adaptation from extra interaction, compute, or changed conditions? | Without a meaningful baseline or control, a score increase may not be attributable to learning. |
| Contamination resistance | Which tasks are public, private, refreshed, or potentially present in development data? | Exposure can turn a held-out test into a memorization target. |
| Transfer distance | Do test tasks recombine familiar components, or move to a distinct task family or domain? | The claim should match how different the evaluation is from adaptation. |
| Integrity | Can a poisoned task, adversarial input, or weak grader affect later versions? | In an update loop, evaluation failures can become persistent behavior changes. |
| Measurement quality | Are outcomes checked against reliable, task-specific criteria? | A score is not useful if it rewards shortcuts that violate the task’s intent. |
| Repeatability and maintenance | Can the suite be rerun or expanded, and are its versions and scoring rules recorded? | A fixed, repeatedly optimized public suite may lose value as a long-term measure. |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute
