The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →The benchmark stopped at N=22 because its Python agent tried to turn a very large integer into a string, triggering CPython’s integer-to-string digit limit. Removing that unnecessary conversion restored the N=24 result. The investigation then uncovered eight more issues in parsing, precision, run state, and chart interpretation. The figures below are the author’s account of what happened, not independently reproduced measurements.
Why did the benchmark stop at N=22?
The cutoff was not an intentional limit on the benchmark. In the author’s July 15, 2026 account, the Python implementation attempted to convert a large Mersenne prime to a string. CPython’s configured limit for integer-to-string conversion was 4,300 digits; the 24th Mersenne prime, 219937−1, has 6,002 digits. The conversion failed before the benchmark could record N=24.
As an Amazon Associate I earn from qualifying purchases.
The tool returned elapsed time, not the prime’s decimal representation, so converting the integer to a string was unnecessary. After removing that conversion, the author reports a Python N=24 result of 2,425.9 ms. The earlier N=22 boundary had concealed the failure rather than explained it. Read the author’s account.
What were the nine bugs?
The first failure made the cutoff visible; the other problems affected whether measurements were extracted, complete, comparable, or chartable.
#1 Best Overall
- Unnecessary integer stringification. Python converted the prime despite needing only elapsed time. Go also had an unnecessary string conversion inside its timed region, adding work to the measurement.
- Timing parsed from prose. The harness expected a particular phrase in Gemini’s natural-language response. The author says Python results survived only because a structured tool artifact served as a fallback.
- Incorrect prime counts. Node.js and Rust could report finding 100 primes even though their exponent tables contained 26. A plausible-looking count was not evidence that the measured workload matched the intended inputs.
- Precision discarded before plotting. Rust timings formatted to two decimal places became 0.00 ms for very fast runs. Zero cannot be represented usefully on a logarithmic chart, so the measurements disappeared or became unplotable.
- A missing duration unit. After formatting work was removed, Go emitted nanoseconds for small inputs. The parser recognized microseconds, milliseconds, and seconds, but not nanoseconds.
- Old session state reused during reruns. Deterministic context IDs allowed ADK to retain conversation history. In one rerun, Gemini replied, “I already did that. Do you want to do it again?” The author says assigning unique IDs to runs restored missing datapoints.
- Direct and routed timings treated as one comparison. The chart put direct HTTP handlers beside Gemini-routed agents, then presented the result as a language comparison. These paths include different overhead and are not a controlled comparison of languages alone.
- Benchmark scope narrowed instead of diagnosed. Reducing the sweep to N=22 hid the crash. The author’s lesson: “The workaround you commit is the bug you keep.”
- Completeness and chart assumptions went unchecked. Missing measurements, zero-rounded values, mismatched counts, and parser blind spots were allowed to shape what the chart showed. Treating the harness and chart as software to validate—not as neutral reporting layers—was part of fixing the result.
How should you interpret the reported numbers?
The author describes a comparison of four agents written in Python, Go, Node.js, and Rust, with a sweep from N=1 through N=24. Python and Go used Gemini tool calling through ADK; Node.js and Rust used direct HTTP handlers. Each implementation calculated Mersenne primes with Lucas–Lehmer.
| Reported figure | What it describes | How to read it |
|---|---|---|
| 4,300 digits | Integer-to-string digit limit described by the author for CPython | Author-reported configuration in the 2026 account; not independently verified here. |
| 6,002 digits | Author’s stated digit count for 219937−1 | Explains why string conversion at N=24 encountered the reported limit. |
| 2,425.9 ms | Python result at N=24 after the unused conversion was removed | Author-reported result, not an independently reproduced timing. |
| 2.6 ms and 4.6 ms | Reported median round-trip times for the two direct-handler agents | The author’s account does not identify which number maps to which language. |
| About 1.6 s and 1.8 s | Reported median round-trip times for the two Gemini-routed agents | Author-reported approximate medians; these include a different execution path. |
| 96/96 datapoints | Final sweep completeness after the reported fixes | Author-reported outcome, not an independently inspected result archive. |
The round-trip medians are not a language-only result: direct HTTP handling and Gemini/ADK-mediated tool calls do different work. They also describe end-to-end latency, which should not be conflated with the time spent on the prime calculation itself. A fair comparison needs labels that distinguish execution path, measured quantity, and input size.
How can you prevent the same class of benchmark failure?
- Trace the failure before accepting a cutoff. When a sweep ends at a surprising boundary, inspect the error and stack trace rather than shrinking the test range and treating the new limit as a design choice.
- Keep machine-readable results structured. Have the program or tool return timing and status in structured fields. Do not make a parser depend on wording in generated prose when a structured artifact is available.
- Preserve numeric precision. Store raw durations at sufficient precision; round only for display. Avoid converting a valid small positive duration into zero before plotting.
- Parse units explicitly. Normalize supported units—such as ns, µs, ms, and s—to one base unit, and reject or flag unknown units rather than silently dropping values.
- Validate workload counts. Check that the number of inputs actually processed agrees with the benchmark plan and reported count. A run that says “100” while its table has 26 entries should fail validation.
- Isolate stateful runs. Give each independent agent run a unique context identifier when the framework can retain conversation history. Otherwise, an apparent rerun may be a continuation of an earlier session.
- Separate unlike execution paths. Report direct-handler latency separately from model-routed round trips, and identify whether each figure measures calculation time or total request time.
- Check completeness before charting. Verify expected versus observed datapoints, missing values, unit conversions, and log-scale constraints. A chart cannot repair a broken harness; it can make the harness’s omissions look authoritative.
What does the final 96/96 result establish?
The author says the fixes produced all 96 expected datapoints. That supports the account’s claim that the sweep ran to completion after the harness problems were addressed. It does not, by itself, make the cross-language chart a controlled language comparison: the reported agents still used different execution architectures, and the figures were not independently reproduced for this article.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Rank #3
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
