Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsChoose the model that solves your kind of task best under the conditions you will actually use—not the one with the highest score on an unrelated benchmark. Compare candidates on the same representative problems, with the same tools, prompt, attempt limit, time or token budget, and scoring rules. Treat cryptanalysis and puzzle solving as separate capabilities, and restrict cryptanalysis tests to authorized exercises, toy schemes, or systems you are permitted to assess.
Start by defining what you want the model to solve
“Puzzle solving” can mean a known-answer classical cipher, a mathematical problem, an abstract grid transformation, or an interactive environment. Cryptanalysis has its own range, from recognizing a known weakness in a toy scheme to investigating a full-strength cryptographic primitive. These tasks exercise different capabilities; success on one is not evidence of success on another.
As an Amazon Associate I earn from qualifying purchases.
Before comparing models, write down the task family and the difficulty you care about. If you need help with a specific kind of cipher puzzle, test that kind—not a broad reasoning benchmark and not a different cipher. If the aim is to assess a cryptographic design, establish that you are authorized to test it and define a controlled scope.
What cryptanalysis benchmarks can—and cannot—tell you
The July 20, 2026 preprint CryptanalysisBench: Can LLMs do Cryptanalysis? by Lukas Fluri, Avital Shafran, Nicholas Carlini, Matthew Jagielski, Milad Nasr, Orr Dunkelman, Eyal Ronen, and Florian Tramèr describes 191 tasks across six families of cryptographic primitives, drawn primarily from four NIST standardization competitions. Its three tiers distinguish schemes with known practical breaks, schemes with no known practical break tested at full strength and in scaled-down forms, and a challenge set of production primitives at the frontier of cryptanalysis.
In that paper’s benchmark and evaluation setup, the five evaluated frontier models—Claude Opus 4.8, Sonnet 5, Mythos 5, GPT 5.5, and open-weights GLM 5.2—broke 65%–86% of Tier 1 schemes. The authors also report that the models broke 6–12 Tier 2 schemes at full strength and 24–61 scaled-down variants. These are benchmark findings, not general success rates for cryptanalysis: the paper’s results do not establish how a model will perform on a different cipher, protocol, model release, or operational attack. The authors report examples of newly surfaced attacks, while noting that the harder tiers remain unsaturated.
For a model-selection decision, the tier matters as much as the headline score. A result on schemes with known practical breaks says something different from a result on full-strength schemes without known practical breaks or on frontier challenges. Do not collapse those categories into a single “can break encryption” claim.
Keep puzzle-solving results in their own category
ARC-AGI benchmarks are a narrow family of abstract reasoning tests, not a proxy for every kind of puzzle. The ARC-AGI-2 technical report describes abstract, puzzle-like reasoning intended to provide a more granular signal about problem-solving ability. ARC-AGI-3 is interactive; its technical report emphasizes novel environments, compositional generalization, out-of-distribution design, and human calibration. A score from one version or protocol should not be silently applied to another, or to puzzles outside that benchmark family.
Setup can change the outcome. In a July 2026 account, OpenAI reports different ARC-AGI-3 scores under different harness settings, including retaining reasoning and context compaction. That is a provider’s account of its own system: it illustrates why harness choices should be recorded, but it is not independent evidence that one model ranks above another.
Rank #3
Run a fair comparison on your own task set
The useful comparison is between complete systems under matched conditions. If you expect to use a model with Python, a solver, or a local environment, include that support for every candidate and score the model-plus-tools system. NIST AI 800-1’s second public draft distinguishes static question-answer evaluations from tool-enabled and computer-environment tasks, and notes that including tools may better indicate performance under realistic conditions.
- Choose representative problems. Use tasks that resemble your intended use, with known solutions or a defensible scoring rubric. Include held-out problems where possible; public benchmark items may have been exposed during model development.
- Fix the system configuration. Record the exact model version and date, prompt, tool access, context handling, and any relevant environment. Do not compare one model with code execution against another restricted to a static prompt.
- Match the budget. Give candidates the same number of attempts and the same time, token, or compute budget. If those limits differ from your real use, report both the test limit and the practical conditions you intend to use.
- Define what counts as a solve. Decide in advance how you will verify answers, handle partial solutions, and score a result. For a cryptanalysis task, distinguish a demonstrated attack from a plausible-sounding explanation.
- Repeat tests or quantify uncertainty. A small observed difference may be chance rather than a real performance gap. NIST AI 800-3, Expanding the AI Evaluation Toolbox with Statistical Models (February 2026), cautions that without uncertainty quantification it cannot be known whether one model truly performed better on a benchmark or whether the observed difference is due to chance. It also distinguishes a benchmark score from a broader claim about performance across a wider population of tasks.
- Check practical terms at decision time. Verify current access, price, latency, privacy provisions, and usage limits with the providers. These terms are volatile and are not established by the benchmark findings described here.
NIST’s AI Testing, Evaluation, Verification and Validation (AITE) program describes blind, sequestered tasks as one way to reduce train/test contamination and improve objective assessment. Its initial published examples are not cryptanalysis or puzzle-solving evaluations, so that approach is a useful evaluation principle rather than direct evidence about a candidate model in these areas.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use a comparison sheet instead of a single leaderboard
Record the conditions alongside each result. A compact comparison sheet makes it harder to mistake a setup advantage for a model advantage.
| What to record | Why it matters |
|---|---|
| Task family and difficulty tier | A known-break scheme, full-strength primitive, abstract grid task, and interactive puzzle test different things. |
| Model version and evaluation date | Results are tied to the version tested; a new release is a new candidate to evaluate. |
| Prompt, tools, and harness | Code execution, solver access, context retention, and interaction rules can change system performance. |
| Held-out task set and scoring rule | These show what was tested and how a correct or partial result was counted. |
| Attempts and time, token, or compute budget | A result is interpretable only alongside the resources allowed to produce it. |
| Verified solve rate and uncertainty | Repeated trials and uncertainty estimates help separate meaningful differences from noise. |
| Access, privacy, price, latency, and usage limits | These determine whether a strong test result is practical for your intended use; verify them directly with the provider. |
There is no universal ranking supported by the cited evidence. Choose the candidate that performs best on your representative tasks under matched conditions, then weigh its practical terms for your use case.
Keep cryptanalysis testing authorized and controlled
Cryptanalysis is dual-use: it can support defensive assessment, but capability can also enhance attacks. NIST’s AI security overview describes both possibilities and notes that AI security research is changing quickly; existing guidance does not comprehensively address several classes of machine-learning attacks. A score on a puzzle or cryptanalysis benchmark is not a security certification.
Quick Recap
- Use toy schemes, published exercises, or systems you own or have explicit permission to assess.
- Keep tests within the agreed scope and avoid applying benchmark techniques to third-party systems without authorization.
- Interpret results as evidence about the tested setup, not as proof that a cryptographic system is secure or insecure in general.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
