Can step selection beat thinking?
Thinking solves 3.3× as many AIME problems as Jev step selection.
21 Sept 2026 · 30 AIME 2025 problems × 3 seeds · Qwen3-1.7B, Jev 1.13
Accuracy by method
Loading completed trials…
90 trials per method: 30 problems × 3 fixed seeds. Exact integer match.
Accuracy vs cost or runtime
Loading measured accuracy and resource use…
Up and left is better. The line joins the best measured trade-offs.
Background
Can a small model solve harder problems by picking better steps instead of thinking longer? We gave Qwen3-1.7B two ways to spend extra compute: turn on its built-in thinking, or write eight candidate next steps and let a judge pick one.
Methodology
We ran all 30 AIME 2025 problems with three fixed seeds per method: thinking, no thinking, Qwen as step judge, and Jev as step judge. A later 90-run follow-up let Jev reject a batch and ask for up to two more. Every method used the same Qwen weights and exact-answer grading. Judges picked next steps, not complete solutions.
Results
Thinking scored 36.7%, versus 11.1% with Jev judging, 10.0% without thinking, and 6.7% with Qwen judging. Rerolls reached 12.2%, a third of thinking’s score. Thinking also cost less per problem than either Jev method: $0.0064, versus $0.0126 for Jev judging and $0.0584 with rerolls.
Discussion
Thinking’s lead over Jev judging is clear: 25.6 points, with a 95% interval of 15.6 to 35.6. Jev’s small lead over Qwen judging is unresolved. This tests one search recipe with unequal compute on a small public dataset, not every way to use a judge. The follow-up changed the judging rule and the retry policy together, so it cannot isolate either.
Conclusion
Turn on native thinking before building a step-selection loop for a model like this. Jev judging and rerolls added work without catching up. A new search policy needs to beat that simple baseline first.
Data and run details
Loading experiment status…
Results
Full result summary
Results by problem
Paired differences
95% intervals resample the 30 problems, keeping all three seeds together.
Follow-up: rerolls
Loading 90 trials…
Jev accepts a step at ≥0.8. Otherwise it tries up to two more batches of eight and keeps the best step seen. Generator, seeds, limits, and grading stay fixed.
This changes judging and rerolls together, so neither effect is isolated. The original 360 trials stay unchanged, and costs use the original GPU rate.
Follow-up protocolFollow-up sourceCombined resultsFollow-up auditFollow-up expensesOriginal resultsOriginal traces
Method
Qwen proposes eight next steps, and Qwen or Jev picks one. We compare that with Qwen’s thinking mode on and off. Judges pick steps, not complete answers.
30 problems × 3 seeds. Search allows 32 steps of 384 tokens plus one 2048-token finalization; the baselines cap output at 38912 tokens. Sampling settings and stopping rules are in the protocol.
Budgets, timing, and caveats
Only the judge changes between search methods. Qwen makes a constrained, non-thinking A–H choice; Jev makes one typed Choice over the same state and rubric.
No thinking means Qwen’s thinking mode is off, not that it cannot show working. Both baselines get the same request to reason step by step and put the final answer in a box.
All methods run on the same rented RTX 4090 with vLLM 0.10.2, bfloat16, no quantization, and prefix caching off. Four problems run at a time, and each proposal call batches eight continuations. Three fixed seeds (2025, 2026, 2027) cover all 30 problems, and method order rotates between seeds. Sampling seeds do not guarantee bitwise reproducibility under dynamic batching.
Runtime is the mean end-to-end time per problem under that load, including judge calls. Cost splits each method’s GPU batch time at $0.411111/hour across its 30 trials, then adds Jev input tokens at $0.042/million. GPU time spent waiting for Jev counts. Setup, warmup, downloads, and this website are excluded. This is amortized rental cost, not a token-provider quote or single-request latency.
Search and baseline compute budgets differ. The two search judges share generator settings, limits, rubric, and seeded candidate order; their paths diverge after the first different pick. The Qwen judge has no hidden thinking or free-form critique, so this does not cover every possible Qwen judge design. There is no random-pick or N = 1 stepwise ablation.
Generators and judges never see the answer key. Prompts were smoke-tested on synthetic arithmetic, then frozen before any AIME run. Missing or invalid final answers count wrong. For thinking, only the answer after the closing think tag is graded. No prompt or seed was chosen by AIME accuracy.
The first batch hit a 300-second HTTP client timeout after 29 no-thinking completions. That incomplete, unscored batch was archived and restarted with streaming; prompts and seeds were unchanged. Its spend is excluded from per-method costs.
AIME 2025 is small and public, and training contamination is unknown. The intervals describe this experiment, not performance on unseen competition math. The trade-off chart compares measured configurations on one serving setup, not a full N or compute sweep.
Token limits and stopping
Invalid or missing boxed integers count wrong. Hitting the step limit triggers one finalization call. Candidate steps can also hit the 384-token cap; every finish reason is in the full traces.
Downloads
A Jev trace logging bug meant some requests had to be reconstructed and labeled. Raw logs and scores are unchanged. Trace note · Original audit · Parser check · Expense estimate
Original protocolOriginal sourceLive results JSONLive trials JSONFull tracesSource hashes
Sources: math-ai/aime25 · Qwen3-1.7B · Jev model and pricing