Three roads to reasoning in language models

A model that plans is not born that way. Somewhere between a next-token predictor and an agent that can hold a goal across twenty steps, something has to install the habit of deliberation — trying an idea, noticing it fails, backing up, trying another.

There are three places that habit can come from. It can be written into the training data, so the model imitates deliberation. It can be discovered by search during training, so the model absorbs the results of a process it never sees at inference. Or it can be spent at inference time, so the deliberation happens in the open, while the user waits.

Almost every reasoning system shipped since 2022 is some mixture of these three. Telling them apart is useful, because they fail differently, cost differently, and — this is the part most write-ups skip — the industry has quietly picked two of them and largely abandoned the third. Understanding why is what makes the third one interesting again.

Road 1 — Put the reasoning in the training data

The first road is the oldest and the cheapest to explain. If you want a model to show its work, show it work.

Wei et al. found in 2022 that simply prompting a large model with a few worked examples containing intermediate steps unlocked arithmetic and symbolic reasoning that the same model could not do when asked for the answer directly [1]. Nothing about the weights changed. The chain of thought was already latent; the prompt was the key.

That discovery immediately became a training recipe. If chain-of-thought traces help, generate a lot of them from a strong model and fine-tune a small one on them. Hsieh et al.'s Distilling Step-by-Step showed that training a small model on rationales — not just labels — beat standard fine-tuning while using dramatically less data [2]. Microsoft's Orca pushed the same idea further, training on rich explanation traces rather than terse answer pairs [3]. DeepSeek-R1's release in 2025 included a set of dense models distilled from the reasoning model's own traces, which is Road 1 applied to the output of Road 2 [4].

The economics here are excellent and the ceiling is real. A distilled model imitates the shape of deliberation without necessarily acquiring the judgment behind it. It will produce a confident five-step derivation with a wrong step three, in exactly the register of a correct one. Nothing in the training signal taught it to distinguish those two, because both looked like fluent reasoning text.

Road 2 — Let search do the teaching

The second road inverts the relationship. Instead of showing the model good reasoning, you let a search process find good reasoning and train on whatever it discovers.

The lineage starts outside language models entirely. AlphaGo Zero learned superhuman Go from self-play alone, with Monte Carlo tree search acting as a policy-improvement operator: search produces better moves than the raw network, and the network is then trained to predict what search found [5]. AlphaZero generalized the same loop to chess and shogi [6]. The structural insight — search generates the training signal, the network internalizes it, the improved network makes search better — is the single most transplanted idea in this area.

Transplanting it to language took a few different forms:

• STaR (Zelikman et al., 2022) is the minimal version: let the model generate rationales, keep the ones that reach the correct answer, fine-tune on those, repeat [7]. Search here is just sampling with a correctness filter, and it bootstraps anyway.
• Process reward models (Lightman et al., Let's Verify Step by Step, 2023) supervise the steps rather than the final answer, which gives search something much denser to hill-climb on [8].
• Stream of Search (Gandhi et al., 2024) takes the idea to its logical conclusion: serialize the entire search trace — including the dead ends and the backtracking — into the training text, so the model learns to search in-context rather than learning only the winning path [9].
• RLVR — reinforcement learning from verifiable rewards, given its canonical treatment in AI2's Tülu 3 (2024) — replaces the learned reward model with a program that checks the answer: unit tests, a math grader, a parser [10].
• DeepSeek-R1 (2025) demonstrated publicly that large-scale RL against verifiable rewards produces long, self-correcting chains of thought as an emergent behavior, and published the recipe [4].

OpenAI described their own version in one sentence that stays worth quoting: "Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process." [11]

The key economic property of Road 2: the search cost is paid once, by the lab, at training time. Every user afterward gets a model that behaves as if it had searched, without paying for the search.

Road 3 — Spend the compute at inference time

The third road refuses to pay in advance. Keep the model as it is; when a hard question arrives, spend more compute on that question.

The simplest version is self-consistency: sample the same chain-of-thought prompt many times and take the majority answer [12]. No new model, no new training, a large accuracy gain on math — because independent errors scatter and correct answers agree.

Tree of Thoughts (Yao et al., 2023) made the search explicit: treat partial reasoning as nodes in a tree, generate several continuations at each step, have the model evaluate them, and use breadth- or depth-first search over the result [13]. This is the canonical "give the model a tree" paper, and the ancestor of a great deal of subsequent work, including ours.

Then came the scaling studies. Brown et al.'s Large Language Monkeys showed that coverage — the chance that some sample is correct — climbs steeply with the number of samples, which reframes the problem as one of selecting the right sample rather than generating it [14]. Snell et al. asked the sharper question: given a fixed compute budget, is it better spent on a bigger model or on more inference for a smaller one? Their answer, roughly: it depends on how hard the problem is, and for many problems test-time compute wins [15].

The correction most write-ups skip

Here is where the tidy three-road story usually goes wrong, and it is worth being precise about.

When o1-style and R1-style reasoning models arrived, a lot of commentary described them as "doing tree search at inference time." They are not. Their chains of thought are forward and sequential — one long stream, with backtracking expressed as text inside that stream rather than as an external controller managing branches. Nothing enumerates siblings; nothing prunes; nothing holds a frontier in memory. The search moved into training.

So Road 3, honestly stated, is inference-time compute scaling — of which sampling-based methods (self-consistency, best-of-n, verifier-reranked sampling) are the branch the industry actually took, and explicit structured search over reasoning states is the branch it largely did not.

Why not? Because in the general case, explicit search needs something the general case doesn't have: a way to tell whether a branch is good. In Tree-of-Thoughts-family methods, the model grades its own thoughts. Huang et al. examined that assumption directly and found that self-correction without external feedback fixes wrong answers about as often as it breaks right ones [16]. Search over a noisy evaluator inherits the noise, multiplies the cost, and returns something that looks more rigorous. Branches without a verifier are opinions with better formatting.

Meanwhile, Road 2 was quietly solving the same problem in a way that scales: RLVR uses a real verifier — but at training time, where you can afford to run it a million times and where its cost never touches the user.

The branch not taken, and where it still pays

That leaves a specific, narrow, and genuinely useful gap.

If the evaluator is the problem with inference-time search, then inference-time search should work exactly where a real evaluator is cheap and available at inference time: domains with executable checks. Code that can be run against tests. A board position that a legal-move generator can validate. A schedule a constraint solver can accept or reject. An arithmetic claim a program can recompute.

That is where our own work sits. Reason Tree is a bounded, verifier-grounded inference-time controller: candidate actions enumerated by code, each played out into a real successor state, an adversary pushing back on each branch, hard compute budgets, and one stubborn rule — the model never gets the last word on anything a machine can check. The model's job is moved from deciding to explaining.

The honest scope matters as much as the idea. On a frozen rated chess holdout, that controller took a fast model from 1 correct out of 25 to 21 out of 25, with the same unchanged controller producing a comparable rescue one model tier up. And on AIME with tools enabled — where the model writes its own checker and the one-shot agent nearly saturates the benchmark — the tree added only cost. Against matched-compute sampling without tools, sampling won. Both results are published, because the second one is what makes the first one meaningful: this is not a general reasoning upgrade, it is a technique with a precondition.

Three roads, then, and a way to choose between them. If you can afford training runs and you have verifiable tasks, Road 2 is where the frontier is. If you need a small model to punch above its weight on a task a big model already does well, Road 1 is cheap and effective. And if you're building an agent inside a domain where a machine can check the work — a compiler, a test suite, a solver, a rules engine — Road 3 in its abandoned form is still sitting there, unfashionable and undervalued, because the thing that made it fail in general is exactly the thing your domain hands you for free.

References

1. Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — https://arxiv.org/abs/2201.11903
2. Hsieh et al. (2023), Distilling Step-by-Step! — https://arxiv.org/abs/2305.02301
3. Mukherjee et al. (2023), Orca: Progressive Learning from Complex Explanation Traces — https://arxiv.org/abs/2306.02707
4. DeepSeek-AI (2025), DeepSeek-R1 — https://arxiv.org/abs/2501.12948
5. Silver et al. (2017), Mastering the game of Go without human knowledge (AlphaGo Zero), Nature 550:354–359 — https://www.nature.com/articles/nature24270
6. Silver et al. (2017), Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (AlphaZero) — https://arxiv.org/abs/1712.01815
7. Zelikman et al. (2022), STaR: Bootstrapping Reasoning With Reasoning — https://arxiv.org/abs/2203.14465
8. Lightman et al. (2023), Let's Verify Step by Step — https://arxiv.org/abs/2305.20050
9. Gandhi et al. (2024), Stream of Search (SoS): Learning to Search in Language — https://arxiv.org/abs/2404.03683
10. Lambert et al. (2024), Tülu 3: Pushing Frontiers in Open Language Model Post-Training — https://arxiv.org/abs/2411.15124
11. OpenAI (2024), Learning to reason with LLMs — https://openai.com/index/learning-to-reason-with-llms/
12. Wang et al. (2022), Self-Consistency Improves Chain of Thought Reasoning in Language Models — https://arxiv.org/abs/2203.11171
13. Yao et al. (2023), Tree of Thoughts: Deliberate Problem Solving with Large Language Models — https://arxiv.org/abs/2305.10601
14. Brown et al. (2024), Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — https://arxiv.org/abs/2407.21787
15. Snell et al. (2024), Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — https://arxiv.org/abs/2408.03314
16. Huang et al. (2023), Large Language Models Cannot Self-Correct Reasoning Yet — https://arxiv.org/abs/2310.01798
17. Xu et al. (2025), Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models — https://arxiv.org/abs/2501.09686
18. Besta et al. (2025), Reasoning Language Models: A Blueprint — https://arxiv.org/abs/2501.11223
19. Zhang et al. (2025), A Survey on Test-Time Scaling in Large Language Models — https://arxiv.org/abs/2503.24235
20. Rush & Ritter, awesome-o1 — https://srush.github.io/awesome-o1/