How Production LLMs Reason Better at Inference Time (as researched by Google, OpenAI, and Anthropic)
8 techniques to make an LLM reason better at inference time, covered with tradeoffs and practical notes. Every one of them is running in production at a frontier lab today, and the research behind them comes from Google, OpenAI, and Anthropic.
When accuracy on an LLM-driven task falls short, the cheapest place to intervene is inference.
Nothing gets retrained, so weights stay the same since the whole technique is managed via the prompt and the orchestration around the LLM call.
There's actually a paper by Google that measured whether there's any benefit of doing this, and they found that under a FLOPs-matched comparison, test-time compute beat a model 14x larger:
In this article, let's walk through eight such techniques, their tradeoffs, and the single property that decides whether any of them would work for your use-case or not.
Two ways to allocate compute
Inference compute scales in two directions.
You sample the same prompt several times and select among the results, or you extend one trajectory further before the model answers.
Tree methods compose both of the techniques.
Branching is the parallel step, extending a surviving path is the sequential step, and they inherit both cost profiles along with both failure modes.
Their typical failure modes also lie in different directions.
1) Chain of thought
The model is prompted to think step by step, so intermediate results land in the context where later tokens can condition on them instead of a single forward pass carrying everything.
# Illustrative only.
# The helper functions are placeholders, not a real API.
prompt = question + "Think step by step, then give the final answer."
answer = model(prompt)
Practically speaking, the gains are smaller than the popularity of this technique suggests.
A study covered over 100 papers, 20 datasets, and 14 models, and found the benefit concentrated on math and symbolic tasks.
On MMLU, answering directly matched CoT unless the question or the response contained an equals sign.
In fact, if you have an LLM app that's running on a current reasoning model, this one is mostly redundant.
The model already writes those steps on its own, so prompting for them again adds tokens without adding structure.
2) Majority voting
Run the same prompt several times at nonzero temperature, extract the final answer from each trace, and return whichever appears most often.
# Illustrative only.
# The helper functions are placeholders, not a real API.
answers = []
for i in range(N):
trace = model(prompt, temperature=0.8)
answers.append(extract_final_answer(trace))
return most_common(answers)Agreement across samples acts as a proxy for a correctness signal, so you don't need a reward model at all.
Self-consistency added 17.9 points on GSM8K with PaLM-540B when it was introduced.
That margin, of course, has compressed as base models improved. For instance, on Gemini 2.5, 20 samples improved MATH-500 by 1.6 points and HotpotQA by 0.4, while token cost scaled close to linearly with sample count.
There's also a structural limit to it.
Voting reduces variance and does nothing about bias, so when the model misreads a problem the same way on every sample, all the traces agree, and the vote returns that wrong answer with a higher confidence estimate attached.
It also requires answers you can compare for equality, which rules out open-ended generation. So mostly just math, classification, and extraction work. Summaries and code with many valid forms naturally aren't a good fit for majority voting.
3) Best-of-N
Generate N complete answers, score each with a reward model, keep the highest scorer. This covers the cases voting can't handle, since the scorer works on any output format.
# Illustrative only.
# The helper functions are placeholders, not a real API.
candidates = []
for i in range(N):
candidates.append(model(prompt, temperature=0.8))
return max(candidates, key=reward_model.score)
N is not a hyperparameter that you can turn up indefinitely.
A paper measured that as N increased, true reward rose, peaked, and then declined as optimization pressure against the proxy reward model increased.
Past that peak, the search was finding answers the reward model liked rather than answers that were actually correct.
Another problem is that the reward model has to discriminate better than the policy generates.
That property is expensive to obtain, which is why many best-of-N setups plateau at low N and stay there.
4) Extended thinking
The model spends a token budget on internal reasoning before producing an answer.
# Illustrative only.
# The helper functions are placeholders, not a real API.
answer = model(prompt, thinking={"budget_tokens": 8000})Every major provider exposes this capability now, like:
This is the one technique on the list that moved out of prompting and into training.
R1-Zero went from 15.6% to 71.0% on AIME 2024 through RL alone, with no external scaffolding running at inference.
Longer traces are not uniformly better. Anthropic's inverse scaling work built tasks where accuracy falls as reasoning length grows: counting problems where irrelevant numbers pull the model off track, correlation tasks where it drifts from a sound prior toward a weaker feature, and constraint-satisfaction puzzles where it reopens deductions it had already settled.
For production, that makes maximum effort the wrong default. The useful setting is per-task, and finding it takes an eval rather than an assumption.
5) Self-refinement
As the name suggests, the model writes an answer, critiques its own answer, and rewrites it, with the loop repeating until some stopping condition.
# Illustrative only.
# The helper functions are placeholders, not a real API.
answer = model(prompt)
for i in range(max_rounds):
critique = model("Find errors in this answer: " + answer)
if "no errors" in critique:
break
answer = model("Rewrite using this critique: " + answer + critique)
return answer
A paper measured what this loop does when nothing external feeds it and found something weird.
On GSM8K, GPT-3.5 corrected 7.6% of its wrong answers and changed 8.8% of its correct ones, which made the overall loop net negative + the extra tokens that went into it.
The earlier positive results on self-correction used oracle labels to decide when to stop refining, so the loop only ever ran on answers already known to be wrong.
This implies filtering against ground truth, and it isn't available at inference on real traffic.
Refinement does work when the critique comes from outside the model. A compiler, a test suite, or a type checker carries information the model's own distribution doesn't have.
6) Tree of thought
The model proposes several candidate next steps instead of one, scores them, keeps the best few, and backs out of branches that don't seem promising:
# Illustrative only.
# The helper functions are placeholders, not a real API.
paths = [""]
for depth in range(max_depth):
candidates = []
for path in paths:
candidates += propose_next_steps(path, n=5)
paths = top_k(candidates, key=score_step, k=5)
if any(is_complete(p) for p in paths):
break
return best(paths)
On Game of 24, this technique took GPT-4 from 4% with chain of thought to 74% at breadth 5.
That task punishes early commitment to a wrong branch with no way to unwind, which is exactly the shape where backtracking pays.
The cost of a tree of thought can be roughly what 50-100 independent CoT attempts consume.
So before trying this, it's worth verifying whether 20-40 generations plus a vote help you attain the same accuracy, since it would consume far fewer tokens.
7) Beam search
Beam search keeps K partial solutions alive and scores every step as it's written, rather than waiting for a complete answer.
# Illustrative only.
# The helper functions are placeholders, not a real API.
beams = [""]
for depth in range(max_depth):
candidates = []
for beam in beams:
candidates += model.continue_one_step(beam, n=4)
beams = top_k(candidates, key=prm.score, k=K)
return max(beams, key=prm.score)
The step-level scorer is a process reward model (depicted as the second technique in the visual below):
The ranking against best-of-N inverts as budget grows.
A paper found that beam search scored well above best-of-N at small compute budgets, then fell below it as budgets grew, because the search started satisfying the process reward model instead of actually solving the problem.
Over-optimized runs produced repetitive low-information steps, and some solutions collapsed to one or two steps.
Ultimately, difficulty decided which method was better.
That's difficult to practically utilize in production, since difficulty isn't known before the answer exists.
8) Monte Carlo tree search
Pick a promising partial path, extend it, run it out to a full answer, push the score back up the path, and repeat across many iterations.
# Illustrative only.
# The helper functions are placeholders, not a real API.
root = Node(prompt)
for i in range(n_iterations):
node = select_most_promising(root)
child = node.expand()
answer = rollout_to_completion(child)
backpropagate(child, value_model.score(answer))
return best_child(root)
MCTS came out of AlphaGo and AlphaZero, where the move set at each turn is small and enumerable.
DeepSeek tried it too, but token generation opened a far larger space, so DeepSeek capped how far any node could be extended. This led the search to land in local optima.
They dropped process reward models as well alongside it because step correctness is hard to define in general reasoning, automated labeling doesn't scale, and the policy learns to exploit the scorer.
R1 put the search into the weights through RL with rule-based rewards instead.
Practical takeaways
Every technique here does the same two things, i.e., it generates candidates, and it picks among them.
The generation part is well understood and mostly solved. But the methods primarily differ in how they pick the final answer.
Where an exact checker exists (like unit tests, a compiler, a schema validator), these methods hold up as budgets grow.
They give you a signal that can't be tricked, and scaling the search against it keeps returning value.
But when the selector is a learned reward model, both axes degrade the same way and the degradation widens with budget.
So practically, you should build the verification signal first and scale the search second.
On tasks where you can't build a good checker in any way, inference-time compute will usually not work as well as you'd hope, and further training should be explored.
To dive deeper into that, RULER was built for that case.
It uses an LLM as the reward function, which extends RL training to agentic tasks with no verifiable answer and removes the need to hand-write a reward function per task.
I traced the full path from RLHF to GRPO with verifiable rewards and then to RULER in the article below:
👉 Over to you: which of these techniques have you actually kept in production, and what made you drop the ones you didn't?
That's a wrap!
If you enjoyed this tutorial:
Find me → @_avichawla
Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.
