Reasoning Models: When to Use Them and When Not To

Reasoning models — models that generate an internal chain of thought before producing an answer — are becoming standard. Most major providers now offer them. And the marketing makes them sound like a free upgrade: same model, but smarter.

It's more complicated than that. Reasoning models are better at some things, worse at others, and significantly more expensive for everything. Here's how to think about when to reach for them.

What reasoning models actually do

Standard LLMs produce tokens one by one, left to right. They don't "think" before answering — the answer emerges directly from the forward pass.

Reasoning models (o3, claude-3-7-sonnet, Gemini 2.0 Flash Thinking, and others) generate a hidden chain of thought first — often thousands of tokens of working — before producing the final answer. You don't see this thinking, but it shapes the output.

The result: better performance on tasks that require multi-step logic, but higher latency and cost because you're paying for all those thinking tokens.

Where reasoning models win

Complex coding problems. When a problem requires understanding how multiple pieces interact — debugging a subtle concurrency issue, designing a data structure for specific constraints, implementing a non-trivial algorithm — reasoning models consistently outperform standard models. The thinking step lets them work through the problem systematically instead of pattern-matching to a plausible-looking answer.

Math and logic. Multi-step math, formal proofs, logic puzzles. Tasks where getting step 2 wrong invalidates everything downstream. Standard models often produce convincing but wrong intermediate steps. Reasoning models check their own work.

Planning with constraints. "Schedule these tasks given these dependencies and resource limits." The answer requires holding many constraints in mind simultaneously and checking for conflicts. Reasoning models handle this better.

Ambiguous tasks requiring judgment. When the right answer depends on weighing trade-offs that aren't stated explicitly, reasoning models make those trade-offs visible in their thinking.

Where standard models are better

Simple, fast tasks. Summarizing a document, answering a factual question, classifying a short text, translating a sentence. Using a reasoning model here is paying 10x for the same result with 5x the latency.

High-volume pipelines. If you're processing thousands of documents, the cost difference between a reasoning model and a standard model can be the difference between a viable product and an uneconomical one.

Creative tasks. Writing, brainstorming, tone-matching. These don't benefit from systematic logical reasoning. A reasoning model trying to write a casual email will overthink it.

Real-time interactive use. If your user is waiting for a response, a reasoning model's 10–30 second latency is often unacceptable. Standard models that respond in 1–2 seconds feel much more usable even if the output is slightly lower quality.

The practical routing pattern

The right architecture for most products uses both:

def route_to_model(task: str, complexity: str) -> str:
    if complexity == "high" and task in REASONING_TASKS:
        return "claude-opus-4-8"  # or o3, Gemini 2.0 Pro
    return "claude-haiku-4-5"     # fast, cheap, good enough

REASONING_TASKS = {
    "code_review",
    "architecture_design", 
    "multi_step_math",
    "constraint_planning",
    "debugging",
}

This is the same pattern used in compilers: the expensive analysis pass runs when it's needed, not on every token.

Budget tokens: controlling the thinking

Most reasoning model APIs let you cap the thinking budget — the number of tokens the model can use for its internal chain of thought. A larger budget produces better answers but costs more and takes longer.

For most tasks, a medium thinking budget (1–2K tokens) is the right default. Reserve the maximum budget for the hardest problems where quality matters most.

---

Reasoning models are a genuine capability improvement for hard problems. They're also easy to misuse by applying them where they add cost and latency without adding quality. The skill is knowing which problems are actually hard enough to need them.