Chapter 07 · Planning & Reasoning
Ask an LLM a hard question and it might get it right. Ask it to solve a hard problem and it needs a strategy. This chapter is about teaching your agent to think before it acts.
Here is what happens when you do not.
You give a naive agent this task: "Plan a marketing campaign for a new product launch, including budget allocation, channel strategy, and timeline." The agent dives in immediately. It produces a response in one shot -- a few paragraphs about social media, some hand-waving about "targeting the right audience," a budget number pulled from nowhere. It recommends Instagram ads for a B2B industrial sensor. It allocates 70% of the budget to "viral content" with no explanation. It forgets to mention the launch date entirely. When you ask it to justify the numbers, it contradicts itself.
Now give the same task to an agent that reasons before it acts. The agent pauses. It breaks the problem down: first, understand the product and audience. Second, research comparable launches. Third, allocate budget across channels with justification. Fourth, build a timeline with dependencies. Fifth, synthesize everything into a coherent plan. It works through each step, using the output of one to inform the next. The result is thorough, internally consistent, and defensible.
Same model. Same task. The difference is not the brain -- it is the strategy wrapped around the brain.
Anchor
Reasoning is what separates agents that answer questions from agents that solve problems. A single LLM call can answer "What is the capital of France?" It cannot plan a product launch, debug a distributed system, or research a competitive landscape. Those tasks require structured thinking across multiple steps -- decomposing the problem, exploring alternatives, verifying intermediate results, and synthesizing a final answer.
This chapter covers the four major reasoning patterns that make this possible. You will learn what each pattern is, when to use it, how to implement it, and what it costs. By the end, you will have a single agent that can switch between all four strategies depending on the task.
These patterns are not academic curiosities. They are the difference between a demo and a system. Every production agent you build will use at least one of them. Most will use several.
Section 1: Chain of Thought (CoT)
Chain of Thought is the simplest reasoning pattern. It is also the most widely applicable. The idea is almost too obvious: instead of asking the model to produce an answer directly, you ask it to show its work.
How It Works
A standard prompt: "If a store sells apples for $2 each and oranges for $3 each, and I buy 4 apples and 3 oranges with a $20 bill, how much change do I get?"
A standard response: "$3."
A CoT prompt: "If a store sells apples for $2 each and oranges for $3 each, and I buy 4 apples and 3 oranges with a $20 bill, how much change do I get? Think step by step."
A CoT response:
4 apples × $2 = $8
3 oranges × $3 = $9
Total spent = $8 + $9 = $17
Change = $20 - $17 = $3
Same answer. Different process. The CoT version forces the model to generate intermediate reasoning steps before the final answer. Each step constrains the next. The model cannot jump to a conclusion without showing its work, and showing the work makes the conclusion more likely to be correct.
Why It Works
LLMs are next-token predictors. When you ask for an answer directly, the model predicts the answer token based on the question tokens. It has one shot. If the answer requires multiple inferential steps, the model has to do them all implicitly in a single forward pass. Sometimes it succeeds. Often it fails.
When you force the model to generate intermediate steps, each step becomes part of the context for the next step. The model is no longer doing multi-step reasoning in one shot. It is doing single-step reasoning, repeatedly, with each step grounded in the concrete output of the previous one. The probability of a correct final answer goes up because the probability of each individual step being correct is higher than the probability of the entire chain being correct in one shot.
CoT transforms a hard problem into a sequence of easy problems. The model is good at easy problems. Let it be good at what it is good at.
Zero-Shot CoT
The simplest form of CoT requires no examples. You add one sentence to your prompt: "Let's think step by step." That is it. The model does the rest.
def zero_shot_cot(client, question: str) -> str:
"""Chain of Thought with no examples -- just add the magic phrase."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a careful reasoner. "
"Always think step by step before giving a final answer."},
{"role": "user", "content": f"{question}\n\nLet's think step by step."}
]
)
return response.choices[0].message.content
This is surprisingly effective. The phrase "Let's think step by step" triggers the model's training on reasoning traces -- textbook solutions, forum explanations, tutorial walkthroughs. The model has seen millions of examples of step-by-step reasoning during training. The phrase activates that pattern.
Zero-shot CoT improves accuracy on math word problems by 10-30 percentage points depending on the model and task. It costs nothing to implement. It should be your default for any task that requires more than one inferential step.
Few-Shot CoT
For harder problems, show the model examples of good reasoning chains. This is few-shot CoT:
def few_shot_cot(client, question: str) -> str:
"""Chain of Thought with explicit examples of reasoning."""
system_prompt = """You are a careful reasoner. Always break problems
into steps and show your work before giving a final answer.
Here are examples of how to reason:
Example 1:
Q: A bakery sells cookies in boxes of 12. If Sarah needs 80 cookies
for a party, how many boxes should she buy?
A: Let me think step by step.
Each box has 12 cookies. Sarah needs 80 cookies.
80 / 12 = 6.67 boxes. Since she can't buy partial boxes, she needs 7 boxes.
7 boxes × 12 cookies = 84 cookies. She'll have 4 extra.
Answer: Sarah should buy 7 boxes.
Example 2:
Q: A train travels 240 miles at 60 mph, then 180 miles at 45 mph.
What is the average speed for the entire trip?
A: Let me think step by step.
First leg: 240 miles at 60 mph = 240/60 = 4 hours.
Second leg: 180 miles at 45 mph = 180/45 = 4 hours.
Total distance = 240 + 180 = 420 miles.
Total time = 4 + 4 = 8 hours.
Average speed = 420/8 = 52.5 mph.
Answer: The average speed is 52.5 mph.
Now solve this problem using the same step-by-step approach."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": question}
]
)
return response.choices[0].message.content
The examples teach the model how to structure its reasoning, not just that it should reason. The format matters: state the step, compute the result, move to the next step. The model pattern-matches on the structure and applies it to the new problem.
Structured CoT Output
For agent systems, you do not want free-form reasoning text. You want structured reasoning you can parse, validate, and act on. Here is a CoT agent that produces structured output:
import json
from pydantic import BaseModel, Field
from typing import Optional
class CoTStep(BaseModel):
step_number: int
reasoning: str = Field(description="What you are thinking at this step")
conclusion: str = Field(description="What you conclude from this step")
class CoTResponse(BaseModel):
steps: list[CoTStep]
final_answer: str
COT_SYSTEM_PROMPT = """You are a careful reasoner. For every problem, you
must break your reasoning into explicit steps before giving a final answer.
Respond in this JSON format:
{
"steps": [
{"step_number": 1, "reasoning": "...", "conclusion": "..."},
{"step_number": 2, "reasoning": "...", "conclusion": "..."}
],
"final_answer": "Your final answer after all steps"
}
Rules:
- Each step must build on the conclusions of previous steps.
- The final_answer must be directly supported by your steps.
- If you realize a previous step was wrong, add a step that corrects it.
- Be specific. "Think about the problem" is not a valid step."""
def run_cot_agent(client, task: str) -> CoTResponse:
"""Run a Chain of Thought agent with structured output."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": COT_SYSTEM_PROMPT},
{"role": "user", "content": task}
],
response_format={"type": "json_object"}
)
raw = response.choices[0].message.content
data = json.loads(raw)
return CoTResponse(**data)
Run it on a multi-step problem:
result = run_cot_agent(client,
"A company has 3 offices. Office A has 45 employees, Office B has "
"38 employees, and Office C has 52 employees. The company wants to "
"redistribute employees so each office has the same number. How many "
"employees need to move, and from which offices to which?"
)
for step in result.steps:
print(f"Step {step.step_number}: {step.reasoning}")
print(f" -> {step.conclusion}")
print(f"\nFinal: {result.final_answer}")
Output:
Step 1: First, find the total number of employees.
-> Total = 45 + 38 + 52 = 135 employees
Step 2: Divide by 3 to find the target per office.
-> Target = 135 / 3 = 45 employees per office
Step 3: Compare each office to the target.
-> Office A: 45 - 45 = 0 (already at target)
-> Office B: 38 - 45 = -7 (needs 7 more)
-> Office C: 52 - 45 = +7 (has 7 extra)
Step 4: Determine the moves.
-> Move 7 employees from Office C to Office B. Office A stays the same.
Final: 7 employees need to move, all from Office C to Office B.
When CoT Helps (and When It Doesn't)
CoT helps when the problem requires multiple inferential steps: math, logic, planning, multi-step reasoning, debugging, analysis. If the answer cannot be produced in a single cognitive hop, CoT improves it.
CoT does not help when the task is simple factual recall ("What is the capital of France?"), creative writing, or tasks where reasoning is not the bottleneck. Adding "think step by step" to "Write a poem about autumn" produces a worse poem, not a better one. The model over-intellectualizes.
CoT also does not help when the model needs external information. "Think step by step about today's stock price" produces hallucinated reasoning about a hallucinated price. For that, you need the next pattern.
Section 2: ReAct (Reasoning + Acting)
ReAct is the pattern that powers most modern agents. You have been building it since Chapter 4. Now give it a name and understand why it works.
The ReAct Loop
ReAct interleaves reasoning steps with actions. The agent does not just think -- it thinks, then acts, then observes the result, then thinks again. The loop:
Thought -> Action -> Observation -> Thought -> Action -> Observation -> ... -> Final Answer
Each thought is a reasoning step: "I need to find X. To do that, I should search for Y." Each action is a tool call: search("Y"). Each observation is the tool's output: search results, API response, calculation result. The observation feeds into the next thought, and the cycle continues.
The key insight: reasoning informs action, and action results inform the next reasoning step. The model does not have to get everything right in its head. It can offload work to tools, then reason about what the tools returned.
A Complete ReAct Trace
Here is a ReAct agent working through a complex query:
Task: "What's the market cap of the company that makes the most popular smartphone?"
Thought 1: I need to find the most popular smartphone first, then
identify the company that makes it, then find that company's market cap.
Action 1: search_web("most popular smartphone 2024 global sales")
Observation 1: The iPhone 15 was the best-selling smartphone globally
in 2024, with approximately 65 million units sold. Samsung's Galaxy S24
followed at 35 million units.
Thought 2: Apple makes the iPhone. I need Apple's current market cap.
Action 2: search_web("AAPL market cap July 2026")
Observation 2: Apple Inc. (AAPL) market capitalization is approximately
$3.8 trillion as of July 2026.
Thought 3: I have the answer. The most popular smartphone is the iPhone 15,
made by Apple, whose market cap is approximately $3.8 trillion.
Final Answer: Apple Inc. (AAPL) has a market cap of approximately $3.8
trillion. The iPhone 15 was the best-selling smartphone globally in 2024.
Three thoughts, two actions, one answer. The agent did not know the most popular smartphone or Apple's market cap. It found both by searching. The reasoning steps connected the searches. The search results informed the reasoning.
Implementing ReAct
Here is a complete ReAct agent. It is the agent loop from Chapter 4, upgraded with tools from Chapter 5, and now given an explicit reasoning structure:
import json
from typing import Any
from dataclasses import dataclass, field
@dataclass
class ReActTrace:
"""A single step in the ReAct loop."""
thought: str
action: str | None = None
action_input: dict | None = None
observation: str | None = None
@dataclass
class ReActAgent:
"""An agent that interleaves reasoning with tool use."""
client: Any
model: str = "gpt-4o"
tools: dict[str, callable] = field(default_factory=dict)
max_steps: int = 10
def run(self, task: str) -> dict:
"""Run the ReAct loop on a task. Returns the trace and final answer."""
trace: list[ReActTrace] = []
messages = [
{"role": "system", "content": self._build_system_prompt()},
{"role": "user", "content": task}
]
for step_num in range(1, self.max_steps + 1):
response = self.client.chat.completions.create(
model=self.model,
messages=messages,
response_format={"type": "json_object"}
)
raw = response.choices[0].message.content
parsed = json.loads(raw)
thought = parsed.get("thought", "")
action_name = parsed.get("action")
action_input = parsed.get("action_input", {})
final_answer = parsed.get("final_answer")
step = ReActTrace(thought=thought)
print(f"\n--- Step {step_num} ---")
print(f"THOUGHT: {thought}")
if final_answer:
print(f"FINAL ANSWER: {final_answer}")
trace.append(step)
return {"trace": trace, "answer": final_answer}
if action_name and action_name in self.tools:
step.action = action_name
step.action_input = action_input
print(f"ACTION: {action_name}({action_input})")
try:
result = self.tools[action_name](**action_input)
step.observation = str(result)
print(f"OBSERVATION: {result[:200]}...")
except Exception as e:
step.observation = f"Error: {e}"
print(f"OBSERVATION ERROR: {e}")
messages.append({"role": "assistant", "content": raw})
messages.append({"role": "user",
"content": f"Tool result: {step.observation}\n\n"
f"What is your next thought?"})
else:
# No action and no final answer -- nudge the agent
messages.append({"role": "assistant", "content": raw})
messages.append({"role": "user",
"content": "You must either call a tool or provide a "
"final_answer. What do you do?"})
trace.append(step)
return {"trace": trace, "answer": "Agent did not finish within step limit."}
def _build_system_prompt(self) -> str:
tool_descriptions = ""
for name, fn in self.tools.items():
tool_descriptions += f"- {name}: {fn.__doc__}\n"
return f"""You are a ReAct agent. You solve problems by interleaving
reasoning with tool use. Follow this pattern on every step:
1. THOUGHT: Reason about what you know and what you need to find out.
2. ACTION: If you need information, call a tool.
3. OBSERVATION: The tool result will be provided to you.
4. Repeat until you can give a final answer.
Available tools:
{tool_descriptions}
Respond in this JSON format on every turn:
{{
"thought": "Your reasoning about the current state and what to do next",
"action": "tool_name or null",
"action_input": {{"param": "value"}} or {{}},
"final_answer": "Your answer, only when you are completely done, or null"
}}
Rules:
- Always include a thought, even when giving a final answer.
- If you need information, call a tool. Do not guess.
- If you have enough information, set action to null and provide final_answer.
- If a tool returns an error, think about why and try a different approach."""
Now give it tools and a task:
def search_web(query: str) -> str:
"""Search the web for current information."""
# In production, this calls a real search API
results = {
"most popular smartphone 2024 global sales":
"iPhone 15: 65M units. Samsung Galaxy S24: 35M units.",
"AAPL market cap July 2026":
"Apple Inc. market cap: $3.8 trillion as of July 2026.",
}
return results.get(query, f"No results found for: {query}")
def calculate(expression: str) -> str:
"""Evaluate a mathematical expression."""
return str(eval(expression))
agent = ReActAgent(
client=client,
tools={"search_web": search_web, "calculate": calculate}
)
result = agent.run(
"What's the market cap of the company that makes the most popular "
"smartphone? Also, if that market cap were divided equally among "
"the US population (335 million), how much would each person get?"
)
print(f"\n{'='*50}")
print(f"FINAL ANSWER: {result['answer']}")
print(f"Steps taken: {len(result['trace'])}")
The ReAct pattern is not complicated. It is the agent loop with explicit reasoning steps. What makes it powerful is the interleaving: the model thinks, acts, observes, and thinks again. Each observation grounds the next thought in reality. The model cannot hallucinate its way through a ReAct loop because every factual claim gets checked against a tool result.
ReAct is the minimum viable reasoning pattern for any agent that uses tools. If your agent calls tools, it is doing ReAct whether you named it or not. Naming it lets you debug it, optimize it, and teach it to do it better.
Section 3: Tree of Thought (ToT)
Chain of Thought explores one path. ReAct explores one path with tool support. But some problems require exploring multiple paths and picking the best one. That is Tree of Thought.
When One Path Is Not Enough
Consider this problem: "Design a feature to increase user retention for a fitness app."
A CoT agent picks one approach and runs with it: "Add social features so users can compete with friends." It produces a reasonable answer. But it never considered gamification, personalized coaching, habit stacking, or content variety. It picked the first idea that came to mind and committed.
A ToT agent generates multiple approaches, evaluates each one, pursues the most promising, and backtracks if it hits a dead end. It explores the solution space instead of tunneling down the first path.
How ToT Works
Tree of Thought treats reasoning as a search problem over a tree of possible thought sequences:
[Root: Increase retention]
/ | \
[Social] [Gamification] [Personalization]
/ \ / \ / \
[Leader- [Friend [Points & [Streak [AI [Custom
boards] feeds] badges] system] coach] plans]
| | | | | |
... ... ... ... ... ...
At each node, the agent generates N possible next thoughts. It scores each one. It pursues the highest-scoring path. If a path dead-ends, it backtracks and tries the next best.
BFS vs DFS of Thought
Two exploration strategies:
Breadth-First Search (BFS): Generate all possible next steps at the current depth. Score them. Keep the top K. Expand all of them to the next depth. Repeat. BFS explores broadly. It is good when you need to compare many alternatives at each stage. It is expensive -- you make N x K LLM calls per depth level.
Depth-First Search (DFS): Generate possible next steps. Pick the best one. Go deep on that path until you hit a dead end or a solution. If dead end, backtrack to the last branch point and try the next best. DFS is cheaper but can miss better solutions hiding in unexplored branches.
Implementing a ToT Agent
Here is a BFS Tree of Thought agent:
import json
from dataclasses import dataclass, field
@dataclass
class ThoughtNode:
"""A node in the tree of thought."""
content: str
score: float = 0.0
children: list["ThoughtNode"] = field(default_factory=list)
depth: int = 0
TOT_GENERATE_PROMPT = """You are solving a problem by exploring multiple
approaches. Given the current state of thinking, generate {num_branches}
distinct next steps. Each step should explore a different angle.
Current thinking:
{current_state}
Generate {num_branches} different next steps. For each, provide:
1. The step content (what to explore next)
2. A brief rationale for why this direction is promising
Respond in JSON:
{{"branches": [{{"content": "...", "rationale": "..."}}]}}"""
TOT_EVALUATE_PROMPT = """You are evaluating potential next steps for solving
a problem. Score each step from 0.0 to 1.0 based on:
- How likely it is to lead to a good solution
- How specific and actionable it is
- Whether it addresses the core problem
Problem: {problem}
Current state: {current_state}
Steps to evaluate:
{steps_text}
Respond in JSON:
{{"scores": [0.8, 0.3, ...]}}"""
def run_tot_agent(client, problem: str, num_branches: int = 3,
max_depth: int = 3, beam_width: int = 2) -> ThoughtNode:
"""
Tree of Thought agent using BFS with beam search.
At each depth, generates num_branches candidates, scores them,
and keeps the top beam_width to expand further.
"""
root = ThoughtNode(content=problem, depth=0)
frontier = [root]
for depth in range(max_depth):
next_frontier = []
all_candidates = []
for node in frontier:
# Build the path from root to this node
path = _get_path(node)
# Generate candidate next steps
candidates = _generate_candidates(
client, problem, path, num_branches
)
for candidate in candidates:
child = ThoughtNode(
content=candidate["content"],
depth=depth + 1
)
node.children.append(child)
all_candidates.append((child, candidate, node))
if not all_candidates:
break
# Score all candidates
scores = _evaluate_candidates(
client, problem, all_candidates
)
for (child, _, _), score in zip(all_candidates, scores):
child.score = score
# Keep top beam_width for next depth
all_candidates.sort(key=lambda x: x[0].score, reverse=True)
next_frontier = [c[0] for c in all_candidates[:beam_width]]
print(f"\nDepth {depth + 1}:")
for node in next_frontier:
print(f" [{node.score:.2f}] {node.content[:100]}...")
frontier = next_frontier
# Return the highest-scoring leaf
best = _get_best_leaf(root)
return best
def _get_path(node: ThoughtNode) -> list[str]:
"""Reconstruct the path from root to this node."""
path = []
current = node
while current is not None:
path.append(current.content)
current = current.parent if hasattr(current, 'parent') else None
return list(reversed(path))
def _generate_candidates(client, problem: str, path: list[str],
num_branches: int) -> list[dict]:
"""Generate candidate next steps from the current state."""
current_state = "\n".join(f"- {p}" for p in path)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": TOT_GENERATE_PROMPT.format(
num_branches=num_branches,
current_state=current_state
)}],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)["branches"]
def _evaluate_candidates(client, problem: str,
candidates: list[tuple]) -> list[float]:
"""Score all candidates."""
current_state = _get_path(candidates[0][2]) if candidates else []
current_state_str = "\n".join(f"- {p}" for p in current_state)
steps_text = ""
for i, (child, candidate, _) in enumerate(candidates):
steps_text += f"{i}: {candidate['content']}\n"
steps_text += f" Rationale: {candidate.get('rationale', 'N/A')}\n\n"
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": TOT_EVALUATE_PROMPT.format(
problem=problem,
current_state=current_state_str,
steps_text=steps_text
)}],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)["scores"]
def _get_best_leaf(node: ThoughtNode) -> ThoughtNode:
"""Find the highest-scoring leaf in the tree."""
if not node.children:
return node
best_child = max(node.children, key=lambda c: _get_best_leaf(c).score)
return _get_best_leaf(best_child)
Run it on a creative problem:
best = run_tot_agent(client,
"Design a feature to increase user retention for a fitness app. "
"Users sign up, use the app for 2-3 weeks, then stop. "
"The app tracks workouts, calories, and steps."
)
print(f"\nBest path score: {best.score:.2f}")
print(f"Best solution: {best.content}")
A sample trace:
Depth 1:
[0.85] Social accountability: pair users with workout buddies...
[0.72] Adaptive goal-setting: AI adjusts targets based on...
Depth 2:
[0.91] Buddy matching algorithm based on fitness level and schedule...
[0.78] Group challenges with shared goals and leaderboards...
Depth 3:
[0.94] "Commitment Contracts": users pledge workouts with friends,
missed sessions trigger a small donation to charity...
The agent explored social features, gamification, and personalization. It scored social accountability highest, then drilled into buddy matching, group challenges, and commitment contracts. The final solution -- commitment contracts with charitable stakes -- is specific, creative, and grounded in behavioral science. A CoT agent would not have found it because it would have committed to the first idea and never backtracked.
When ToT Helps (and When It's Overkill)
ToT helps with creative problem-solving, strategy, puzzles, debugging complex systems, and any task where the solution space is large and the best path is not obvious from the start.
ToT is overkill for most everyday tasks. It is expensive -- each depth level costs (num_branches x beam_width) LLM calls for generation plus one call for evaluation. A 3-depth tree with 3 branches and beam width 2 costs roughly 27 LLM calls. A CoT solution to the same problem costs 1-3 calls. ToT adds 10-50x the token cost.
Use ToT when the cost of a suboptimal solution exceeds the cost of the extra LLM calls. If you are generating marketing copy, CoT is fine. If you are designing a system architecture that will cost millions to build wrong, ToT earns its keep.
Section 4: Plan-and-Solve
For complex, structured tasks, neither CoT nor ReAct is enough. The agent needs a plan -- an explicit decomposition of the task into steps, with dependencies, before it starts executing. That is Plan-and-Solve.
The Two-Phase Approach
Plan-and-Solve splits the agent's work into two distinct phases:
Phase 1 (Plan): "Given this task, create a detailed step-by-step plan. Do not execute any steps yet. Just produce the plan."
Phase 2 (Solve): "Now execute each step of the plan in order. After each step, note what you learned and whether the plan needs to change."
The plan serves as working memory. It is stored in the agent's scratchpad and referenced at each step. The agent does not get lost in the details because the plan provides a map. When a step reveals new information that invalidates the plan, the agent updates the plan.
Why Plan-and-Solve Beats Pure ReAct for Complex Tasks
ReAct is reactive. At each step, the agent asks: "What should I do right now?" It has no long-term strategy. For a task with 15 steps and complex dependencies, a pure ReAct agent will thrash. It will start down one path, realize it needed information from a different path, backtrack, lose context, and produce an incoherent result.
Plan-and-Solve is strategic. The agent sees the whole task before it starts. It identifies dependencies. It orders steps correctly. It knows what "done" looks like before it begins.
The plan also provides a natural checkpoint system. After each step, the agent can ask: "Did this step succeed? Does the plan still make sense? Do I need to replan?" This is dynamic replanning -- the plan is a living document, not a fixed script.
Implementing Plan-and-Solve
import json
from dataclasses import dataclass, field
@dataclass
class PlanStep:
"""A single step in a plan."""
step_id: int
description: str
depends_on: list[int] = field(default_factory=list)
status: str = "pending" # pending, in_progress, completed, failed
result: str | None = None
@dataclass
class Plan:
"""A plan composed of ordered steps."""
steps: list[PlanStep]
goal: str
PLANNER_PROMPT = """You are a planning agent. Given a complex task, create a
detailed step-by-step plan. Do NOT execute any steps. Just produce the plan.
For each step, specify:
- step_id: A unique number starting from 1
- description: What to do in this step, with enough detail to execute it
- depends_on: List of step_ids that must complete before this step
Rules:
- Order steps logically. Later steps should build on earlier ones.
- Identify dependencies explicitly. If step 3 needs the output of step 1,
list step 1 in depends_on.
- Each step should be a single, concrete action or analysis.
- The final step should be "Synthesize findings into a final answer."
Task: {task}
Respond in JSON:
{{"goal": "restated goal", "steps": [{{"step_id": 1, "description": "...",
"depends_on": []}}, ...]}}"""
EXECUTOR_PROMPT = """You are executing a plan step by step. You have access
to tools and can reason about results.
Current plan:
{plan_text}
Completed steps and their results:
{completed_text}
Current step to execute:
Step {step_id}: {step_description}
Execute this step. Use tools if needed. After executing, report:
1. What you did
2. What you learned
3. Whether the remaining plan needs to change (and if so, how)
Respond in JSON:
{{
"action_taken": "What you did",
"findings": "What you learned",
"plan_change_needed": true or false,
"revised_plan": "If plan_change_needed, describe the revision. Otherwise null."
}}"""
def run_plan_and_solve(client, task: str,
tools: dict[str, callable] | None = None) -> dict:
"""Plan-and-Solve agent: plan first, then execute step by step."""
if tools is None:
tools = {}
# Phase 1: Plan
print("=== PHASE 1: PLANNING ===\n")
plan_response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": PLANNER_PROMPT.format(task=task)}],
response_format={"type": "json_object"}
)
plan_data = json.loads(plan_response.choices[0].message.content)
plan = Plan(
goal=plan_data["goal"],
steps=[PlanStep(**s) for s in plan_data["steps"]]
)
print(f"Goal: {plan.goal}")
print(f"Plan: {len(plan.steps)} steps")
for step in plan.steps:
deps = f" (depends on: {step.depends_on})" if step.depends_on else ""
print(f" {step.step_id}. {step.description}{deps}")
# Phase 2: Solve
print("\n=== PHASE 2: EXECUTION ===\n")
completed_steps: list[PlanStep] = []
for step in plan.steps:
# Check dependencies
incomplete_deps = [
d for d in step.depends_on
if not any(cs.step_id == d and cs.status == "completed"
for cs in completed_steps)
]
if incomplete_deps:
print(f"Step {step.step_id}: BLOCKED -- waiting for steps {incomplete_deps}")
step.status = "blocked"
continue
step.status = "in_progress"
print(f"\n--- Executing Step {step.step_id}: {step.description} ---")
# Build the execution prompt
plan_text = "\n".join(
f"{s.step_id}. [{s.status}] {s.description}"
for s in plan.steps
)
completed_text = "\n".join(
f"Step {cs.step_id}: {cs.description}\nResult: {cs.result}"
for cs in completed_steps
) or "(none yet)"
exec_prompt = EXECUTOR_PROMPT.format(
plan_text=plan_text,
completed_text=completed_text,
step_id=step.step_id,
step_description=step.description
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": exec_prompt}],
response_format={"type": "json_object"}
)
result = json.loads(response.choices[0].message.content)
step.result = result.get("findings", "")
step.status = "completed"
completed_steps.append(step)
print(f"Action: {result.get('action_taken', '')[:200]}")
print(f"Findings: {result.get('findings', '')[:200]}")
# Dynamic replanning
if result.get("plan_change_needed"):
revision = result.get("revised_plan", "")
print(f"\n*** PLAN REVISED: {revision[:200]} ***")
# In a full implementation, you would parse the revision
# and update plan.steps accordingly.
# Synthesize final answer
print("\n=== FINAL SYNTHESIS ===")
findings_summary = "\n".join(
f"Step {cs.step_id}: {cs.result}" for cs in completed_steps
)
final_response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": f"""
Based on the following findings from executing a plan, produce a
comprehensive final answer.
Goal: {plan.goal}
Findings:
{findings_summary}
Synthesize these into a clear, well-structured final answer."""}]
)
final_answer = final_response.choices[0].message.content
return {
"plan": plan,
"completed_steps": completed_steps,
"final_answer": final_answer
}
Run it on a complex research task:
result = run_plan_and_solve(client,
"Research whether electric cargo bikes are a viable replacement for "
"delivery vans in urban last-mile logistics. Consider cost, speed, "
"environmental impact, and regulatory factors. Produce a recommendation."
)
print(f"\n{'='*60}")
print(f"FINAL ANSWER:\n{result['final_answer']}")
The agent first produces a plan:
Goal: Determine if electric cargo bikes can replace delivery vans in urban last-mile logistics.
Plan: 6 steps
1. Research cost comparison: cargo bikes vs delivery vans (TCO)
2. Research speed and efficiency in urban settings
3. Research environmental impact (emissions, congestion)
4. Research regulatory landscape (bike lane access, parking, subsidies)
5. Identify limitations and edge cases (weather, distance, cargo volume)
6. Synthesize findings into a recommendation
Then it executes each step, building a structured body of evidence before producing the final recommendation. The plan keeps the agent focused. It does not wander. It does not forget to cover regulatory factors. It produces a thorough, well-structured answer because the plan forced it to.
Dynamic Replanning
The real power of Plan-and-Solve emerges when the plan needs to change. Suppose step 2 reveals that cargo bikes are actually faster than vans in dense urban cores -- a finding that changes the entire cost analysis. The agent flags plan_change_needed: true and revises the plan. Step 3 now incorporates the speed advantage into the cost model. The plan adapts to new information instead of blindly following the original script.
This is what separates Plan-and-Solve from a static workflow. A static workflow executes steps in order regardless of what it learns. A Plan-and-Solve agent treats the plan as a hypothesis that gets tested and refined during execution.
Section 5: Choosing the Right Reasoning Strategy
You now have four reasoning patterns. The question is: which one do you use, and when?
The Decision Framework
Task arrives
|
v
Is the answer a single fact? ---> Direct answer (no special reasoning)
|
v
Does it require external info? ---> ReAct (think, act, observe, repeat)
|
v
Is the solution path obvious? ---> Chain of Thought (step by step)
|
v
Are there multiple valid approaches? ---> Tree of Thought (explore, score, pick)
|
v
Is it a complex, structured task? ---> Plan-and-Solve (plan first, then execute)
Concrete examples:
| Task | Strategy | Why |
|---|---|---|
| "What is the capital of France?" | Direct | Single fact, no reasoning needed |
| "If a shirt costs $25 after a 20% discount, what was the original price?" | CoT | Multi-step math, linear path |
| "What's the current stock price of the company that makes ChatGPT?" | ReAct | Needs external search |
| "Design a pricing strategy for a new SaaS product" | ToT | Multiple valid approaches, need to find best |
| "Research the competitive landscape for electric cargo bikes and produce a report" | Plan-and-Solve | Complex, structured, multi-domain |
The Reasoning Router
You can automate this decision. A reasoning router uses an LLM to classify the task and select the strategy:
from enum import Enum
class ReasoningStrategy(str, Enum):
DIRECT = "direct"
COT = "cot"
REACT = "react"
TOT = "tot"
PLAN_SOLVE = "plan_solve"
ROUTER_PROMPT = """You are a reasoning strategy classifier. Given a task
description, determine which reasoning strategy is most appropriate.
Strategies:
- direct: Simple factual queries, single-step answers, creative writing.
No multi-step reasoning required.
- cot: Multi-step reasoning where the path is linear and clear. Math,
logic, step-by-step analysis. No external information needed.
- react: Tasks that require external information from tools (search,
APIs, databases). The agent needs to find information, then reason about it.
- tot: Creative or strategic problems with multiple valid approaches.
Need to explore alternatives and pick the best one.
- plan_solve: Complex, structured tasks with multiple domains and
dependencies. Need a plan before execution.
Classify this task:
{task}
Respond in JSON:
{{"strategy": "cot", "reasoning": "Why you chose this strategy"}}"""
def route_reasoning_strategy(client, task: str) -> ReasoningStrategy:
"""Classify a task and return the appropriate reasoning strategy."""
response = client.chat.completions.create(
model="gpt-4o-mini", # Fast, cheap model for classification
messages=[{"role": "user", "content": ROUTER_PROMPT.format(task=task)}],
response_format={"type": "json_object"}
)
result = json.loads(response.choices[0].message.content)
return ReasoningStrategy(result["strategy"])
Now your agent can select its own reasoning strategy:
def solve_with_best_strategy(client, task: str, tools: dict) -> str:
"""Route a task to the best reasoning strategy and solve it."""
strategy = route_reasoning_strategy(client, task)
print(f"Selected strategy: {strategy.value}")
match strategy:
case ReasoningStrategy.DIRECT:
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": task}]
)
return response.choices[0].message.content
case ReasoningStrategy.COT:
result = run_cot_agent(client, task)
return result.final_answer
case ReasoningStrategy.REACT:
agent = ReActAgent(client=client, tools=tools)
result = agent.run(task)
return result["answer"]
case ReasoningStrategy.TOT:
best = run_tot_agent(client, task)
return best.content
case ReasoningStrategy.PLAN_SOLVE:
result = run_plan_and_solve(client, task, tools)
return result["final_answer"]
Cost Considerations
Reasoning is not free. Each strategy has a different cost profile:
| Strategy | LLM Calls (typical) | Token Multiplier | When to Use |
|---|---|---|---|
| Direct | 1 | 1x | Simple queries |
| CoT | 1-3 | 2-3x | Linear multi-step |
| ReAct | 3-10 | 3-5x | External info needed |
| Plan-and-Solve | 5-15 | 3-5x | Complex structured tasks |
| Tree of Thought | 10-50 | 10-50x | Creative/strategic, high-value |
The token multiplier is relative to a direct answer. A CoT response is typically 2-3x longer than a direct answer because it includes the reasoning steps. A ReAct trace includes tool calls and observations. A ToT run includes multiple branches and evaluations.
These costs are not theoretical. If a direct answer costs $0.001, a ToT exploration of the same problem costs $0.01-$0.05. Run that 10,000 times and the difference is $10 versus $100-$500. Choose your strategy based on the value of the answer, not just the elegance of the approach.
The best reasoning strategy is the cheapest one that produces an acceptable answer. Start with CoT. If CoT fails, try ReAct. If ReAct produces shallow results, try Plan-and-Solve. Reserve ToT for problems where the cost of a wrong answer exceeds the cost of the extra compute.
Section 6: The Complete Reasoning Agent
You now have all four strategies. Let us assemble them into a single agent that can select and execute any strategy based on the task.
import json
from typing import Any, Callable
from dataclasses import dataclass, field
from enum import Enum
class Strategy(str, Enum):
DIRECT = "direct"
COT = "cot"
REACT = "react"
TOT = "tot"
PLAN_SOLVE = "plan_solve"
@dataclass
class ReasoningAgent:
"""An agent that can use any of four reasoning strategies."""
client: Any
model: str = "gpt-4o"
router_model: str = "gpt-4o-mini"
tools: dict[str, Callable] = field(default_factory=dict)
max_steps: int = 15
def solve(self, task: str) -> dict:
"""Select a strategy and solve the task."""
strategy = self._classify_task(task)
print(f"Strategy: {strategy.value} | Task: {task[:80]}...")
match strategy:
case Strategy.DIRECT:
return self._solve_direct(task)
case Strategy.COT:
return self._solve_cot(task)
case Strategy.REACT:
return self._solve_react(task)
case Strategy.TOT:
return self._solve_tot(task)
case Strategy.PLAN_SOLVE:
return self._solve_plan_solve(task)
def _classify_task(self, task: str) -> Strategy:
"""Use a fast model to classify the task."""
response = self.client.chat.completions.create(
model=self.router_model,
messages=[{"role": "user", "content": ROUTER_PROMPT.format(task=task)}],
response_format={"type": "json_object"}
)
return Strategy(json.loads(response.choices[0].message.content)["strategy"])
def _solve_direct(self, task: str) -> dict:
response = self.client.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": task}]
)
return {"strategy": "direct", "answer": response.choices[0].message.content}
def _solve_cot(self, task: str) -> dict:
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": COT_SYSTEM_PROMPT},
{"role": "user", "content": task}
],
response_format={"type": "json_object"}
)
data = json.loads(response.choices[0].message.content)
return {
"strategy": "cot",
"steps": data.get("steps", []),
"answer": data.get("final_answer", "")
}
def _solve_react(self, task: str) -> dict:
agent = ReActAgent(
client=self.client,
model=self.model,
tools=self.tools,
max_steps=self.max_steps
)
result = agent.run(task)
return {
"strategy": "react",
"trace": [{
"thought": t.thought,
"action": t.action,
"observation": t.observation
} for t in result["trace"]],
"answer": result["answer"]
}
def _solve_tot(self, task: str) -> dict:
best = run_tot_agent(self.client, task)
return {
"strategy": "tot",
"best_path_score": best.score,
"answer": best.content
}
def _solve_plan_solve(self, task: str) -> dict:
result = run_plan_and_solve(self.client, task, self.tools)
return {
"strategy": "plan_solve",
"plan": [{"id": s.step_id, "description": s.description,
"status": s.status} for s in result["plan"].steps],
"answer": result["final_answer"]
}
This is about 100 lines. It is not production code -- it needs error handling, retry logic, logging, and cost tracking. But it is the complete skeleton of a reasoning agent. Every production reasoning system you build will be a variation on this pattern.
Running the Comparison
Let us run the same task through different strategies and compare:
agent = ReasoningAgent(client=client, tools={"search_web": search_web})
tasks = [
"What is 15% of 87?",
"If a train leaves at 60 mph and another at 80 mph from 800 miles apart, "
"when do they meet?",
"What's the market cap of the company that makes ChatGPT?",
"Design a feature to increase user retention for a meditation app.",
"Research whether remote work increases or decreases productivity. "
"Consider studies from 2020-2026. Produce a balanced analysis."
]
for task in tasks:
result = agent.solve(task)
strategy = result["strategy"]
answer_preview = str(result.get("answer", ""))[:120]
print(f"\nTask: {task[:60]}...")
print(f" Strategy: {strategy}")
print(f" Answer: {answer_preview}...")
Expected output:
Strategy: direct | Task: What is 15% of 87?...
Strategy: direct
Answer: 15% of 87 is 13.05....
Strategy: cot | Task: If a train leaves at 60 mph and another at 80 mph...
Strategy: cot
Answer: Step 1: Combined speed = 60 + 80 = 140 mph. Step 2: Time = 800/140...
Strategy: react | Task: What's the market cap of the company that makes ChatGPT?...
Strategy: react
Answer: OpenAI is a private company and does not have a public market cap...
Strategy: tot | Task: Design a feature to increase user retention for a...
Strategy: tot
Answer: Based on exploring multiple approaches, the most promising feature...
Strategy: plan_solve | Task: Research whether remote work increases or decreases...
Strategy: plan_solve
Answer: After analyzing studies from 2020-2026, the evidence is mixed...
The agent selected a different strategy for each task. Simple math got a direct answer. The train problem got CoT. The market cap question triggered ReAct (it needs a search). The feature design triggered ToT (multiple approaches to explore). The research task triggered Plan-and-Solve (complex, structured, multi-domain).
This is the meta-reasoning pattern in action: use a fast, cheap model to decide how the smart, expensive model should think.
The Turn
You now have a toolkit of reasoning strategies. You understand that "the model is smart" is not enough -- how you structure the model's thinking determines the quality of the output.
Chain of Thought forces the model to show its work. ReAct grounds reasoning in tool results. Tree of Thought explores multiple paths and picks the best. Plan-and-Solve builds a map before taking the first step. Each strategy is a different way of structuring cognition. Each has a cost profile. Each has a class of problems it solves best.
The agent is not just reacting anymore. It is planning, exploring, and reasoning systematically. You can look at a task and know which strategy to apply. Better: your agent can look at a task and decide for itself.
This is the difference between a script and a system. A script follows instructions. A system chooses its approach.
Close
You have now built agents from scratch -- the loop, the tools, the memory, the reasoning. You understand the fundamentals. You can look at any agent framework and see the patterns underneath: the loop, the tool definitions, the prompt templates, the reasoning strategies. You know what the framework is doing because you have built it yourself.
But you have been writing everything by hand. Every loop. Every tool definition. Every message format. Every retry. Every parser. This is the right way to learn. It is not the right way to build every project.
In the next chapter, you will discover the ecosystem of frameworks that can accelerate your development. You will learn what LangChain, LangGraph, CrewAI, and the Anthropic SDK actually do -- and, more importantly, what they do to your code. You will learn when a framework saves you months and when it costs you control. You will learn to read framework source code and recognize the patterns you already know.
The frameworks are not magic. They are the patterns from Chapters 4 through 7, packaged and productized. You are about to understand them from the inside out.
What you built in this chapter:
| Component | What It Does |
|---|---|
| Chain of Thought agent | Forces step-by-step reasoning before answering |
| ReAct agent | Interleaves reasoning with tool use in a loop |
| Tree of Thought agent | Explores multiple solution paths, scores them, picks the best |
| Plan-and-Solve agent | Creates a plan first, then executes step by step with replanning |
| Reasoning router | Classifies tasks and selects the optimal strategy |
| Complete reasoning agent | Combines all four strategies with automatic selection |
Key takeaways:
- CoT is the default. Add "think step by step" to any prompt for multi-step problems.
- ReAct is the minimum for any agent that uses tools. Thought -> Action -> Observation -> repeat.
- ToT is for problems where the best path is not obvious. Expensive but powerful.
- Plan-and-Solve is for complex, structured tasks. The plan is working memory.
- The best strategy is the cheapest one that produces an acceptable answer.
- A reasoning router lets the agent choose its own strategy based on the task.
- Every strategy is a different way of structuring the same underlying loop.