Chapter 02 · LLM Foundations
Most LLM explainers start with "Attention is all you need." This one starts with a curl command.
curl https://api.anthropic.com/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{
"model": "claude-sonnet-4-20250514",
"max_tokens": 1024,
"messages": [
{"role": "user", "content": "What is 7 * 13?"}
]
}'
The response comes back:
{
"id": "msg_01Xq8Y7zJ3kL5mN2pR4vW6tH",
"type": "message",
"role": "assistant",
"content": [
{
"type": "text",
"text": "7 * 13 = 91"
}
],
"model": "claude-sonnet-4-20250514",
"stop_reason": "end_turn",
"usage": {
"input_tokens": 15,
"output_tokens": 12
}
}
That's it. That's the entire interface. Everything in this book -- every agent, every tool, every multi-agent swarm, every retrieval-augmented pipeline -- is built on this single primitive: send text, receive text. You send a JSON payload with an array of messages. You get back a JSON payload with a response. There is no state. There is no memory. There is no magic.
If you understand this curl command, you understand the atomic unit of every agent system you will ever build. The rest is engineering.
What This Chapter Covers (and What It Doesn't)
This chapter covers the 20% of LLM knowledge that matters for 80% of agent building. You will not find attention mechanisms here. You will not find backpropagation, transformer architecture diagrams, or loss function derivations. Those are interesting. They are not useful when your agent is hallucinating tool calls at 2 AM and you need to figure out why.
What you will find: tokens, context windows, temperature, system prompts, the chat completion API, and model selection. These are the knobs you actually turn when building and debugging agents. Master these six concepts and you will understand why your agent behaves the way it does -- and how to fix it when it doesn't.
1. Tokens: The Real Currency
Tokens are not words. Tokens are not characters. Tokens are subword units -- the fragments a language model actually processes.
Take the sentence "The quick brown fox jumps over the lazy dog." You might assume the model sees nine words. It doesn't. Feed it through a tokenizer and you get something like this:
"The quick brown fox" -> ["The", " quick", " brow", "n", " fox"]
Notice what happened. "The" is a common word, so it gets its own token. " quick" has a leading space because the tokenizer treats spaces as part of the token. "brown" got split into " brow" and "n" because the tokenizer's vocabulary doesn't contain "brown" as a single unit. " fox" is common enough to be one token.
This is subword tokenization -- specifically Byte-Pair Encoding (BPE), the algorithm used by most modern LLMs. The tokenizer builds a vocabulary of common subword units from its training data. Common words get their own tokens. Rare words get split into pieces. This is why LLMs can handle typos, made-up words, and code: everything decomposes into known subword fragments.
Why Tokens Matter for Agents
Every tool call your agent makes consumes tokens. Every retrieved document chunk consumes tokens. Every turn of conversation history consumes tokens. Tokens are the unit of cost, the unit of latency, and the unit of capacity. You cannot build production agents without understanding token economics.
Input tokens are the tokens you send to the model -- your prompt, conversation history, tool definitions, retrieved documents. Output tokens are the tokens the model generates in response. They are priced differently, and for good reason: output tokens require the model to run forward through every layer for every single token it generates, one at a time. Input tokens can be processed in parallel. This is why output tokens typically cost 3-5x more than input tokens.
Here is a practical token-counting function. You call this before making an API request so you know what you're spending:
import tiktoken
def count_tokens(text: str, model: str = "gpt-4") -> int:
"""Count the number of tokens in a text string for a given model."""
try:
encoding = tiktoken.encoding_for_model(model)
except KeyError:
encoding = tiktoken.get_encoding("cl100k_base")
return len(encoding.encode(text))
def estimate_cost(
input_tokens: int,
output_tokens: int,
model: str = "gpt-4o"
) -> dict:
"""Estimate the cost of an API call based on token counts."""
pricing = {
"gpt-4o": {"input": 2.50, "output": 10.00}, # per 1M tokens
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"claude-sonnet-4-20250514": {"input": 3.00, "output": 15.00},
"claude-haiku-4-20250514": {"input": 0.80, "output": 4.00},
}
p = pricing.get(model, pricing["gpt-4o-mini"])
input_cost = (input_tokens / 1_000_000) * p["input"]
output_cost = (output_tokens / 1_000_000) * p["output"]
return {
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"input_cost": round(input_cost, 4),
"output_cost": round(output_cost, 4),
"total_cost": round(input_cost + output_cost, 4),
}
# Example: a conversation with 2,000 input tokens and 500 output tokens
print(estimate_cost(2000, 500, "gpt-4o"))
# {'input_tokens': 2000, 'output_tokens': 500, 'input_cost': 0.005, 'output_cost': 0.005, 'total_cost': 0.01}
Run this before every agent loop iteration. Token costs look trivial at small scale -- fractions of a cent per call. They are not trivial when your agent makes 50 calls in a single task, across thousands of tasks per day. A $0.01 call repeated 100,000 times is $1,000. Token counting is not premature optimization. It is basic operational awareness.
The Context Window
Every model has a maximum context window -- the total number of tokens it can accept in a single request. Exceed it and one of three things happens: the API rejects your request with an error, the model silently truncates the beginning of your input, or the model's performance degrades as it loses track of information.
Context window sizes as of mid-2026:
| Model | Context Window |
|---|---|
| Claude Opus 4 | 200,000 tokens |
| Claude Sonnet 4 | 200,000 tokens |
| GPT-4o | 128,000 tokens |
| GPT-4o-mini | 128,000 tokens |
| Gemini 2.5 Pro | 1,000,000+ tokens |
| Llama 4 (open-source) | 128,000 tokens |
A 200K context window sounds enormous. It is roughly 150,000 words, or about 500 pages of text. But in an agent system, that window fills fast. A single tool call with a large result can consume 10,000 tokens. Ten turns of conversation with tool results and you are at 100,000. Add a system prompt, tool definitions, and a few retrieved documents, and you are fighting for every token.
The context window is your agent's working memory. Everything the agent "knows" in the moment must fit inside it. There is no long-term memory unless you build one. There is no persistent state unless you maintain it. The model wakes up fresh on every API call with no knowledge of anything that happened before.
2. The Context Window: Your Agent's Working Memory
The context window is the single most important constraint in agent design. It is not just a technical limit. It is the boundary of what your agent can reason about at any given moment.
What Goes In
Every API call packs the following into the context window:
-
System prompt. Your instructions for the agent's behavior, constraints, and output format. This is always present, consuming a fixed overhead on every call.
-
Conversation history. Every previous user message, every previous assistant response, every tool call and its result. This grows linearly with conversation length.
-
Tool definitions. The JSON schemas describing available tools, their parameters, and their descriptions. More tools = more tokens consumed before the conversation even starts.
-
Retrieved documents. If your agent uses RAG (retrieval-augmented generation), every chunk of retrieved text consumes context. Ten chunks of 500 tokens each is 5,000 tokens gone.
-
The current user message. Whatever the user just asked.
Add these up. A typical agent setup might look like:
System prompt: 1,500 tokens
Tool definitions: 2,000 tokens (10 tools, ~200 tokens each)
Conversation history: 8,000 tokens (6 turns with tool results)
Retrieved documents: 4,000 tokens (8 chunks at 500 tokens each)
Current message: 200 tokens
-------------------------------------------
Total: 15,700 tokens
That is 15,700 tokens before the model has generated a single word. In a 128K context window, you have room. But run that agent for 30 turns with heavy tool use and you will hit the limit.
The Context Budget
You are always fighting to fit everything into the context window. This is the context budget -- the finite number of tokens you can spend on each API call. Every token you spend on conversation history is a token you cannot spend on retrieved documents. Every token you spend on tool definitions is a token you cannot spend on reasoning.
Prioritization is critical. You must decide what the model needs to see right now and what can be summarized, truncated, or omitted. This is not a one-time decision. It is a continuous process that runs on every turn of every conversation.
Context Window Strategies
Three strategies dominate production agent systems:
Sliding window. Keep the last N turns of conversation and drop everything older. Simple, predictable, and brutal. The agent forgets anything that happened before the window. Works for short-lived tasks. Fails for long-running conversations where early context matters.
Summarization. Before the context fills up, ask the model (or a cheaper model) to summarize the conversation so far. Replace the full history with the summary. The agent retains the gist but loses details. This is a lossy compression algorithm applied to conversation. It works surprisingly well for most use cases.
Selective retention. Mark certain messages as "critical" and never drop them. Everything else is eligible for eviction. This requires you to build a retention policy -- a set of rules about what stays and what goes. The policy is domain-specific. A customer support agent might retain the customer's account details and the current issue but drop resolved sub-issues. A coding agent might retain the current file and error messages but drop earlier debugging dead ends.
The "Lost in the Middle" Problem
There is a well-documented phenomenon in LLM research: models pay the most attention to the beginning and end of their context window. Information placed in the middle gets ignored or underweighted. This is the "lost in the middle" problem, and it has direct practical consequences for agent design.
If you put critical instructions in the middle of a long system prompt, the model may miss them. If a retrieved document containing the answer is the fifth of ten chunks, the model may overlook it. If a tool result that the agent needs is buried in conversation history, the agent may act as if it never saw it.
The fix: Put critical information at the start or the end of the context. System prompts go at the beginning. The most important retrieved documents go at the beginning or the end. If you need the agent to act on a specific piece of information, surface it in the most recent user message -- the very end of the context, where attention is highest.
Design rule: The beginning and end of the context window are prime real estate. The middle is a swamp. Put your most important content at the poles.
3. Temperature, Top-p, and the Creativity Dial
An LLM does not pick the single "correct" next token. It produces a probability distribution over its entire vocabulary -- a list of every possible next token, each with a score representing how likely the model thinks it is. The word "The" might get 0.15 probability. "A" might get 0.08. "It" might get 0.03. And 50,000 other tokens split the remaining probability mass.
How you sample from this distribution determines everything about the model's output style.
Temperature
Temperature is a scaling factor applied to the logits (raw scores) before they are converted to probabilities. At temperature 0, the model always picks the highest-probability token. The same input always produces the same output. At temperature 1, the distribution is unchanged. At temperature 2, the distribution flattens -- low-probability tokens become more likely, and the output becomes more random.
Temperature 0: "The capital of France is Paris."
Temperature 0.5: "The capital of France is Paris, a city known for its art and cuisine."
Temperature 1.0: "Paris serves as France's capital, a bustling metropolis of culture and history."
Temperature 1.5: "France's storied capital, Paris, stands as a beacon of art, romance, and revolution."
Same question. Same model. Radically different outputs. At temperature 0, the model gives you the shortest, most probable answer. At higher temperatures, it explores less likely phrasings, adds detail, and varies sentence structure.
Why Temperature Matters for Agents
Different parts of your agent system need different temperature settings:
Tool selection: temperature 0. When the model decides which tool to call, you want determinism. The same input should produce the same tool choice every time. A non-deterministic tool selector is a bug, not a feature.
Structured output: temperature 0-0.2. When the model generates JSON, function call arguments, or any structured data, keep temperature low. Creativity in JSON structure means malformed JSON.
Code generation: temperature 0-0.2. Code is deterministic by nature. You want the most probable correct solution, not a creative interpretation.
Conversational responses: temperature 0.3-0.7. Some variation makes the agent feel natural rather than robotic. But too much variation and it becomes unpredictable.
Creative generation: temperature 0.7-1.0. Brainstorming, writing assistance, idea generation -- this is where higher temperature shines.
The rule for agents: Most of your agent's work should run at temperature 0-0.3. Crank it higher only when you explicitly want variation. The cost of unpredictability in an agent system is higher than the cost of repetitiveness.
Top-p (Nucleus Sampling)
Top-p is an alternative to temperature. Instead of scaling the entire distribution, top-p truncates it: the model considers only the smallest set of tokens whose cumulative probability exceeds P. If top-p is 0.9 and the top three tokens have probabilities 0.5, 0.3, and 0.12 (cumulative 0.92), the model samples only from those three tokens. Everything else is zeroed out.
Top-p is dynamic. When the model is confident (one token has 0.95 probability), top-p = 0.9 uses only that token. When the model is uncertain (many tokens have similar probabilities), top-p = 0.9 includes more options. This makes top-p more adaptive than temperature alone.
In practice, most API providers let you set both. The standard advice: set temperature to 0.7-1.0 for creative tasks and top-p to 0.9-0.95 as a safety cap. For agent work, set temperature to 0 and ignore top-p entirely -- deterministic sampling makes top-p irrelevant.
The Tradeoff
Low temperature is reliable but repetitive. Your agent will give the same answer to the same question every time. This is exactly what you want for tool calls and structured output. It is boring for conversation.
High temperature is creative but unpredictable. Your agent might say something brilliant. It might also say something nonsensical. In a production system where the agent is making API calls, modifying data, or interacting with users, unpredictability is dangerous.
Start at temperature 0. Increase only when you have a specific reason and you have tested the behavior at the higher setting.
4. System Prompts: The Agent's Constitution
The system prompt is the first message in the messages array. It sets the rules. The model is trained to treat system instructions with higher authority than user messages -- a user cannot override the system prompt by saying "ignore your previous instructions." (In practice, determined users can jailbreak this. But the model's training gives the system prompt structural priority.)
System Prompt vs. User Message
A user message says "do this." A system prompt says "you ARE this." The distinction matters. The system prompt defines the agent's identity, capabilities, constraints, and output format. It persists across all turns of the conversation. The user message is a single request within that framework.
Think of the system prompt as the agent's operating system. It defines what the agent IS. The user messages are applications running on that OS. You can change the applications without reinstalling the OS.
Anatomy of a Good System Prompt for Agents
A production system prompt for an agent has five components:
-
Role. Who the agent is and what it does. One sentence. Be specific.
-
Capabilities. What the agent can do. List the tools it has access to and what each tool does. Be explicit about limitations.
-
Constraints. What the agent must NOT do. Hard boundaries. "Never make up information. If you don't know, say you don't know." "Never call the delete tool without confirmation."
-
Output format. How the agent should structure its responses. "Always respond in JSON." "Use markdown for code blocks." "Keep responses under 200 words."
-
Tool-use instructions. When and how to use tools. "Call the search tool before answering any factual question." "If the user asks about their account, call get_user_data first."
Here is a real system prompt for a simple research agent:
RESEARCH_AGENT_SYSTEM_PROMPT = """You are a research assistant agent. Your job is to answer
user questions by searching the web and synthesizing information from multiple sources.
CAPABILITIES:
- You have access to a web_search tool that returns search results with titles, snippets,
and URLs.
- You have access to a fetch_page tool that retrieves the full text of a web page given
a URL.
- You can call multiple tools in sequence to gather information before answering.
CONSTRAINTS:
- NEVER fabricate information. If search results are insufficient, say so explicitly.
- ALWAYS cite your sources. Include URLs for every factual claim.
- If sources contradict each other, present both sides and note the disagreement.
- Do not provide medical, legal, or financial advice. Redirect users to professionals.
OUTPUT FORMAT:
- Structure answers with a clear summary first, then supporting details.
- Use numbered citations [1], [2], etc. that correspond to a sources list at the end.
- Keep answers concise but complete. Prefer bullet points for lists.
TOOL USE:
- Call web_search before answering any question that requires current or factual
information.
- If search snippets are insufficient, call fetch_page on the most relevant results.
- Do not call fetch_page on more than 3 URLs per question unless the user asks for
deeper research.
- If a tool returns an error, try an alternative approach before giving up.
"""
This is not a toy example. This is the level of specificity you need for an agent to behave reliably. Vague system prompts produce vague behavior. Specific system prompts produce specific behavior. The system prompt is your primary lever for controlling agent behavior -- invest time in writing it well.
The System Prompt as the Agent's Operating System
When you change the system prompt, you change what the agent IS. A research agent with the prompt above becomes a customer support agent if you swap the role, tools, and constraints. The underlying model is the same. The behavior is entirely different.
This is the power and the danger of system prompts. A small change in wording can produce a large change in behavior. "You are a helpful assistant" produces a very different agent than "You are a skeptical fact-checker who questions every claim." Test your system prompts. Iterate on them. Treat them as code -- because in an agent system, they are code.
5. The Chat Completion API: Your Only Interface
Every LLM provider exposes a chat completion API. The details vary, but the structure is universal: you send an array of messages, you get back a response. That is the entire interface. There is no session object. There is no memory management. There is no conversation state. The API is stateless.
The Messages Array
The messages array is a list of objects, each with a role and content. The roles are:
- system: Instructions that define the assistant's behavior. Typically the first message.
- user: A message from the human user. Questions, requests, follow-ups.
- assistant: A response from the model. You must include previous assistant responses to maintain conversation continuity.
- tool: The result of a tool call. Contains the tool_call_id and the output.
A multi-turn conversation looks like this:
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
{"role": "assistant", "content": "The capital of France is Paris."},
{"role": "user", "content": "What is its population?"},
]
The model sees the entire array. It has no memory of the previous API call. It only knows what is in the messages array right now. If you omit the first assistant response, the model has no idea it already answered the capital question. It will answer again as if the question is new.
The Statelessness Problem
This is the single most important thing to internalize about LLM APIs: the model has no memory between API calls. Each call is independent. The model does not remember you. It does not remember your conversation. It does not remember what it said three turns ago.
YOU maintain the conversation array. YOU append each new user message. YOU append each assistant response. YOU manage the growing list of messages. If your server crashes and you lose the messages array, the model has amnesia. The conversation is gone.
This is not a bug. It is the architecture. And it is empowering once you understand it: because YOU control the messages array, YOU decide what the model remembers. You can edit history. You can inject context. You can remove irrelevant turns. The model's "memory" is entirely under your control.
Streaming vs. Non-Streaming
Non-streaming mode: you send the request, wait, and receive the complete response all at once. Simple to parse. Higher perceived latency because the user sees nothing until the entire response is generated.
Streaming mode: tokens arrive one at a time as they are generated. The user sees the response build in real time. Lower perceived latency. Harder to parse because you must reassemble tokens and handle partial data.
For agents, non-streaming is usually the right choice. Your agent needs the complete response to parse tool calls, extract structured data, or decide the next action. Streaming is for user-facing chat interfaces where perceived speed matters more than parse simplicity.
A Complete Chat Completion Function
Here is a production-ready function that calls an LLM API with full conversation history. It handles both streaming and non-streaming, counts tokens, and returns structured results:
import json
import requests
from typing import Optional
def chat_completion(
messages: list[dict],
model: str = "claude-sonnet-4-20250514",
system: Optional[str] = None,
temperature: float = 0.0,
max_tokens: int = 4096,
stream: bool = False,
api_key: Optional[str] = None,
) -> dict:
"""
Send a chat completion request to the Anthropic API.
Args:
messages: List of message dicts with 'role' and 'content'.
Roles: 'user', 'assistant'. 'system' is passed separately.
model: Model identifier string.
system: System prompt (Anthropic accepts this as a top-level param).
temperature: Sampling temperature (0.0 = deterministic).
max_tokens: Maximum tokens in the response.
stream: Whether to stream the response.
api_key: Anthropic API key. Falls back to ANTHROPIC_API_KEY env var.
Returns:
Dict with keys: 'content', 'model', 'usage', 'stop_reason', 'raw_response'.
"""
import os
api_key = api_key or os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
raise ValueError("API key required. Set ANTHROPIC_API_KEY or pass api_key.")
headers = {
"x-api-key": api_key,
"anthropic-version": "2023-06-01",
"content-type": "application/json",
}
payload = {
"model": model,
"max_tokens": max_tokens,
"temperature": temperature,
"messages": messages,
"stream": stream,
}
if system:
payload["system"] = system
response = requests.post(
"https://api.anthropic.com/v1/messages",
headers=headers,
json=payload,
timeout=120,
)
if response.status_code != 200:
raise RuntimeError(
f"API error {response.status_code}: {response.text}"
)
body = response.json()
# Extract text content from the response
content_blocks = body.get("content", [])
text_content = "".join(
block["text"] for block in content_blocks if block["type"] == "text"
)
return {
"content": text_content,
"model": body.get("model"),
"usage": body.get("usage", {}),
"stop_reason": body.get("stop_reason"),
"raw_response": body,
}
# --- Usage example: multi-turn conversation ---
conversation = [
{"role": "user", "content": "What is the capital of France?"},
]
# First turn
result = chat_completion(
messages=conversation,
system="You are a concise geography assistant. Answer in one sentence.",
temperature=0.0,
)
print(result["content"])
# "The capital of France is Paris."
# Append the assistant's response to maintain history
conversation.append({"role": "assistant", "content": result["content"]})
# Second turn -- the model "remembers" because we maintained the array
conversation.append({"role": "user", "content": "What is its population?"})
result = chat_completion(messages=conversation, temperature=0.0)
print(result["content"])
# "The population of Paris is approximately 2.1 million within the city limits."
print(f"Tokens used: {result['usage']}")
# Tokens used: {'input_tokens': 58, 'output_tokens': 22}
Study this function. This is the engine of every agent you will build in this book. Every agent loop, every tool call, every multi-step reasoning chain -- it all runs through a function like this one. The messages array is the agent's memory. The system prompt is the agent's identity. The temperature is the agent's reliability dial. Everything else is built on top.
6. Choosing a Model
Not all models are created equal. The model you choose determines your agent's reasoning ability, speed, cost, and context capacity. There is no single best model. There is only the right model for your specific task and budget.
The Decision Matrix
You are trading off four dimensions:
| Dimension | What It Means | Why It Matters |
|---|---|---|
| Reasoning ability | How well the model handles complex logic, multi-step problems, and nuanced instructions | Determines whether your agent can actually solve the problem |
| Speed (latency) | Time to first token and tokens per second | Determines whether your agent feels responsive or sluggish |
| Cost | Dollars per million tokens (input and output) | Determines whether your agent is economically viable at scale |
| Context window | Maximum tokens per request | Determines how much information your agent can consider at once |
No model wins all four. Frontier models (Claude Opus, GPT-4, Gemini Ultra) maximize reasoning ability at the cost of speed and price. Fast models (Claude Haiku, GPT-4o-mini) maximize speed and minimize cost at the cost of reasoning depth. Open-source models (Llama, Mistral) give you control and data privacy at the cost of convenience and sometimes capability.
When to Use Which
Frontier models (Claude Opus 4, GPT-4, Gemini 2.5 Pro): Use when the task requires genuine reasoning -- multi-step planning, complex code generation, nuanced analysis, or anything where a wrong answer is expensive. These are your "brain" models. They are slower and more expensive. Use them when quality matters more than speed or cost.
Fast models (Claude Sonnet 4, GPT-4o, Claude Haiku 4): Use for the majority of agent work. Sonnet 4 and GPT-4o sit in a sweet spot: strong reasoning at reasonable cost and speed. Haiku 4 and GPT-4o-mini are for high-volume, low-complexity tasks -- classification, extraction, simple routing decisions.
Open-source models (Llama 4, Mistral, DeepSeek): Use when you need data privacy (no data leaves your infrastructure), when you need guaranteed uptime (no API dependency), or when you are operating at massive scale where API costs become prohibitive. The tradeoff: you manage the infrastructure.
The Router Pattern
The most cost-effective agent architectures do not use a single model. They use a router: a fast, cheap model handles simple decisions and escalates to a powerful model when the task is hard.
User message -> Haiku (classify: simple or complex?)
-> Simple -> Haiku handles it ($0.0005)
-> Complex -> Sonnet handles it ($0.01)
The router itself is an LLM call -- a cheap one. You ask Haiku: "Is this a simple factual question or a complex multi-step task? Answer 'simple' or 'complex'." Based on the answer, you route to the appropriate model. This pattern can reduce costs by 60-80% while maintaining quality on the tasks that need it.
You will implement this pattern in Chapter 8. For now, internalize the principle: not every task needs a frontier model. Most agent tasks are simple. Use cheap models for cheap work.
Model Versioning
Model names change. Providers release new versions, deprecate old ones, and sometimes change behavior without changing the name. Pin your model versions in production.
Do not use model="claude-sonnet" and assume it will always point to the same thing. Use model="claude-sonnet-4-20250514". When a new version ships, test it before switching. Model upgrades are not transparent -- a new version might be better at reasoning but worse at following your specific system prompt format. Test before you deploy.
The Turn
You now understand what an LLM actually is. Not a brain. Not a database. Not a reasoning engine in the philosophical sense. An LLM is a stateless function that takes a sequence of tokens and returns a probability distribution over the next token. That is the entire thing.
Everything else -- conversation, memory, tool use, reasoning chains, multi-agent coordination -- is engineering built on top of this primitive. The model does not remember you. The model does not know what it said three turns ago. The model does not have goals, intentions, or understanding. It has a context window, a temperature setting, and a system prompt. You provide the rest.
This is not a limitation. It is a design property. Because the model is stateless, you control its state. Because the model has no memory, you decide what it remembers. Because the model is a function, you can compose it with other functions to build systems more capable than any single model call.
You do not need to understand transformer architecture to build agents. You do not need to know how attention heads work. You need to understand the API contract: send tokens, receive tokens. The curl command at the start of this chapter is the entire interface. Master it, and you can build anything.
Close
Now you know what the model does and how to talk to it. You understand tokens, context windows, temperature, system prompts, and the stateless API that underlies everything. You can count tokens, estimate costs, and choose the right model for the job.
But knowing how to talk to the model is not the same as knowing how to talk to it WELL. The difference between a prompt that works once and a prompt that works every time is the difference between a prototype and a production system. The difference between "write me a function" and an agent that reliably generates correct, tested, documented code is prompt engineering -- real prompt engineering, not the cargo-cult tricks that fall apart the moment you need reliability.
In the next chapter, you will learn prompt engineering patterns that survive production. Not "think step by step." Not "you are an expert." Those work in demos. You will learn what works when your agent is handling thousands of requests and a single bad response costs real money.
Next: Chapter 3 -- Prompt Engineering for Agents