Chapter 17 · Computer Use & Browser Agents

Part of Part IV · Production

"Your agent can now see. And click. And type. This chapter is about agents that use computers the way humans do."

---

You need to extract data from a legacy web application. No API. Just HTML tables, multi-page forms, and JavaScript-rendered content. Traditional scraping fails — the anti-bot protection blocks your requests, the dynamic content doesn't load, the multi-step workflow breaks.

A computer-use agent doesn't care. It sees the screen, understands the layout, clicks through pages, fills forms, and extracts the data — just like a human would. It's slower and more expensive than an API call, but it works with ANYTHING.

---

Section 1: What Is Computer Use?

Computer use is the frontier of agentic AI. Instead of calling structured APIs, the agent receives screenshots and decides where to click and what to type.

The action space:

Action	Description
mouse_move(x, y)	Move cursor to coordinates
left_click	Click at current position
right_click	Right-click at current position
double_click	Double-click at current position
drag(start, end)	Click and drag
type(text)	Type text at current focus
key(combination)	Press key combination (e.g., "Enter", "Ctrl+C")
screenshot	Capture current screen
scroll(direction, amount)	Scroll up/down

How it works:

1. Take a screenshot
2. Send screenshot to the model with instructions
3. Model returns an action: "click at (342, 518)"
4. Execute the action on the computer
5. Take a new screenshot
6. Repeat until task complete

The key difference from tool use: tools are structured APIs. Computer use is unstructured visual understanding. The agent must interpret pixels, not JSON.

---

Section 2: The Technology Stack

Anthropic Computer Use (most mature):

import anthropic

client = anthropic.Anthropic()

computer_tool = {
    "type": "computer_20241022",
    "name": "computer",
    "display_width_px": 1024,
    "display_height_px": 768,
    "display_number": 1,
}

response = client.beta.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=4096,
    tools=[computer_tool],
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Navigate to github.com and find the top trending Python repository."},
            {"type": "image", "source": screenshot},
        ]
    }],
)

# Claude returns a computer action
if response.stop_reason == "tool_use":
    for block in response.content:
        if block.type == "tool_use" and block.name == "computer":
            action = block.input["action"]
            coordinate = block.input.get("coordinate")
            text = block.input.get("text")
            # Execute the action...

Other options:

• Playwright + AI: Use Playwright for browser automation, add AI for visual understanding and decision-making
• Browserbase: Managed browser infrastructure with AI agent support
• Open-source: OSWorld and WebArena provide benchmarks and reference implementations

---

Section 3: Building a Computer-Use Agent

The complete computer-use loop:

import base64
from io import BytesIO
from PIL import Image

class ComputerUseAgent:
    def __init__(self, display_width: int = 1024, display_height: int = 768):
        self.display_width = display_width
        self.display_height = display_height
        self.client = anthropic.Anthropic()

    def run(self, task: str, max_steps: int = 50) -> dict:
        messages = [{
            "role": "user",
            "content": [{"type": "text", "text": task}]
        }]

        for step in range(max_steps):
            # Take screenshot
            screenshot = self._capture_screenshot()
            screenshot_b64 = self._encode_image(screenshot)

            # Add screenshot to messages
            messages[-1]["content"].append({
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": screenshot_b64,
                }
            })

            # Get next action from Claude
            response = self.client.beta.messages.create(
                model="claude-sonnet-4-20250514",
                max_tokens=4096,
                system=self._get_system_prompt(task),
                tools=[self._computer_tool()],
                messages=messages,
            )

            if response.stop_reason == "end_turn":
                return {"success": True, "result": response.content[0].text}

            if response.stop_reason == "tool_use":
                for block in response.content:
                    if block.type == "tool_use" and block.name == "computer":
                        action = block.input
                        result = self._execute_action(action)
                        messages.append({
                            "role": "user",
                            "content": [{
                                "type": "tool_result",
                                "tool_use_id": block.id,
                                "content": result,
                            }]
                        })

        return {"success": False, "error": "Max steps reached"}

    def _computer_tool(self) -> dict:
        return {
            "type": "computer_20241022",
            "name": "computer",
            "display_width_px": self.display_width,
            "display_height_px": self.display_height,
            "display_number": 1,
        }

    def _get_system_prompt(self, task: str) -> str:
        return f"""You are a computer-use agent. Your task: {task}

Available actions:
- mouse_move(x, y): Move cursor to (x, y) coordinates
- left_click: Click at current cursor position
- type(text): Type the specified text
- key(combination): Press a key (e.g., "Enter", "Escape")
- scroll(direction, amount): Scroll up or down
- screenshot: Take a new screenshot

Rules:
1. After each action, a new screenshot will be provided
2. If an action fails, try an alternative approach
3. If you're unsure what to do, describe what you see and what you need
4. When the task is complete, respond with a summary of what you did"""

The coordinate challenge. The model sees a screenshot at one resolution, but the actual screen may be at another. You must scale coordinates:

def scale_coordinates(
    model_x: int, model_y: int,
    model_width: int, model_height: int,
    actual_width: int, actual_height: int,
) -> tuple[int, int]:
    """Scale coordinates from model's view to actual screen."""
    actual_x = int(model_x * actual_width / model_width)
    actual_y = int(model_y * actual_height / model_height)
    return actual_x, actual_y

---

Section 4: Browser Agents

Browser agents are the most common computer-use application. Most business software is web-based, and browser agents can interact with any of it.

The hybrid approach — DOM + visual:

from playwright.sync_api import sync_playwright

class BrowserAgent:
    def __init__(self):
        self.playwright = sync_playwright().start()
        self.browser = self.playwright.chromium.launch()
        self.page = self.browser.new_page()

    def get_page_state(self) -> dict:
        """Get both visual and structural information about the page."""
        screenshot = self.page.screenshot()
        accessibility_tree = self.page.accessibility.snapshot()
        interactive_elements = self.page.evaluate("""() => {
            const elements = document.querySelectorAll(
                'a, button, input, select, textarea, [role="button"]'
            );
            return Array.from(elements).map(el => ({
                tag: el.tagName.toLowerCase(),
                text: el.textContent?.trim()?.slice(0, 100),
                id: el.id,
                name: el.name,
                type: el.type,
                href: el.href,
                rect: el.getBoundingClientRect(),
                visible: el.offsetParent !== null,
            }));
        }""")

        return {
            "screenshot": screenshot,
            "accessibility_tree": accessibility_tree,
            "interactive_elements": interactive_elements,
            "url": self.page.url,
            "title": self.page.title(),
        }

    def execute_action(self, action: dict):
        """Execute a browser action."""
        action_type = action["action"]

        if action_type == "click":
            x, y = action["coordinate"]
            self.page.mouse.click(x, y)

        elif action_type == "type":
            self.page.keyboard.type(action["text"])

        elif action_type == "navigate":
            self.page.goto(action["url"])

        elif action_type == "scroll":
            self.page.evaluate(f"window.scrollBy(0, {action['amount']})")

        elif action_type == "press":
            self.page.keyboard.press(action["key"])

        elif action_type == "select":
            # Use DOM for reliable element selection
            selector = action["selector"]
            self.page.click(selector)

        # Wait for any navigation or dynamic content
        self.page.wait_for_load_state("networkidle")

Use DOM when you can, screenshots when you must. DOM-based element targeting is more reliable than coordinate-based clicking. But for complex UIs, canvas-based applications, or when the DOM is messy, visual targeting is the fallback.

---

Section 5: Safety and Constraints

Computer use is the most dangerous agent capability. The agent can delete files, send emails, make purchases, and access sensitive information visible on screen.

Safety measures:

class SafeComputerUseAgent(ComputerUseAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.allowed_domains = set()
        self.blocked_actions = {"right_click", "drag"}  # Example
        self.action_log = []

    def _execute_action(self, action: dict) -> str:
        action_type = action["action"]

        # Block dangerous actions
        if action_type in self.blocked_actions:
            return f"Action '{action_type}' is blocked for safety."

        # Domain restriction (for browser agents)
        if action_type == "navigate":
            domain = extract_domain(action["url"])
            if self.allowed_domains and domain not in self.allowed_domains:
                return f"Domain '{domain}' is not in the allowed list."

        # Human approval for sensitive actions
        if action_type in ("click", "type") and self._is_sensitive(action):
            approval = self._request_approval(action)
            if not approval:
                return "Action was not approved by human."

        # Log everything
        self.action_log.append({
            "timestamp": datetime.now().isoformat(),
            "action": action,
            "screenshot_before": self._last_screenshot,
        })

        # Execute
        result = super()._execute_action(action)
        return result

    def _is_sensitive(self, action: dict) -> bool:
        """Check if an action requires human approval."""
        sensitive_texts = ["submit", "confirm", "pay", "delete", "send", "publish"]
        if "text" in action:
            return any(t in action["text"].lower() for t in sensitive_texts)
        return False

The safety checklist:

1. Run in a VM or container — never on your real desktop
2. Action allowlists — only allow specific actions
3. Domain restrictions — only approved websites
4. Human approval for sensitive actions
5. Comprehensive logging — every screenshot, every action
6. Timeouts and action limits — prevent infinite loops

---

Section 6: Practical Patterns

Look Before You Leap. After each action, verify the expected change occurred:

def click_and_verify(self, x: int, y: int, expected_element: str) -> bool:
    before = self.page.screenshot()
    self.page.mouse.click(x, y)
    self.page.wait_for_timeout(500)  # Wait for any animation
    after = self.page.screenshot()

    # Check if the page changed
    if images_are_identical(before, after):
        return False  # Click had no effect

    # Check if expected element appeared
    return self.page.locator(expected_element).is_visible()

Retry with Adjustment. If a click misses, adjust and retry:

def click_with_retry(self, target_text: str, max_attempts: int = 3):
    for attempt in range(max_attempts):
        # Find element by text
        element = self.page.get_by_text(target_text).first
        if element.is_visible():
            box = element.bounding_box()
            self.page.mouse.click(box["x"] + box["width"] / 2,
                                  box["y"] + box["height"] / 2)
            return True
        # Scroll and try again
        self.page.evaluate("window.scrollBy(0, 300)")
        self.page.wait_for_timeout(500)
    return False

Scroll and Scan. For long pages, scroll incrementally and scan for content:

def scroll_until_found(self, target_text: str, max_scrolls: int = 20) -> bool:
    for _ in range(max_scrolls):
        if self.page.get_by_text(target_text).first.is_visible():
            return True
        self.page.evaluate("window.scrollBy(0, 500)")
        self.page.wait_for_timeout(300)
    return False

---

Section 7: When Computer Use Is the Right Choice

Use computer use when:

• The target application has no API
• The API is insufficient (doesn't expose needed functionality)
• You need to interact with legacy or internal software
• You're automating a workflow that spans multiple applications
• You're testing a web application from a user's perspective

Don't use computer use when:

• A stable API exists → use tools instead (faster, more reliable, cheaper)
• The task is simple and repetitive → traditional automation (Selenium, Playwright scripts) is better
• Latency matters → computer use is slow (screenshots + LLM inference per action)
• Cost is a primary concern → computer use is expensive (many image tokens per step)

Computer use is the ultimate fallback. When there's no API, no tool, no integration — the agent can still interact with software the way humans do: by seeing and clicking. It's the slowest and most expensive option, but it works with ANYTHING.

---

The Turn

Your agents can now see, click, and type. They can interact with any software a human can use. This is the frontier — the point where agents stop being "API callers" and start being "computer users."

But remember: computer use is a last resort, not a first choice. Every time you reach for it, ask: "Is there an API I could use instead?" If yes, use the API. It'll be faster, cheaper, and more reliable. Computer use is for the 20% of cases where no API exists — and in those cases, it's indispensable.

---

In the next chapter: A demo is not a product. Your agents run in notebooks and scripts. But users need APIs, streaming, authentication, and reliability. You'll learn to deploy agents to production — the unglamorous work that turns your prototype into a service.