Put Claude to Work on Your Computer, from Anywhere

1. What Computer Use Actually Is

When software engineers first encounter the phrase “computer use,” they often assume it is a sophisticated API integration — a system that calls web APIs, reads structured data, and writes code to interact with services. It is not. Computer use is considerably more direct, and considerably more strange when you first see it working.

Claude computer use gives the model the ability to operate a real graphical desktop interface the same way a human does. It receives screenshots of the current screen state, analyzes what it sees, and then issues low-level input commands: move the mouse to a specific pixel coordinate and click, type a string of text, press a keyboard shortcut, scroll a scrollbar. After each action, it takes another screenshot, reasons about what changed, decides what to do next, and repeats. The interaction loop is perception followed by action, iterated until the task is complete.

This means Claude is not calling a hypothetical “click the submit button” API. It is literally asking the operating system to move the cursor to the coordinates where it believes the submit button to be based on visual analysis of the screenshot, and then requesting a mouse click at those coordinates. If the button has moved — because the window resized, or a dialog appeared in front of it — Claude needs to figure that out from the next screenshot and adjust. This is the same challenge a human remote-controlling a computer would face, and Claude handles it the same way a careful human would: by looking before acting, and looking again after.

The practical consequence of this architecture is that computer use can work with any application, anywhere on the screen, regardless of whether that application has an API. A legacy desktop application with no external interface, an internal tool that was never designed to be automated, a website that blocks programmatic access but loads fine in a browser — computer use can interact with all of them because it uses the same interface a human would use. The application has no way to know it is talking to an AI rather than a person.

This makes computer use qualitatively different from any previous form of software automation. Traditional robotic process automation (RPA) requires recording exact pixel coordinates or element IDs in advance and breaks when the UI changes. API integration requires the target system to expose a structured interface. Computer use requires neither: it reasons about the current visual state at runtime, the same way a human would if they sat down at the computer for the first time.

2. The Two Products: Cowork and Claude Code

Computer use in the Anthropic ecosystem is currently available through two distinct products, each designed for a different primary user and workflow. Understanding which product to use requires understanding the difference in their intended contexts.

Claude Cowork is designed for non-technical knowledge workers — people who want to delegate complex multi-step tasks to Claude without writing any code. Cowork provides a managed environment where Claude can be given permission to use a computer on your behalf, handling workflows that would normally require sustained manual attention: researching across multiple websites, filling out forms, managing files, compiling information from several applications into a single output. The interaction model is conversational — you describe what you want in plain language, and Claude figures out the sequence of computer actions required to do it. Cowork is positioned as the consumer and enterprise product for delegating digital work.

Claude Code is the developer tool. It is a command-line interface that Claude operates to assist with software development: writing code, running tests, reading documentation, navigating file systems, using development tools. But computer use in Claude Code goes further than the command line — it allows Claude to operate the full desktop, opening applications, using GUI tools, interacting with browser-based developer consoles, and handling tasks that require visual interaction with software development environments. For developers, Claude Code’s computer use is most useful for tasks that span CLI operations and GUI tools in the same workflow.

The key distinction between the two products is audience and primary use case. Cowork is for people who want to delegate work to Claude and observe the result. Claude Code is for developers who want Claude as an active collaborator in development workflows, where computer use is one of several capabilities alongside file editing, code generation, and shell command execution.

Both products share the same underlying computer use technology and the same security model. The difference is in the interface, the intended workflow, and the level of technical control exposed to the user.

3. Task Types, Security, and When to Use Computer Use

Not every task is a good fit for computer use, and part of developing competence with this capability is building a clear intuition for when to use it and when a different approach is better. The decision is shaped by three factors: whether the task genuinely requires visual UI interaction, whether the latency and cost of screenshot-based iteration is acceptable, and whether the task’s risk profile is appropriate for autonomous computer control.

Task types well-suited to computer use share a common characteristic: they require navigating a graphical interface that has no programmatic alternative. Web scraping from a site that blocks bots but loads normally in a browser. Filling out forms in a legacy enterprise system with no API. Using a GUI-based design tool to export assets. Navigating a government or regulatory portal to submit documents. Compiling information from multiple applications into a single report. These tasks are time-consuming for humans, have a clear definition of “done,” and do not require real-time responsiveness.

Task types that are poor fits for computer use are those where a more direct approach is available and superior. If a service has a REST API, call the API — it is faster, cheaper, more reliable, and produces structured output. If you need to read or write files, use a file system tool directly rather than having Claude navigate a file manager in a GUI. If you need to run code, use a subprocess or code execution environment. Computer use’s screenshot-based iteration loop introduces latency and cost at every step; when a direct tool call can accomplish the same thing in one round trip, prefer the direct tool call.

The mental model: computer use is the approach of last resort for automation, used when no programmatic interface exists. It is the approach of first resort for tasks that are genuinely visual, spatial, or that require working with software exactly as a human would.

Security and permissions are the most important considerations before deploying computer use. When Claude is given control of a computer, it can do anything a human at that keyboard could do. That is both the power and the risk. A few principles apply.

First, use a dedicated, sandboxed environment rather than your personal computer. Create a separate virtual machine or container with limited permissions. The computer use environment should not have access to credentials, sensitive files, or systems beyond what is required for the specific task. Think of it like giving a contractor access to your building — you give them a key to the room they need to work in, not a master key to everything.

Second, review Claude’s action plan before authorizing long autonomous sessions. For high-risk tasks (anything involving financial transactions, sending communications, or modifying important data), require Claude to describe its intended action sequence before executing. This is especially important when the task involves irreversible actions — sending an email, submitting a form, deleting a file.

Third, monitor the session. Most computer use deployments display the screen in real time so a human supervisor can observe and interrupt if something goes wrong. Treat computer use sessions the way you would treat an intern running an important task — you do not need to watch every click, but you should be close enough to intervene.

4. Real-World Examples and the API vs. Computer Use Decision

Understanding computer use in the abstract is useful; seeing how the decision plays out in real scenarios is more useful. Here are several examples that illustrate when computer use is the right choice and when it is not.

Scenario 1: Extracting data from a competitor’s website. Their site loads normally in a browser but detects and blocks automated HTTP requests. Computer use is appropriate here — Claude can open a browser, navigate to each page, scroll through the content, and record what it sees. A direct HTTP scraper would be blocked; computer use succeeds because it behaves like a human browser session.

Scenario 2: Sending invoices through your accounting software. Your accounting software is a legacy desktop application with no API. The invoicing workflow requires navigating several screens, filling fields, attaching documents, and clicking through a confirmation dialog. Computer use is appropriate — it is the only realistic automation path for an application with no programmatic interface.

Scenario 3: Querying your company’s analytics database. Your analytics system has a SQL interface and an API. Do not use computer use. Write a query via the API or run a direct database call. You will get structured output that is machine-readable, the operation will complete in milliseconds instead of the seconds or minutes that a screenshot-based UI loop would require, and you will not risk Claude misreading a number from a screenshot.

Scenario 4: Generating a weekly status report by pulling data from five different internal tools. Three of the five tools have APIs; two are legacy web applications with no programmatic interface. A hybrid approach is best: use direct API calls for the three tools that have them, and computer use for the two that do not. Build an orchestrator that combines the outputs.

import anthropic

client = anthropic.Anthropic()

computer_tool = {
    "type": "computer_20241022",
    "name": "computer",
    "display_width_px": 1280,
    "display_height_px": 800,
    "display_number": 1
}

# Pull a desktop-enabled container image
docker pull anthropics/anthropic-quickstarts:computer-use-demo-latest

# Run it with the API key injected
docker run -it \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -p 5900:5900 \
  -p 8501:8501 \
  anthropics/anthropic-quickstarts:computer-use-demo-latest

import anthropic
import base64
import subprocess
from PIL import ImageGrab  # or use pyautogui, scrot, etc.

client = anthropic.Anthropic()

def take_screenshot() -> str:
    """Capture the current screen and return as base64 PNG."""
    screenshot = ImageGrab.grab()
    import io
    buf = io.BytesIO()
    screenshot.save(buf, format='PNG')
    return base64.b64encode(buf.getvalue()).decode()

def execute_action(action: dict) -> str:
    """Execute a computer use action and return the result."""
    action_type = action.get("type")
    
    if action_type == "screenshot":
        return take_screenshot()
    
    elif action_type == "mouse_move":
        x, y = action["coordinate"]
        subprocess.run(["xdotool", "mousemove", str(x), str(y)])
        return "moved"
    
    elif action_type == "left_click":
        x, y = action["coordinate"]
        subprocess.run(["xdotool", "click", "--clearmodifiers", "1"])
        return "clicked"
    
    elif action_type == "type":
        subprocess.run(["xdotool", "type", "--clearmodifiers", action["text"]])
        return "typed"
    
    elif action_type == "key":
        subprocess.run(["xdotool", "key", "--clearmodifiers", action["key"]])
        return "key pressed"
    
    return f"unknown action: {action_type}"

def run_computer_use_task(task: str, max_steps: int = 20):
    """Run a computer use task with a perception-action loop."""
    
    initial_screenshot = take_screenshot()
    
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": initial_screenshot
                    }
                },
                {
                    "type": "text",
                    "text": task
                }
            ]
        }
    ]
    
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=[{
                "type": "computer_20241022",
                "name": "computer",
                "display_width_px": 1280,
                "display_height_px": 800,
            }],
            messages=messages
        )
        
        messages.append({"role": "assistant", "content": response.content})
        
        if response.stop_reason == "end_turn":
            print("Task complete.")
            for block in response.content:
                if hasattr(block, "text"):
                    print(f"Final response: {block.text}")
            break
        
        tool_results = []
        for block in response.content:
            if block.type == "tool_use" and block.name == "computer":
                result = execute_action(block.input)
                
                if block.input.get("type") == "screenshot":
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": [
                            {
                                "type": "image",
                                "source": {
                                    "type": "base64",
                                    "media_type": "image/png",
                                    "data": result
                                }
                            }
                        ]
                    })
                else:
                    new_screenshot = take_screenshot()
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": [
                            {
                                "type": "image",
                                "source": {
                                    "type": "base64",
                                    "media_type": "image/png",
                                    "data": new_screenshot
                                }
                            }
                        ]
                    })
        
        if tool_results:
            messages.append({"role": "user", "content": tool_results})

run_computer_use_task(
    "Open a text editor, type 'Hello from Claude computer use', and save the file as test.txt on the desktop.",
    max_steps=15
)

CONSEQUENTIAL_ACTIONS = ["submit", "send", "delete", "confirm", "purchase"]

def safe_run_with_review(task: str):
    print(f"Task: {task}")
    print("Requesting action plan before execution...")
    
    plan_response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        messages=[
            {
                "role": "user",
                "content": f"I want you to complete this task using computer use: '{task}'. Before you start, list the exact steps you plan to take, including any actions that would send, submit, delete, or otherwise make irreversible changes. Do not take any actions yet."
            }
        ]
    )
    
    plan = plan_response.content[0].text
    print(f"\nProposed plan:\n{plan}\n")
    
    approval = input("Do you approve this plan? (yes/no): ").strip().lower()
    if approval == "yes":
        run_computer_use_task(task)
    else:
        print("Task cancelled.")