Computer Use: When Claude Operates the Desktop
On October 22, 2024, Anthropic released computer use in public beta: Claude 3.5 Sonnet operates desktops through screenshots, mouse, and keyboard. We explain the agent loop, put the OSWorld results into context (14.9 percent versus roughly 72 percent for humans), and use the known failure modes and safety gates to identify which workflows suit the technique today — and which do not.
The Last Mile of Automation
A substantial share of enterprise work still happens in graphical interfaces with no API: legacy ERP clients, vendor portals, Citrix sessions, and internal tools whose maintainers left years ago. Classic RPA controls these surfaces through recorded pixel positions and UI selectors. The workflow breaks as soon as a dialog moves, a font scales differently, or an unexpected popup appears. The scripts are deterministic; the screens are not — a mismatch that has kept GUI automation expensive and fragile for two decades.
Large language models have been able to plan such workflows for some time, but not execute them: they had no hands. Computer use closes exactly this gap: a model that reads the screen as pixels and returns mouse and keyboard actions in response. The relevant question is not whether this works in a demo. It is where the technique is reliable enough to deploy — and where it is not.
What Anthropic Released on October 22
On October 22, 2024, Anthropic released computer use as a public beta, alongside an upgraded Claude 3.5 Sonnet (claude-3-5-sonnet-20241022) and the announcement of Claude 3.5 Haiku. The beta is available on the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI. Anthropic's own framing is unusually blunt for a product launch: the capability is experimental and "at times cumbersome and error-prone."
The upgraded Sonnet also improves the agentic baselines that computer use depends on: SWE-bench Verified rises from 33.4 to 49.0 percent, TAU-bench retail from 62.6 to 69.2 percent — at the same price as its predecessor, 3 US dollars per million input tokens and 15 per million output tokens. Claude 3.5 Sonnet is the first frontier model to expose this capability in a public beta.
The Agent Loop: Screenshot to Action
Technically, computer use is a tool-use pattern, activated by the beta header computer-use-2024-10-22 and three Anthropic-defined tools: computer_20241022 for screenshots, mouse, and keyboard; text_editor_20241022 for file edits; bash_20241022 for shell commands. The model executes nothing itself. It returns tool calls; the client application implements every action. That client is therefore the natural enforcement point for whatever restrictions the operator wants.
The loop is plain: the client sends a screenshot, the model responds with an action and pixel coordinates, the client executes it and captures the next screenshot. Anthropic's reference implementation in the anthropic-quickstarts repository runs everything inside a Docker container with a virtual X11 display at 1024x768 and a Streamlit frontend — a disposable sandbox, not the developer's own desktop. That default is the correct one, and we recommend keeping it.
Benchmark Reality on OSWorld
The honest numbers come from OSWorld, a benchmark of 369 real computer tasks published in April 2024. The new Claude 3.5 Sonnet scores 14.9 percent in the screenshot-only category — roughly double the 7.8 percent of the next-best system — and 22.0 percent when allowed more steps per task. Trained human users reach about 72 percent. The state of the art therefore sits at one fifth of human reliability.
The failure modes are mundane. Anthropic itself names scrolling, dragging, and zooming as unreliable. Every step costs one screenshot in image tokens plus seconds of round-trip latency, and per-step errors compound over long action chains. During one recorded demo, Claude abandoned the coding task and began browsing photos of Yellowstone National Park. Anthropic reports this anecdote itself — and it captures the current reliability level precisely.
| System | OSWorld score |
|---|---|
| Claude 3.5 Sonnet (screenshot-only) | 14.9% |
| Claude 3.5 Sonnet (more steps allowed) | 22.0% |
| Next-best AI system (screenshot-only) | 7.8% |
| Human baseline | ~72% |
Safety Gating for Desktop Agents
Anthropic evaluated computer use against its Responsible Scaling Policy and kept the model at AI Safety Level 2: no new frontier risks, but concrete present-day ones. The dominant technical threat is prompt injection. An agent that reads the screen treats everything on that screen as input — including text on a prepared web page instructing it to ignore its previous instructions. No robust model-side defense exists today.
Gating therefore happens at the operator level: run the agent in an isolated VM or container, grant no access to credentials or sensitive data, require human confirmation before consequential actions, and restrict reachable domains to an allowlist. Anthropic additionally operates classifiers that flag sensitive usage patterns, with explicit reference to the upcoming US election. None of this is optional hardening. It is the precondition for running the beta at all.
What to Automate and What to Leave Alone
Our working rule at Blue IT Systems is explicit: computer use remains a fallback, not a default. Wherever a stable API exists, it takes precedence — it is more deterministic, faster, and orders of magnitude cheaper per operation than a screenshot loop. Screen control is justified only where no programmatic interface exists: legacy GUI applications, closed vendor portals, or exploratory UI testing. The ordering sounds obvious, yet projects regularly reverse it because the demo is more impressive.
Reasonable today are supervised, read-heavy, low-blast-radius tasks in sandboxes: smoke tests of legacy applications, data extraction from GUI-only systems, form transfer between internal tools in test environments. Not reasonable are payments, credential handling, production data, irreversible actions, and unattended long-running jobs. A system at 14.9 percent on OSWorld must never be the last instance before an irreversible click. That boundary will move; today it stands here.
Outlook: Agents at the Screen
Anthropic expects the capability to improve rapidly, and we consider that plausible. Computer use is largely a perception and robustness problem, and both respond to training data and to feedback from this beta. We expect OSWorld scores to rise substantially within a year, and hybrid agents — combining accessibility trees and DOM structure with raw pixels — to become the practical norm, because pure pixel control discards information the system already has.
The more consequential shift may happen on the other side of the screen. If agents become regular users, operating systems and applications will expose machine-readable surfaces for them, turning the screenshot from the interface into a fallback. Until then, the operating discipline remains unchanged: sandbox, supervision, small blast radius, and APIs first. The technique works. It is not yet production-mature.
Sources
- Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku (Anthropic, Oct 22, 2024)
- Developing a computer use model (Anthropic, Oct 22, 2024)
- Computer use (beta) — Anthropic API documentation (Oct 2024)
- anthropic-quickstarts: computer-use-demo reference implementation (GitHub, Oct 22, 2024)
- Xie et al.: OSWorld — Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv:2404.07972, Apr 11, 2024)