Computer-use agents are becoming a new automation layer

Computer-use models can read screenshots, move a pointer, type and navigate applications built for people. That matters because many real workflows cross legacy systems, vendor portals, desktop software and websites with incomplete or unavailable APIs. OpenAI, Anthropic and Google now expose or demonstrate this capability across browsers and desktop environments.

The flexibility comes from using the same interface as a person. It also inherits every weakness of that interface: layout changes, animations, hidden state, scroll position, focus, modal dialogs, accessibility defects and ambiguous labels. A click can land correctly while the underlying business action still fails.

Treat visual operation as an adapter layer. When a stable connector, API or scripted contract exists, it usually provides stronger identity, parameter validation, idempotency and outcome verification. The agent should choose the most precise interface that can complete the step.

A computer-use model must map an instruction such as “open the latest invoice” to a specific element on a changing screen. Screenshot-only agents estimate coordinates. Other systems add DOM nodes, accessibility trees, OCR, application metadata or specialized grounding models.

Resolution and scaling matter. Anthropic’s current guidance emphasizes screenshot dimensions because coordinate errors propagate into every later action. VisualWebArena and OSWorld-G show that charts, icons, dense interfaces and fine manipulation remain difficult even when the high-level plan is correct.

Observation should be layered. Screenshots preserve appearance; accessibility and DOM structures provide names and relationships; structured tools expose authoritative values. None is complete alone. The agent also needs to know which window, tab and account is active.

OSWorld 2.0 extends computer-use evaluation into workflows that take people more than an hour and require hundreds of actions. Its reported failures are not limited to clicking. Agents lose constraints, miss information arriving mid-task, guess instead of asking and skip final verification.

A durable plan needs a state ledger: which source supplied a value, which account is active, which rows were already changed, which files were downloaded and which assumptions remain unresolved. The ledger should be compact enough to stay usable but detailed enough to prevent duplicate or contradictory work.

The agent should ask when hidden state matters. Guessing a billing period, recipient, folder or account can be more damaging than stopping. A task contract should define which ambiguities may be resolved automatically and which require clarification.

A model may report success after clicking Submit even though the page showed a validation error, timed out or saved a partial record. Reproducible benchmarks such as WebArena score the resulting application state rather than the agent’s explanation. Production systems need the same discipline.

Verification can inspect confirmation pages, database state, sent-message folders, generated files, API responses or application-specific signals. It should distinguish a reversible navigation error from a duplicated payment, deleted record or message sent to the wrong person.

Recovery rules belong in the workflow before execution begins. Safe retries require idempotency keys, duplicate detection or an independent query of current state. When the system cannot establish whether an action completed, it should stop and escalate rather than repeat blindly.

Computer-use agents treat webpages, email, documents and application text as observations. An attacker can place instructions inside that content and attempt to redirect the agent, reveal data or trigger an unintended action. OpenAI’s agent system card treats prompt injection as a product-specific risk because the agent can combine browser access with files, connectors and other tools.

The defence cannot be one model instruction. Separate user intent from environmental content, restrict destinations and tools, classify sensitive data, screen candidate actions and require confirmation at the transaction boundary. An untrusted page should never gain authority merely because the model can read it.

Persistent browser state increases the stakes. Logged-in sessions, autofill, downloads and cookies can expose more than the current page. Use a dedicated profile, minimal connected applications and short-lived credentials for the task.

OpenAI, Google and Anthropic all describe human control or confirmation around sensitive computer-use actions. The design detail matters. “Continue?” is weak when the user cannot see what will be sent, bought, deleted or shared.

A useful confirmation presents the final object and consequence: recipient and message, item and total, document and permission scope, file and deletion set, account and changed setting. The agent should not alter those parameters after approval without asking again.

Human takeover is also necessary for passwords, multifactor authentication, captchas, unexpected legal terms and ambiguous security warnings. Takeover should pause model observation of sensitive input where the product supports that boundary.

OSWorld, WebArena, VisualWebArena, WorkArena and BrowserGym provide increasingly realistic, reproducible environments. Their results show rapid improvement and a persistent gap on long, dynamic and stateful work. OSWorld-Human also highlights inefficiency: agents may take far more steps and time than a person even when they eventually succeed.

Benchmark comparisons need the exact observation mode, action space, step budget, model, reasoning setting, tool access and evaluator. A model using APIs or MCP tools is not directly comparable with one restricted to screenshots and mouse clicks.

Deployment evaluation should replay the organization’s own workflows in isolated copies. Include changed layouts, slow pages, pop-ups, expired sessions, mid-task messages, conflicting records and tasks that should be refused or escalated.

A browser agent requires more than a model call. Teams maintain browser images, profiles, extensions, site allowlists, login procedures, selectors or grounding prompts, traces, evaluation environments and incident response. Live websites change without notice.

Visual agents can be valuable where integration cost is otherwise prohibitive or where a person already spends time moving information between systems. They are less attractive for high-volume stable processes that deserve a direct API or conventional automation.

Measure cost against verified accepted work. Include model tokens, screenshots, browser runtime, tool calls, failed trajectories, review time, credential administration and break-fix maintenance. The best design is often hybrid: structured tools for important state changes and computer use for the few GUI-only gaps.