How to give Claude computer use on a desktop

Claude can interact with computer environments through the computer use tool, which provides screenshot capabilities and mouse and keyboard control. It is a client toolset. One entry, {"type": "computer_toolset_20260801"}, in the Messages API tools array gives Claude 17 member tools such as screenshot, left_click, type, and zoom. Your application runs every call. Claude does not connect to the desktop. Isolate that environment before you implement a single action.

Isolate the desktop first

Computer use has unique risks distinct from standard API features. These risks are heightened when interacting with the internet. In some circumstances, Claude will follow commands found in content even when they conflict with your instructions. Instructions on webpages, or contained in images, might override your instructions or cause Claude to make mistakes. Isolate Claude from sensitive data and actions to avoid risks related to prompt injection.

Anthropic has trained the model to resist these prompt injections and has added an extra layer of defense. If you use the computer use tools, classifiers automatically scan what the tools return, such as screenshots, to flag potential prompt injections. When these classifiers identify a potential prompt injection, they steer the model to check whether the instruction really came from you before acting on it. This extra protection will not be ideal for every use case, for example use cases without a human in the loop. Opting out means contacting support. The precautions above remain important even with these classifiers in place.

Inform end users of relevant risks and obtain their consent prior to enabling computer use in your own products.

For tasks that stay inside webpages, the browser use tool is the closer fit. Its members read and act on the page itself and do not need a full desktop. Declared together, the two toolsets work independently, each in its own coordinate frame. Shared names such as screenshot or key are told apart by toolset_name.

Declare the toolset

Add {"type": "computer_toolset_20260801"} to the tools array of a Messages API request. The request needs no beta header. The quick-start sample also declares the text editor tool and the bash tool, which Claude typically uses alongside computer use. The sample uses the model string claude-opus-5-5. You do not provide an input schema. The schema is built into the model and cannot be modified.

The entry accepts a required type, optional configs, optional cache_control, and optional allowed_callers limited to direct. configs is keyed by member name. Each member accepts enabled, default true for all 17 including zoom, and defer_loading, default false. Omitted members keep those defaults. If the environment cannot produce zoom images, withhold zoom in configs instead of leaving it enabled and returning errors.

The entry rejects name, display_width_px, display_height_px, display_number, and enable_zoom. Including any of them returns an invalid_request_error. Member names are fixed by the toolset version, and coordinates always come from the screenshots you return. Do not declare this entry beside a computer_20251124 entry or another tool named computer. Computer use is not currently available in Claude Managed Agents.

When the tool is on the request, the API generates a computer-use system prompt. Your system parameter is still respected in the combined prompt. If the loop can run only one action per round trip, set disable_parallel_tool_use to true in tool_choice so Claude returns at most one member block per turn.

Run the agent loop

A desktop action comes back with stop_reason of tool_use and one or more member tool_use blocks. Each name is the member, each block carries a computer toolset_name, and input holds only that member's parameters. There is no action field. Several blocks in one response are a batch. Run them in order, not concurrently.

1. Provide the toolset and a user prompt that needs desktop interaction.
2. Claude answers with member calls when a desktop action can help.
3. Run every tool_use block in order. Dispatch on toolset_name and name, because a custom tool can share a member name. Apply that block's input on your container or virtual machine. Send a new user message with one tool_result per call, matched by tool_use_id, each carrying toolset_name set to computer. A result that omits it, or that names a different toolset, is rejected. Return an image for screenshot and zoom. A short text acknowledgment is enough for other actions. cursor_position returns coordinates as text.
4. If Claude asks for more actions, repeat step 3. Otherwise it returns text.

Steps 3 and 4, repeated without user input, are the agent loop. A simplified sample stops after 10 iterations so a stuck task cannot produce unexpected API costs. If you also declared the text editor or bash tool, dispatch those blocks in the same pass. A turn with no answered calls is finished.

Stop the batch at the first failure. Later actions usually depend on earlier ones. Every block still needs a tool_result, or the next request is rejected with an invalid_request_error. Return a normal result for each success, an error result that describes the failure, and an error result for every later action. For those skipped actions, use the exact sentence in the Batch actions section. Do not paraphrase it. Claude then sees what succeeded, what failed, and what was skipped, and replans. If a human must confirm consequential actions, check before each block runs. A batch can finish a multistep action within one turn.

Claude typically ends a batch with screenshot. If it does not, you can attach a screenshot as an extra image on the last result. You can also ask, in the system prompt, that each group of actions end with a screenshot.

Keep coordinates in screenshot pixels

Every coordinate, start coordinate, and region, and the position cursor_position reports, is in the pixel space of the full-display screenshots you return. The origin is the top left. After zoom, Claude still uses that full-screenshot space, never coordinates relative to the zoomed image. If you scale screenshots down before returning them, scale Claude's coordinates back up before applying them. That is why this toolset rejects display_width_px, display_height_px, and display_number.

All 17 members are enabled by default, including zoom. Screenshot takes no input and returns the full display. Zoom takes a region of two corners and returns that area at full resolution, scaled to fit usual screenshot dimensions with aspect ratio preserved. Clicks, drag, move, and scroll use the same pixel space. Type sends literal text at the current keyboard focus. Key presses a key or a plus-joined combination, with an optional repeat from 1 to 100, default 1. Hold-key and wait take a duration in seconds, up to 300. If Claude calls a member you withheld or do not implement, return an error result for that block.

Use a sandbox. The docs describe a virtual X11 display using Xvfb, a Linux UI with Mutter and Tint2, and applications such as Firefox and LibreOffice. Your application translates tool requests into actions and returns results. The reference implementation runs that stack in a Docker container. A custom environment needs a virtualized or containerized desktop, action handlers, an agent loop, and a way for a user to start the loop.

Give simple, well-defined tasks. Ask Claude to screenshot after each step, state in one sentence what the image shows and whether the step succeeded, and continue only after the step is confirmed. Place instruction text before a screenshot in the user turn. If the model must log in, provide the username and password inside XML tags such as robot_credentials, and review the Platform guidance on mitigating jailbreaks and prompt injections first. Login increases the risk of a bad outcome from prompt injection.

What to do next

Read the computer use tool page before you write a handler, and copy the skipped-action sentence from the Batch actions section rather than inventing one. Declare {"type": "computer_toolset_20260801"} with no beta header on a containerized desktop that holds no sensitive data. Run one trivial prompt until ordered batches and screenshot-pixel coordinates match the images you return.

Sources

• Computer use tool