In a pitch to investors last spring, Anthropic said it intended to build AI to power virtual assistants that could perform research, answer emails, and handle other back-office jobs on their own. The company referred to this as a “next-gen algorithm for AI self-teaching” — one it believed that could, if all goes according to plan, automate large portions of the economy someday.

It took a while, but that AI is starting to arrive.

Anthropic on Tuesday released an upgraded version of its Claude 3.5 Sonnet model that can understand and interact with any desktop app. Via a new “Computer Use” API, now in open beta, the model can imitate keystrokes, button clicks, and mouse gestures, essentially emulating a person sitting at a PC.

“We trained Claude to see what’s happening on a screen and then use the software tools available to carry out tasks,” Anthropic wrote in a blog post shared with TechCrunch. “When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click in the correct place.”

Developers can try out Computer Use via Anthropic’s API, Amazon Bedrock, and Google Cloud’s Vertex AI platform. The new 3.5 Sonnet without Computer Use is rolling out to Claude apps, and brings various performance improvements over the outgoing 3.5 Sonnet model. Automating apps.

A tool that can automate tasks on a PC is hardly a novel idea. Countless companies offer such tools, from decades-old RPA vendors to newer upstarts like Relay, Induced AI, and Automat.

In the race to develop so-called “AI agents,” the field has only become more crowded. AI agents remains an ill-defined term, but it generally refers to AI that can automate software.