The embodied AI world builds a bespoke model for every task. The alternative, argued by Pranjal Aggarwal and Sean Welleck (Carnegie Mellon University) in “Computer-Use Agents Are Embodied Generalists” (arXiv:2611.09847), is startlingly simple: treat the physical world as software.
The move. Build software that renders an agent’s view of a physical environment on a screen and exposes controls for acting in it. A computer-use agent (CUA) then navigates, grasps and segments through the same screenshots and keyboard-and-mouse actions it uses to operate a spreadsheet. Task-specific capability lives in the software, not in the agent design.
The evidence. The authors entered four ECCV 2026 workshop competitions and changed only the software, with no competition-specific agent tuning: first in OpenSUN3D (language-guided 3D segmentation), second in MeViSv2-Text (referring video segmentation), third in DialNav (dialogue-driven navigation) and sixth in VLNVerse (vision-language navigation). The agent was Gemini 3.5 Flash and 3.6 Flash, identical system prompt and action space throughout, able to batch up to 16 inputs per turn and dropping the oldest ten screenshots once context hit twenty.
The numbers are more telling than the placings. In OpenSUN3D the agent beat every other entry by more than 4 points in AP50 and 10 points in AP25. In VLNVerse it placed sixth on success among 74 entries but posted the third-highest NDTW on both tracks — it failed less often but followed the reference route better than nearly anyone. In MeViSv2 it came second ahead of a rival with a higher J&F score, because it recognised when nothing matched the expression more often: 88.89% versus 83.33% no-target accuracy.
GUI or API? A natural question: why not give the agent direct text APIs? On validation sets, the two interfaces landed within four points of each other, with the GUI actually ahead in two of four competitions. The screen is not a handicap.
The argument. Computer use had exactly today’s embodied-AI problems two years ago: imprecise clicking, overlays and separate grounding models. It was fixed by training for visual grounding, scaling simulator data, and making interaction fast — and the fix transferred to a single general model writing click coordinates as plain text. The authors wager embodied AI follows the same road, and that the software wrapper is the cheapest way to start.
Paper: arXiv:2611.09847 · Project site