Computer use agents can look at a screenshot, read a text description of the screen, or do both. Greptile tested all three setups on seven OpenAI models and found that a text view of the screen lifts weaker models sharply, while the strongest models do just as well from pixels alone.
The results shaped how the company’s QA agent, TREX, sees an app.
Why clicking is hard for a model
A person moves a mouse to a button. A model has to write the coordinates as text, such as x=523, y=217, and being off by a few pixels can mean clicking the wrong thing. The same button also sits in different places at different screen sizes.
Vision makes it worse. A screenshot is shrunk to fit the model’s size limit and cut into squares dozens of pixels wide, yet the click must land on one exact pixel. ScreenSpot-Pro, a 2025 test built around small targets on sharp, high-resolution displays, saw the top model of its day land on the right target just 18.9% of the time. Anthropic’s own documentation cautions that Claude can get coordinates wrong or invent them.
An old accessibility idea, reused
The accessibility tree is a text list of every button and control on screen, with positions and sizes. The idea goes back to 1984. Jim Thatcher of IBM teamed up with Jesse Wright, his former thesis adviser, who was blind, on a side project that spoke the PC screen aloud. Microsoft followed with Active Accessibility in 1997, Sun began AT-SPI on Linux in 2001, and Apple introduced its own Accessibility API in 2002.
Agents found it useful too. In the 2024 OSWorld benchmark, models fed only screenshots managed roughly one task in twenty, while most of those given the screen as text did better. People finished 72% of OSWorld tasks when it launched, against about 12% for the best model.
How the test worked
Greptile built three setups, each adding to the last: screenshots alone, screenshots plus the accessibility tree with clicks by coordinates, and then clicks by element ID, where the model names a control and the test framework clicks it. All three ran the same agent through 172 OSWorld tasks. Each task got a single attempt, capped at 50 steps, with low reasoning effort, and OSWorld’s own graders did the scoring.
Because OSWorld runs on Linux, the agents read AT-SPI. One quirk turned up: GIMP did not appear in the tree until an environment variable set at login was restored.
What the numbers showed
With screenshots alone, GPT-4.1 scored 5.4%, GPT-5.2 scored 15.8%, and GPT-6.1 Sol scored 83.8%.
- Adding the tree lifted GPT-5.2 to 36.2% and GPT-4.1 to 20.6%.
- Clicking by ID pushed GPT-5.2 to 52.8% and GPT-4.1 to 29.3%, so GPT-5.2 more than tripled its screenshot-only score.
- GPT-6.1 Sol reached 82.7% with the tree and 82.1% when clicking by ID, a touch below its 83.8% from screenshots alone.
For the three strongest models, no change was statistically significant. Across the four weaker models, scores rose by 9 to 37 points, doubling or better in three of the four. For reference, OpenAI reports 71.4% for GPT-6.1 Sol on the harder OSWorld 2.0 offline set at maximum reasoning effort.
Where the tree falls short
Some apps expose no tree. Figma lists its toolbar buttons but not what is drawn on the canvas, and a game such as Doom is nothing but pixels. Even a well-described app can hide bugs that appear only visually, like text spilling out of its box. Only the screenshot catches those.
What happens next for TREX
Greptile added computer use to TREX because code review catches bugs in the diff, but some bugs only exist at runtime: a button that cannot be clicked, a label running off the edge of the window. TREX now runs the changes from a pull request on a real desktop and operates the app like a user.
The takeaway for builders is practical. If you build computer use agents on weaker models, add the tree and click by ID. If you rely on a frontier model, screenshots may be enough. Either way, keep them around for canvas apps and bugs that only show on screen.
Source: greptile.com