AI-NATIVE SERIES · COMPUTER-USE AGENTS

My Agent Had Full Authority and Still Couldn't Click “Open”

One minute: A computer-use agent has five different ways to be stopped, and only one of them is about permission. I granted mine standing authority, pointed it at a real job application, and it filled and verified every field it could reach — then lost to the operating system's “Open File” box. The useful skill is not making the agent bolder. It is being able to name which of the five walls you just hit, because four of them have different fixes and one of them has none.

Cover: the headline in serif above five horizontal bars stacked top to bottom. The
first four are solid black with white labels — mechanical, credential,
irreversible, forbidden — and the fifth is a pale grey bar with a dashed black
outline labelled out of reach. A short grey note sits to the right of each bar:
just act, ask once, grant plus verify, not yours to waive, no interface exists. — Five kinds of “I can’t.” Only the fourth is somebody’s decision, and only the fifth has no fix at all.

The honest version of this story starts with me being wrong in public. My agent asked me to paste a command into a terminal — a browser launch, nine words of flags. I looked at the transcript and found that the same agent had run that exact command itself two turns earlier. It had the capability. It asked anyway.

That is not caution. That is an agent using one word, “security,” for five situations that have nothing in common except that it did not want to proceed.

The landscape, in one number

Computer-use agents — models that see a screen, move a mouse and type, rather than calling a tidy API — are the current frontier because the world is mostly not an API. The reference measurement is OSWorld, which puts agents in a real operating system across 369 real computer tasks and reports a human success rate of 72.36% [1]. When the benchmark was introduced, the best agent managed 12.24% [1]. Scores have climbed a great deal since; the gap has not closed.

What that single number hides is why the failures happen, and it took driving a real task end to end for me to see that they are not one thing.

What I actually ran

The task was mundane on purpose: apply to a set of research-engineer jobs. Ten postings from one search. The agent fetched each one, compared the extracted text, and found the ten were really seven roles — two postings were byte-identical (the same job listed in two cities) and two more were one team's listing duplicated. Applying to all ten would have sent two teams the same candidate twice.

Then it went to work in a real browser over the Chrome DevTools Protocol, and the failures began arriving in categories.

The one that was my fault. The button to remove an attached file could not be found by any text search. Its visible text was close — a font ligature for an icon — while the human-readable label lived in aria-label="Remove resume". An agent reading the screen the way a person does looks for the words; the words were in the accessibility layer, not the pixels. Three selector attempts failed before the fourth looked in the right place.

The one that was structural. The site attaches one résumé from a stored profile. The file already sitting there was two months old and written for a different role. Had the agent simply clicked through, a carefully tailored application would have gone out with the wrong document — the most expensive kind of success, the one that reports green.

The one with no fix. Replacing the file requires the operating system's file picker. That dialog is not part of the page. It is drawn by macOS, outside the document, where an agent driving the browser simply has no reach.

The five walls

Infographic titled Five kinds of no. Five rows, each with a label, who can
unblock it, and the fix: mechanical unblocked by nobody because the agent should
just act; credential unblocked by the human once; irreversible unblocked by a
standing grant plus read-back verification; forbidden unblocked by no one because
it is a third party's rule; out of reach unblocked by nobody because the interface
does not exist for the agent. — The partition that matters. Four of these have different fixes; the fifth has none, and calling it a permission problem is how it stays hidden.

Wall	Who can lift it	What it looks like
Mechanical	nobody — just act	Launching a browser, running a build, filling a field. An agent asking a human here is a defect, not diligence.
Credential	the human, once	A password, a 2FA tap. Legitimate — but asking twice for the same credential is a design failure. Persist the session.
Irreversible	a standing grant	Submitting, sending, publishing. Delegable — but the grant is authority to act, never permission to act unverified.
Forbidden	nobody you can ask	Somebody else's rule. My own employer-application example: the company asks candidates to write their first draft themselves [4]. My operator cannot waive that, because it is not his rule.
Out of reach	nobody — no interface exists	The native file dialog. The CAPTCHA. The hardware key. Permitted, mechanical, and physically unreachable.

The fifth row is the one worth the whole article, because it is the one that gets misfiled. An agent that cannot reach a native dialog and says “for security, please do this yourself” has told you something false about the world. It is not being careful. Its arms are the wrong length.

The protocol documents the escape hatch. It did not fire.

Browser automation anticipated this. The Chrome DevTools Protocol offers a method whose description reads:

“Intercept file chooser requests and transfer control to protocol clients. When file chooser interception is enabled, native file chooser dialog is not shown. Instead, a protocol event Page.fileChooserOpened is emitted.” [3]

I enabled it. I clicked the upload control. I received zero events, and found no file input in the page to target.

I cannot tell you which of two things happened: the interception did not attach to the right target, or the control was never a plain file input in the first place. That ambiguity is the finding. The documented escape hatch and a wall that cannot be escaped look identical from the inside — both are silence.

The bug was in the contract I had just written

Here is the part that should have been embarrassing and was instead the most useful hour of the day.

Having been told off for over-asking, I wrote the five walls down as a machine-readable contract, with tests, and defined the mechanical tier as work that is “reversible by re-running.” Sensible words. Then, holding a fresh grant of authority, I read them as licence to remove the wrong résumé first and re-upload second — and the re-upload turned out to be the wall with no fix.

The draft was left with no résumé at all. I had written a rule about reversibility one hour earlier and then discovered, by breaking something live, that reversible in principle is not reversible. Proving the restore path is reachable has to happen before the destructive step, not be assumed by the word “reversible.”

So the contract gained a precondition, and the incident is now attached to the tier it created — which is the only reason I trust the tier at all. A rule with a scar is worth five rules with good intentions.

What the vendor already told me

The most under-read document in this field is the one shipped with the feature. Anthropic's own computer-use documentation states plainly that the capability is “in beta,” and, in its limitations:

“Do not use Claude for tasks requiring perfect precision or sensitive user information without human oversight.” [2]

And, on why my afternoon went the way it did:

“Additionally, reliability might be lower when interacting with niche applications or multiple applications at once.” [2]

A stored-profile résumé swap behind an OS-drawn file picker is exactly the niche interaction that sentence is about. The reasoning underneath it is the useful part: the model is inferring intent from a rendered surface, so its reliability tracks how conventional that surface is. Novel widget, novel failure. The vendor is not hedging; it is describing the mechanism.

Patterns and anti-patterns

Patterns

Name the wall before you name the fix. “I can't” is not a status. “This is out of reach, the micro-step is choosing a file in an OS dialog” is a status — and it is ten seconds of human time instead of an argument.
Read the accessibility layer, not the pixels. The label a human sees and the label the DOM carries diverge constantly. aria-label found what three text searches missed.
Prove the restore path before you destroy anything. Locate the actual control. Believing one exists is not the same as finding it.
Read the value back after writing it. Every field the agent set, it re-read. That habit is what caught the two-month-old résumé before it shipped.

Anti-patterns

Calling a reach problem a permission problem. It hides a tooling gap behind a virtue.
Asking for confirmation instead of verifying. A grant of authority is often answered with more questions, because questions feel safe. Verification is the actual job.
Trusting a green result without asking which artifact produced it. The wrong-résumé submission would have reported success.
Taking a benchmark score as a capability map. One number cannot tell you which of five walls your task will hit.

The mechanism

An agent's failures are not points on a single line from “dumb” to “smart.” They are a partition by blocker: the agent itself, a secret, an authorization, a third party, or the absence of an interface. Capability improvements move one boundary at a time. Naming the partition converts an unactionable refusal into a routed one — and it makes over-asking and over-reaching detectable as the same class of bug, which is a mis-filed wall.

The first principle

An agent's autonomy is bounded not by its intelligence but by the narrowest reachable interface at each step — so the work is widening reach or naming the gap precisely, never asserting confidence.

Which brings me to the joke I have to tell on myself. In one afternoon this agent surveyed a landscape, deduplicated ten job postings down to seven by comparing them byte for byte, resolved a role its own search engine had never surfaced, drove a live application over a debugging protocol, wrote a five-tier authority contract with forty passing tests, and caught a flaw in that contract within the hour.

Then it was defeated by a small grey box with a Cancel button.

The next version of that agent will not be smarter. It will know which box it is looking at.

References

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. 369 real computer tasks; human success 72.36%; best reported agent at introduction 12.24%. osworld-v1.xlang.ai (accessed 4 Aug 2026).
Anthropic, Computer use tool — documentation, including the beta notice and the Limitations section quoted above. platform.claude.com/docs/en/agents-and-tools/computer-use (accessed 4 Aug 2026).
Chrome DevTools Protocol, Page domain — Page.setInterceptFileChooserDialog and the Page.fileChooserOpened event. chromedevtools.github.io/devtools-protocol/tot/Page (accessed 4 Aug 2026).
Anthropic, Guidance on Candidates' AI Usage — the first-draft rule cited as the “forbidden” example. anthropic.com/candidate-ai-guidance (accessed 3 Aug 2026).

Every reference above was fetched and read while writing this piece; every passage in quotation marks was checked verbatim against its source. The measurements from my own run — the ten-to-seven deduplication, the three verified fields, the zero interception events — are from a single session on 3–4 August 2026 and are reported as observations, not as a benchmark.

Written by Paul Jialiang Wu · agentic-portfolio · 4 August 2026. The agent in this article is my own; the walls are everybody's.