what claude told me after struggling with agent-desktop on a whatsapp project

[pleabargain]

I installed Agent Desktop, and it worked more or less out of the box, but there were a few things I had to do manually, namely split the screen so that the agent would have full focus on the application. I could not have Claude full screen and the target whatsapp application full screen, so that is a significant limitation in using WhatsApp and Agent Desktop.

Best practices for driving desktop apps with agent-desktop

Field notes from automating a real Windows desktop app (a Microsoft Store / Electron-hybrid chat
client) with agent-desktop. Nothing here is
app-specific advice; it's a set of failure modes that are easy to hit with any semantic,
accessibility-tree-driven automation CLI, plus the patterns that reliably work around them.

1. Read the platform skill before guessing commands

agent-desktop skills get desktop      # core loop, ref system, retry semantics
agent-desktop skills get platform     # OS-specific behavior and gotchas
agent-desktop status                  # platform, permissions, supported surfaces
agent-desktop <command> --help        # exact flags, don't guess them

The platform skill documents exactly which commands need --headed, which are headless-safe,
and which fail closed with POLICY_DENIED or ACTION_NOT_SUPPORTED. Skipping this step and
guessing flags is the single biggest source of wasted round-trips.

2. Know your shell's quoting traps

Accessibility refs look like @s2ymh0aottgom0:e6 and coordinate pairs look like 509,248. Both
are landmines in PowerShell specifically:

• A bare @token is PowerShell's splatting operator. agent-desktop click @s2ymh0aottgom0:e6
  silently drops the argument and fails with INVALID_ARGS / "Missing arguments". Always
  single-quote it: agent-desktop click '@s2ymh0aottgom0:e6'.
• A bare x,y is parsed as a PowerShell array, not a string. agent-desktop mouse-click --xy 509,248 fails with "Invalid coordinates. Expected format: x,y". Quote it:
  agent-desktop mouse-click --xy "509,248" --headed.

Neither trap exists in bash/zsh, but if your agent might run in either shell, quote defensively in
both.

3. Prefer refs, but know when they can't work

The normal, safe loop is: snapshot → act on a @ref → re-snapshot to verify. Refs are semantic
(no mouse movement, no focus stealing, safe to run headless) and should be your default.

They break down on a specific, recognizable class of app: Electron or WebView2-hosted apps that
keep more than one renderer instance alive (one visible, one hidden/suspended, often for fast
resume or a notification preview surface). Symptoms:

• find/click on an element that is clearly unique on screen returns AMBIGUOUS_TARGET with
  details.kind: identity_not_unique, even when scoped with --root to the branch that looks
  right.
• Filtering by a state flag (e.g. --state offscreen=false) to exclude the hidden instance
  doesn't help either; it can come back TIMEOUT with an "activation" step that was never even
  attempted, and the error says retryable: false.

If you see this pattern — identical name, role, and automation id showing up more than once in a
window, with one branch never expanding past a generic container — stop trying to disambiguate
through the accessibility tree. The uniqueness check these tools use is almost always global
(across the whole window, counting elements even under an unexpanded or skeleton-cut branch),
so no amount of clever scoping fixes it. Fall back to coordinate-based interaction (section 5)
for that specific app, and keep using refs everywhere else.

4. Don't trust focus-window, and don't race for focus

A command that programmatically brings a window to the foreground can fail outright:

{"code": "ACTION_FAILED", "message": "Target window did not become foreground before headed delivery"}

This is standard OS foreground-lock behavior (Windows refuses SetForegroundWindow from a
process that doesn't currently own the foreground-change privilege), not a transient bug.
Retrying the identical call will not help even if the error claims retry: safe.

More subtly: whatever terminal or harness is dispatching your commands can itself keep
reclaiming OS focus between calls (it's the thing the human is actively looking at). A physical
mouse click fired moments later can land on that window instead of your intended target, even
though is_focused said the right thing a command or two earlier. Checking focus right before a
click narrows the race but does not close it.

The fix that actually removes the hazard: arrange the target app and your terminal as two
tiled, non-overlapping windows (e.g. side by side). A coordinate click inside the target's
exclusive screen region physically cannot land on a window that occupies different screen space,
no matter which one currently holds keyboard focus. Verify the tiling by comparing both windows'
bounds from a window-listing command and confirming their coordinate ranges don't overlap, before
you start clicking. Treat this as a geometry problem to eliminate, not a timing problem to win.

5. Coordinate clicks: get the geometry right once

When you do need to fall back to raw coordinates:

1. Get the target window's current bounds (position and size) from the window-listing command.
2. Take a screenshot of just that window; note the image's width, height, and reported DPI scale
   factor.
3. Check whether your tool's coordinate space is physical or logical (DPI-scaled) pixels. Many
   modern automation adapters force per-monitor DPI awareness specifically so that window
   bounds and click coordinates are both physical pixels — the same unit a screenshot is
   captured in. If that's documented behavior for your tool, you do not need to divide by the
   scale factor to go from a screenshot pixel to a click coordinate. Don't assume either way;
   confirm it from the tool's own docs or by testing a click against a known target.
4. Account for the invisible resize border many OSes add to a window's reported rectangle (a few
   pixels on each edge is common). In practice this usually means the window's visible content
   starts at (max(bounds.x, 0), max(bounds.y, 0)) in screen space, and a screenshot pixel
   (ix, iy) maps to screen coordinate (content_x + ix, content_y + iy).
5. If a vision step hands you coordinates read off a resized preview image rather than the
   original file, scale them by real_dimension / displayed_dimension first. Always get the real
   pixel dimensions from the screenshot command's own result, never assume a preview's size.
6. Recompute every time the window moves, resizes, or the layout otherwise changes. Never hardcode
   coordinates across sessions or screen configurations; derive them fresh each run and treat any
   previously observed coordinates as a worked example, not a constant.

6. Typing text: clipboard paste beats raw keystrokes

Two tempting options usually don't work well for anything beyond a couple of characters:

• A ref-based "type into this field" command still needs a resolvable ref, so it inherits the same
  AMBIGUOUS_TARGET problem as click/find on the apps described in section 3.
• A "press this key combo" command usually sends one combo per call with no bulk-text mode, so
  typing a sentence means one call per character — technically possible, practically painful and
  slow.

The pattern that reliably works: click to place focus/caret with a verified coordinate, write the
text to the clipboard, then send a paste combo (ctrl+v / cmd+v):

agent-desktop mouse-click --xy "X,Y" --headed
agent-desktop clipboard-set "the text to type"
agent-desktop press "ctrl+v" --headed

Two things to check for every time you reuse a field this way:

• Does the field retain old content between uses? A search box that still has a previous
  query in it will silently concatenate your new paste onto the old text and match nothing.
  Explicitly select-all (ctrl+a / cmd+a) before pasting into a field you've used before in the
  same session, rather than assuming a click clears it.
• Clean up the clipboard afterward if the text was sensitive. clipboard-set leaves content
  sitting in the system clipboard for any other app to read until something overwrites it.

7. Verify by re-observing, not by trusting the action's own result

A physical click or keystroke cannot be verified by the tool itself the way a semantic action can
(there's no "did this actually land" readback for raw input). Expect results like
delivered_unverified or disposition.retry: unsafe, and treat them exactly as labeled: the
input was sent, but whether it had the intended effect is unknown until you check.

After every physical action, re-observe the real state (a fresh screenshot, or a semantic read
where one is still available) before deciding what to do next:

• Confirm you searched/selected the thing you meant to, not a near-duplicate.
• Confirm a typed value landed in the field you intended, not whatever had focus.
• For anything that could trigger a side effect (sending a message, submitting a form, deleting
  something), confirm the side effect did not happen when it wasn't supposed to — e.g. the
  most recent item in a list/feed/history is unchanged from before your action, not just that your
  typed text appears in a box.

Never retry a physical action "just in case" when its result wasn't clearly not_delivered/safe.
A second click can double-click, double-submit, or toggle something back off.

8. Hard rules for anything with side effects

If the automated app can send, publish, delete, or otherwise change state visible to other people
or systems (a messaging app, a form, a destructive admin action):

• Never trigger the "commit" action (Send, Submit, Delete, Confirm) unless a human has explicitly
  approved that specific action, not just the general task.
• Never press Enter in a field whose Enter key submits something, as a way to "confirm" that text
  was typed — check by re-observing instead.
• Keep the target app open on the relevant screen afterward so a human can review and take the
  final action themselves, rather than closing it once your part is done.
• Treat "the text is in the box" and "the thing was sent" as two different states that must be
  verified separately (see section 7); don't conflate them.

Quick checklist

• Read the platform skill and --help output before issuing commands.
• Quote every @ref and every x,y pair if you're in PowerShell.
• Prefer refs; fall back to coordinates only for apps showing duplicate-tree symptoms.
• Don't rely on focus-window or focus timing; tile windows so geometry makes stray clicks
  impossible.
• Recompute coordinates from a fresh screenshot every time the layout could have changed.
• Type nontrivial text via clipboard + paste, clearing any pre-existing field content first.
• Re-observe after every physical action instead of trusting its own result.
• Never trigger a send/submit/delete action without explicit human approval of that exact
  action.