Headed input on Windows: foreground guard, per-line wheel events, and screenshot regions

[ruizemanuel]

Four suggestions for --headed input on Windows, from long sessions with agent-desktop 1.2.0 on Windows 10 22H2. The target app was FL Studio. Its UI is custom-rendered, which the Windows troubleshooting reference lists as out of scope, so the sessions ran on headed coordinate input. None of the suggestions depends on that app: each applies to headed pointer or keyboard input in general.

Refuse headed input when the wrong window is in front

A headed coordinate click lands on whatever window is on top at that point when SendInput runs, and press goes to the foreground window. During one session the user brought another full-screen window to the front between two commands, so a click at that moment would have landed in that window. The workaround was to check is_focused in list-windows before every headed action, which still leaves a gap between the check and the input.

A guard evaluated right before delivery would close that gap:

agent-desktop --headed mouse-click --xy 960,632 --require-foreground FL64.exe
agent-desktop --headed press ctrl+s --require-window w-2230392

When a different window is in front, the command would return not_delivered with retry: safe. The closest existing check is --expected-app on the notification commands (src/cli_args/notifications.rs).

Send one wheel event per line

mouse-wheel --dy 7 posts a single wheel event with a delta of 840, because the Windows adapter packs up to MAX_LINES_PER_EVENT (10) lines into each event (post_wheel_chunks and signed_chunks). A standard physical wheel sends one 120-unit event per notch, so controls that count messages instead of reading the delta move only once. In FL Studio, a knob moved one step for --dy 7, while a preset list that reads the delta scrolled several rows for --dy -12.

Posting one event per line, by default or behind a flag, would make --dy N behave like N notches of a physical wheel.

Capture a region with screenshot

When the accessibility tree has nothing to act on, screenshots are the verification path, and a full 1920×1080 capture costs an agent far more context than the area it needs to check. ScreenshotArgs accepts an app or window scope, a display index, and an output path (src/cli_args/mod.rs), so every capture in these sessions was cropped with a separate script. A --region x,y,w,h option in the same coordinate space as --xy would remove that step.

Document coordinate quoting in PowerShell

The Windows skill explains that PowerShell reads a bare @token as splatting. It doesn't mention that an unquoted --xy 960,632 becomes an array, so the binary receives two arguments and fails with unexpected argument '632'. #545 describes the same trap. A sentence next to the ref-quoting note, or accepting --x and --y on mouse-click the way mouse-wheel already does, would prevent it.