Skip to content

feat(firefox): snapshots, refs and element commands on Firefox - #2206

Open
lahfir wants to merge 55 commits into
laf/bidi-engine-launchfrom
laf/bidi-snapshot-refs-input
Open

lahfir wants to merge 55 commits into
laf/bidi-engine-launchfrom
laf/bidi-snapshot-refs-input

Conversation

@lahfir

@lahfir lahfir commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor
  1. Firefox sessions could not take snapshots or act on elements, so the core agent workflow did not work.
  2. This adds snapshots with refs from Firefox's accessibility tree, plus click, fill, type, keys, queries and dialogs, reusing the shared page JS and snapshot renderer.
  3. Builds on feat(firefox): launch Firefox over WebDriver BiDi #2205; continued in feat(firefox): network control and emulation on Firefox #2207.

@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agent-browser Ready Ready Preview, v0 Nov 9, 2026 10:42pm UTC
agent-browser-env-demo Ready Ready Preview, v0 Nov 9, 2026 10:42pm UTC

@vercel-internal-security-reviewer

Copy link
Copy Markdown

Security review details

Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/firefox/actions.rs Resolved
Comment thread cli/src/native/firefox/snapshot.rs Resolved
Fold the destroyed-tab handling into the top_context model: a destroyed top-level tab drops its ids, labels, refs and documents, and the session falls back to a surviving tab. The getTree lookup stays as the fallback when no tab is left.
Comment thread cli/src/native/firefox/forms.rs Resolved
Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/firefox/input.rs Resolved
Comment thread cli/src/native/firefox/forms.rs Resolved
Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/snapshot.rs Resolved
…text insertion

check and uncheck fall back to a JS click on hidden checkboxes. hover, tap and drag skip the actionability check. Back and forward buttons use BiDi button codes 3 and 4. drag commits mouse state. Text insertion with no editable focus succeeds and drops the text.
Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/firefox/forms.rs Resolved
… fill

Frame points are mapped into the page viewport before they are stored, so later commands do not press at a stale coordinate. fill and type report dialogOpened when focusing opens a prompt.
Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/firefox/snapshot.rs Resolved
Comment thread cli/src/native/firefox/interact.rs Resolved
Comment thread cli/src/native/firefox/query.rs Resolved
let (mut tree_nodes, roots) = build_tree(nodes);
assign_refs(
&mut tree_nodes,
&HashMap::new(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Firefox can only call this with an empty cursor map, so cursor:pointer/onclick/tabindex elements never get refs; a clickable

with text can snapshot as "(empty page)". Reproduced.

.and_then(Value::as_i64)
.unwrap_or(1)
.max(0) as u32;
let actions = pointer_click(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

click, dblclick and drag never read inputMode or seed, so --human and --input-mode are silently ignored on Firefox even though click/drag are documented to move with a curve. Not reproduced.

let (mut tree_nodes, roots) = build_tree(nodes);
assign_refs(
&mut tree_nodes,
&HashMap::new(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Firefox snapshots pass an empty cursor-interactive map, so elements the AX tree exposes without an interactive role (a div with onclick, cursor:pointer, hidden radio labels) get no ref and cannot be targeted by ref. Reproduced.

}

/// Clicks a ref or selector, returning Chrome's `{clicked}` shape.
pub async fn click(cmd: &Value, state: &mut DaemonState) -> Result<Value, String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

click, dblclick and drag never read inputMode or seed, so --human/--input-mode are silently ignored and the pointer jumps to the target in one 0ms move, unlike Chrome. Reproduced.

This branch was successfully deployed

2 active deployments
Preview – agent-browser — ef2455bb Deployed Nov 9, 2026 by vercel[bot]
Preview – agent-browser-env-demo — ef2455bb Deployed Nov 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants