Skip to content

fix(agent): verify consequential single-round tool executions before final answers - #8441

Draft
AmirrezaFarnamTaheri wants to merge 3 commits into
tinyhumansai:mainfrom
AmirrezaFarnamTaheri:upstream/verify-consequential-tool-execution-20261027
Draft

AmirrezaFarnamTaheri wants to merge 3 commits into
tinyhumansai:mainfrom
AmirrezaFarnamTaheri:upstream/verify-consequential-tool-execution-20261027

Conversation

@AmirrezaFarnamTaheri

@AmirrezaFarnamTaheri AmirrezaFarnamTaheri commented Oct 27, 2026 •

Copy link
Copy Markdown

Summary

  • Extend the root orchestrator's final verification trigger to consequential tool executions even when a turn uses fewer than five tool rounds.
  • Require evidence after file writes, patch application, Git operations, workflow execution, or an actually started delegated child.
  • Distinguish executed, side-effecting structural modifications from read-only previews, refused calls, and ordinary reads.
  • Strengthen the harness instruction to require checks against the final exact commit and observable process exit status.
  • Add regression coverage for mutating/delegated tools, read-only and refused paths, and existing multi-round/todo triggers.

Problem

The existing verify_before_finish policy only triggers after five tool rounds or any todo call. A consequential single-round edit or delegation can therefore be declared done without independent verification. The TinyAgents FinishActivity contract already distinguishes requested tools from executors that actually started, and from operations with side effects.

Solution

Use the existing upstream-pinned TinyAgents FinishActivity::{executed,executed_with_side_effects} accessors. Trigger only for enumerated consequential executions, for delegate_* executions, or for started structural writes with side effects. Preserve the root-only orchestrator guard and the linked wrap-up budget suppression. No additional tool is installed or new protocol field introduced.

CI recovery and upstream-base reconciliation

The initial Rust gates-off run measured 343/341 packages (312 names, 3 native) because the then-current upstream main resolved two zstd and two zstd-safe versions. An interim branch-only dependency unification removed those duplicate resolutions, but upstream subsequently merged #8457, which resolved the same version split using the newer zstd 0.14 line.

To avoid reintroducing stale dependencies or a conflicting lockfile, this PR has now been rebased onto upstream main 9d5a06c, preserving only the consequential-execution policy and its tests. The original two-file behavior change is retained; Cargo.toml, Cargo.lock, and the dependency-budget ceiling now match upstream main exactly. No policy gate was disabled or raised. Hosted CI on this exact rebased head remains required.

Submission Checklist

  • Tests added or updated: new single-round mutation/delegation cases and preview/refusal/read-only controls in verify_before_finish_tests.rs.
  • Diff coverage ≥ 80%: pending hosted coverage; not claiming a measured number.
  • Coverage matrix updated: N/A — behavior-only policy change; no new/removed feature row.
  • All affected feature IDs under ## Related: N/A — no feature row added/removed; existing verify-before-finish policy.
  • No new external network dependencies introduced; the dependency graph and lockfile match current upstream main.
  • Manual smoke checklist updated: N/A — no release-cut or platform packaging surface changed.
  • Linked issue via Closes #NNN: N/A — no confirmed matching issue; this is a targeted fork-derived improvement.

Impact

  • Runtime: root orchestrator final-answer verification may run on consequential one-round turns. Read-only and refused attempts remain unchanged, and delegated subagents are not separately checked.
  • Compatibility: no RPC, storage, dependency, or provider changes. The policy uses the existing TinyAgents FinishActivity::{executed, executed_with_side_effects} API; the rebased upstream main supplies that contract.
  • Tradeoff: additional model calls only for consequential started actions, still constrained by existing wrap-up and time budgets.

Related

  • Closes: N/A — no tracked issue.
  • Follow-up PR(s)/TODOs: consider independent audit of the native tool-name inventory and benchmark validation before broadening trigger names.

AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

  • Key: N/A — no Linear issue supplied.
  • URL: N/A — no Linear issue supplied.

Commit & Branch

  • Branch: AmirrezaFarnamTaheri:upstream/verify-consequential-tool-execution-20261027

  • Commit SHA: 1c2518deffd53c4c6a5e0e614d520c8e94bed206

  • Upstream sync: merged tinyhumansai/main at 546cb2b without overlapping source changes; hosted checks rerun on this merge head. The prior Rust coverage lane was cancelled before producing a complete result, so no passing coverage is claimed.

Validation Run

  • pnpm --filter openhuman-app format:check — N/A: no frontend files.
  • pnpm typecheck — N/A: no TypeScript files.
  • Focused tests: cargo test -p openhuman --lib verify_before_finish — pending exact-head hosted verification.
  • Rust fmt/check: cargo fmt --all --check, cargo check -p openhuman — pending.
  • Tauri fmt/check: N/A — no Tauri files.

Validation Blocked

  • command: local Cargo execution and exact-head hosted CI.
  • error: the current work environment has no network checkout of the fork; hosted GitHub CI is the source of validation.
  • impact: no local pass is claimed; PR remains draft pending checks.

Behavior Changes

  • Intended behavior change: require a final independent check after started mutating/delegated work even on short turns.
  • User-visible effect: fewer unverified “done” answers after meaningful actions, with read-only tasks unaffected.

Parity Contract

  • Legacy behavior preserved: existing multi-round/todo triggers, subagent exclusion, wall-clock and wrap-up budget interaction.
  • Guard/fallback/dispatch parity checks: regression tests cover successful execution, read-only previews, and refused calls.

Duplicate / Superseded PR Handling

  • Duplicate PR(s): none identified for this behavior.
  • Canonical PR: this PR.
  • Resolution (closed/superseded/updated): N/A.

@coderabbitai

coderabbitai Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Rebased onto upstream/main (including v3.1.3 and shared zstd unification); preserve only the scoped policy and regression tests. No lockfile or dependency-ceiling changes.

AmirrezaFarnamTaheri commented Oct 27, 2026 •

Copy link
Copy Markdown
Author

CI validation note for the current draft head 3e5ceb8705ee: in CI Fast run #40557971228, the checks and rust-lint lanes passed, but Rust coverage job 122379189316 was cancelled, not passed. The overall CI gate therefore cannot be treated as green. I tried the GitHub Actions rerun endpoint for that job, but this fork contributor's integration receives HTTP 403 (Resource not accessible by integration). Could someone with upstream Actions write access re-run the cancelled coverage job on this exact head? I'll keep the PR in draft until its coverage and diff-coverage gates are verifiably green.

Update — October 28, 2026: The rerun request above applied to an older head, not the current PR. The refreshed head is d93ed039b426 and CI Fast #40664524154 has been triggered automatically. PR Quality has passed; Rust coverage is still running at this inspection. No manual rerun of the old cancelled job is needed. Keep this draft until the current CI gate is green and review acceptance is complete.

Merge latest upstream changes without changing the independently tested verification logic.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant