Run split harnesses on Prime VMs beside the workspace by hallerite · Pull Request #2803 · PrimeIntellect-ai/verifiers

hallerite · 2026-10-07T23:50:10Z

Based on main, which now includes the #2910 native-services stack (#2910, #2918, #2999); followed by #2831 and #2850. Companion: PrimeIntellect-ai/nano-rlm#218.

Tool execution shares the harness's VM, so the agent loop and the model credentials live where the agent's commands run. With deployment.mode = "split", the harness runs on a Prime VM of its own that joins the rollout's private LAN, and its tools execute in the workspace.

Placement: the workspace is placed exactly as in colocated mode, so plain images, ServicesSpec services, native Compose services and tool server VMs all split the same way. deployment.harness sizes the harness VM (1 CPU and 2 GB by default); it shares the workspace's labels, region and creation rate.
Network: the harness VM may open connections to the workspace, to tool server VMs and to task services that host tool servers (ToolsetConfig.service), and nothing may open one to it. It takes the task's network policy plus the model and tool routes; the workspace keeps no route to the model.
Shared LAN rules (changes Run Prime task services as native VMs and fix Harbor setup #2910's code): the ServicesSpec interconnect rule from Run Prime task services as native VMs and fix Harbor setup #2910 changes from * → * to tag:vf-task → tag:vf-task, and every member except the harness, main included, carries vf-task. Without a harness VM this is the same connectivity; with one, it keeps task services from reaching the harness. The native Compose per-network rules are unchanged, and the harness rule is added beside them.
Execution: ExecutionConnection.serve starts a worker in the workspace, as the workspace's agent user. The worker binds a port the kernel picks and announces it, so there is no port to choose or collide on. It carries a VF_EXEC_ID marker in its environment that every process its tools start inherits, through detaching and reparenting. Closing the harness session kills every process carrying the marker, as the worker's own user, so the cleanup needs no root and reaches nothing that user could not signal itself.
RLM: native tools and builtin skills run through nano-rlm's token-authenticated TCP worker, and so do the supervisor's shell jobs, services and path watches (rlm.shell, rlm.services, rlm.watch.path): nothing is refused, and the agent gets the same runtime prompt as in colocated mode. The harness and worker installs run concurrently.
Rejected before provisioning: harnesses without split support, custom skills, colocated task tools, borrowed runtimes and non-Prime workspaces.

This replaces the earlier nested-Docker design (a Docker workspace inside the harness VM). That design depended on the Docker-in-VM support that #2910 removes, and it pulled the task image inside every rollout's VM.

Validation

CI: a new Split Execution integration job runs tests/integration/test_split_execution.py over the harness grid. Split deployment is a grid capability, so every harness with SUPPORTS_SPLIT_EXECUTION gets a Prime cell (RLM here; Codex from Support Codex native tools in split deployments #2850), and the others are collected as skips with their reason.
Unit suite, ruff and ty pass.
Live Prime runs with a scripted tool-calling model: RLM's tools ran in the workspace (the workspace's files, working directory and environment; no credentials in the workspace's environment) and scored 1.0, including with the workspace's egress denied.
LAN reachability on real VMs for split plus ServicesSpec, split plus a tool server VM, and split plus native Compose: the harness reaches the worker in the workspace; the workspace, services and tool VMs cannot reach the harness; the harness cannot reach task services; the workspace still reaches its Compose peers by name.
Worker lifecycle on real VMs: a worker as a non-root user comes up in 0.3 s on its announced port, and after close the worker and a detached setsid child are gone; a root process carrying a copy of the marker survives that user's cleanup; a crashing worker fails in 0.3 s with its log.
Each tool call crosses the LAN: about 25 ms round trip in these runs, and up to about 200 ms between distant regions.

Before merge: nano-rlm#218 must merge first. The RLM pin currently points at that PR's head (efac509, rebased onto nano-rlm main), and moves to its merge commit afterwards.

🤖 Generated with Claude Code

Note

High Risk
Changes authentication placement, Prime LAN rules, and rollout provisioning/teardown paths; regressions could leak credentials into the workspace or break network isolation.

Overview
Adds deployment.mode = "split" so the agent loop and model credentials run on a dedicated Prime harness VM while tool execution stays in the task workspace over the rollout private LAN.

Config and guards: DeploymentConfig gains split plus harness sizing; split requires a Prime workspace, harnesses must opt in via SUPPORTS_SPLIT_EXECUTION / setup_execution(), and Agent rejects borrowed runtimes, colocated task tools, provision(), and non-colocated agentic judges before boot.

Infrastructure: Private-network LAN rules add a one-way harness tag; split provisioning boots a second VM, moves model/tool routes to the harness, strips them from the workspace, and exposes ExecutionConnection (public API) to start workspace workers on announced ports and tear them down via VF_EXEC_ID.

Rollout: Setup runs harness setup() and setup_execution() in parallel; interception, tool serving, and sessions bind to the harness runtime; traces record harness_runtime separately from the workspace.

RLM: Enables split with a TCP execution worker in the workspace, bumps the default nano-rlm ref, and disallows external skills in split mode.

Tests/docs: New integration fixture and grid tests (plus preflight on run / interaction), a CI Split Execution job, and docs/runtimes.md coverage.

^{Reviewed by Cursor Bugbot for commit 95fab65. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add split deployment support: run harness on a separate Prime VM beside the workspace

Introduces split deployment mode where the harness runs on its own Prime VM while tools execute in the workspace VM. DeploymentConfig.mode now accepts split and carries a separate harness PrimeConfig (deployment.py).
Adds ExecutionConnection, which starts an announced workspace worker and kills marked processes on close, and exports it from verifiers.v1 (runtimes/deployment.py, init.py).
The private-network strategy provisions the harness backend beside the workspace, gives the harness the execution routes, and stops the harness VM during verifier preparation (strategies.py). The rollout runs harness setup and setup_execution() concurrently and serves tools against the workspace runtime (rollout.py).
RLMHarness declares split support, starts an RLM execution worker in the workspace via ExecutionConnection.serve(), and includes connection details in runtime metadata; external skills are rejected in split mode (harness.py).
Adds integration coverage: a fixture taskset, a grid-parameterized test asserting distinct harness/workspace runtime IDs and reward 1.0, a preflight test rejecting unsupported configs, and grid capability checks for the split feature (test_split_execution.py, grid.py). Also documents split deployment in runtimes.md.
Behavioral Change: unsupported split configurations now fail fast. Agent._rollout_params rejects harnesses without SUPPORTS_SPLIT_EXECUTION, borrowed runtimes, and colocated task toolsets; Agent.provision and AgenticJudgeEnv reject split mode; _deployment_strategy requires a Prime workspace backend. RLMHarnessConfig changes the default nano-rlm version ref, and AgentInfo gains a harness_runtime field.

^{Macroscope summarized 95fab65.}

An existing but malformed reward.json currently falls back to reward.txt, which can turn an invalid verifier result into a passing score. Honor JSON precedence whenever the file exists: malformed, unreadable, oversized, non-object, or non-finite JSON fails grading. Text fallback is allowed only when JSON is absent. Stale reward files are cleared before verification, and a valid score remains authoritative even if test.sh exits nonzero. Add task.reward_key (default: reward) to select the scalar from named verifier outputs. Missing selected keys fail explicitly instead of implicitly summing dimensions. Unselected named values remain trace metrics. Shared and separate grading use the same selection path; reward.txt defines the reward key. Stack: #2806 → #2807 → this PR. This PR targets codex/harbor-verifier-resolution and remains independent of the deployment refactor (#2803). Validation before removing the PR-added e2e tests: full suite with Harbor installed: 152 passed, 131 skipped. All 18 Harbor-focused checks passed, including real Docker verifier scripts covering JSON/text precedence, missing files, malformed/empty/scalar/string/bool/non-finite JSON, oversized and unreadable files, named selection, stale-file cleanup, nonzero exits, and separate image-owned grading. Documented TOML validated. Ruff, all pre-commit hooks, and pre-push CI-parity type checking passed. Additional validation on real Prime VMs at combined-stack commit 4d82a7d: all 16 verifier/reward checks and 5 complete HarborEnv episodes passed their assertions. Episodes cover shared and fresh grading, valid named rewards, deliberate invalid-JSON failures with reward 0, and Compose with sidecar artifact restoration into a fresh grader. Cleanup audit found zero remaining labeled VMs. Tests used a deterministic harness without model calls; the dedicated custom-image/bundled-tests case was validated on Docker only. These Prime results apply to the combined stack, not independent runs of each intermediate branch. No test changes are included in this stack. The integration validation above was completed against identical production code. Current combined-stack contributor checks: 134 passed, 131 skipped; Ruff and all pre-commit hooks passed.    > [!NOTE] > ### Enforce Harbor reward precedence and add explicit `reward_key` selection > - Makes an existing `reward.json` authoritative: invalid, unreadable, or malformed JSON now fails grading, and `reward.txt` fallback only runs when the JSON file is absent. See [taskset.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2808/files#diff-2bff5dabcb9a41f4d830e8bbb890e8cc6f5b8331ef2ac4c3b19ffad0e046fe00). > - Adds `HarborTaskConfig.reward_key` (default `reward`) to select which named verifier value becomes the training reward; all other named values are kept as trace metrics. > - `reward.json` now accepts only a string-to-float mapping; scalar text rewards from `reward.txt` are validated separately as finite floats. > - A nonzero verifier exit no longer overrides a valid reward file; a missing selected key fails grading with a `TaskError`. > - Risk: scalar JSON rewards that previously passed validation now raise `TaskError` in `HarborTask.run_verifier`; reward files read through `read_reward_json` no longer fall back to text on validation failure. > >  > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 4c2f4da.</sup> >  >

cursor

Stale Bugbot comment from a previous run.

macroscopeapp · 2026-10-12T07:51:17Z

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — Adds opt-in split harness execution on Prime VMs with private-network workspace tools. An unresolved cleanup concern could leave tool processes active during evaluation on images without bash; the cleanup requirement or image compatibility needs resolution.

^{You can add or adjust custom eligibility rules. Learn more.}

macroscopeapp · 2026-10-12T09:19:02Z

Macroscope has since reviewed this pull request. An earlier review was skipped by a cost limit; a review has now completed, so that notice no longer applies.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

With deployment.mode = "split", the harness runs on a Prime VM of its own that joins the rollout's private LAN, and its tools execute in the workspace. The workspace is placed exactly as in colocated mode, so plain images, ServicesSpec services, native Compose services and tool server VMs all split the same way. - The harness VM may connect to the workspace and to tool server VMs; nothing may connect to it. It takes the task's network policy plus the model and tool routes, and the workspace keeps no route to the model. - ExecutionConnection.serve starts a worker in the workspace that listens on the LAN, in a cgroup of its own; closing the harness session kills the worker and every process its tools started. - RLM runs its native tools and builtin skills through nano-rlm's TCP worker. - Harness and worker setup run concurrently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A workspace service that already listened on the worker's random port made the readiness probe succeed for the wrong process. The worker is now ready only once a process in its cgroup listens on the port; a port another process holds before the worker is released is skipped for a fresh one, up to three times. Setting deployment.harness.creates_per_min to 0 no longer inherits the workspace's rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Split deployment accepted tool servers placed in a ServicesSpec or Compose service, but the LAN let the harness connect only to the workspace and to tool server VMs, so those tools failed after provisioning. The rollout now passes the services that host tool servers to the deployment, and the harness may connect to those services, not to the others. Pin nano-rlm to its split-execution head, which stops kernel commands with the kernel and keeps worker interrupts off the supervisor's event loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Root read the worker's PID from a file the worker's user can write, moved that process into the worker's cgroup, and could kill it by that PID on cleanup, so a forged PID file could have aimed root at any process. Root now acts only on a PID it has checked to belong to the worker's user and to carry this launch's release flag; cleanup kills the cgroup, or that checked PID after checking it again, and never reads the file. The release flag lives in a root-owned directory, and the worker's log is read as the worker's user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Split deployment is a grid capability: the Split Execution job runs one Prime cell per bundled harness, and a harness without SUPPORTS_SPLIT_EXECUTION is collected as a skip with its reason, so a harness that opts in is tested without a test change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The worker's launch had grown into root-run shell that held the worker, moved it into a cgroup, matched its listening socket through /proc, and checked its identity against a PID file the worker's user could write. Two choices caused all of it: picking the worker's port, and cleaning up through a cgroup, which needs root. The worker now binds a port the kernel picks and announces it, so there is no port to choose or collide on. It runs with a VF_EXEC_ID marker in its environment, which every process its tools start inherits through detaching and reparenting; close() kills the processes carrying it as the worker's own user, so the cleanup reaches nothing that user could not signal itself. A process that empties its environment escapes it; the workspace VM still ends with the rollout. Pin nano-rlm to the head that announces its bound port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor

Stale Bugbot comment from a previous run.

The head stops native shell commands with their turn and session, and names the worker's history in compaction notices under split execution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A worker that crashes right after announcing its port leaves both its listen line and the exit marker in the log by the first read; serve returned the port. Check for the exit marker first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The head syncs the session ledger to the execution worker incrementally, so long split sessions no longer outgrow one 16 MiB frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The head runs the supervisor's shell jobs, services and path watches in the workspace, so split execution refuses none of them and keeps the runtime prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 95fab65. Configure here.}

cursor · 2026-10-31T13:32:33Z

+        workers, self._workers = self._workers, []
+        for runtime, marker in workers:
+            with contextlib.suppress(Exception):
+                await runtime.run(["bash", "-c", _STOP, "bash", marker], {})


Worker cleanup requires bash

Medium Severity

ExecutionConnection.close runs _STOP with bash, and _STOP needs bash-only read -d '', while serve starts the worker with sh. Task images are only required to provide sh. When bash is missing, close swallows the error, so the worker and every process that inherited VF_EXEC_ID stay alive through finalize and in-workspace scoring.

Additional Locations (1)

verifiers/v1/runtimes/deployment.py#L47-L54

^{Reviewed by Cursor Bugbot for commit 95fab65. Configure here.}

This was referenced Oct 7, 2026

feat: separate persistent tool execution from the agent loop PrimeIntellect-ai/nano-rlm#218

Open

fix: retain explicitly scored execution timeouts in training PrimeIntellect-ai/prime-rl#3930

Draft

hallerite changed the title ~~feat: separate RLM harness and task execution on Prime VMs~~ Unify deployments, split tool execution, and Harbor task integration Oct 8, 2026

hallerite force-pushed the codex/split-execution branch from c0faec4 to 8734f52 Compare October 8, 2026 02:20

This was referenced Oct 8, 2026

Align Harbor with 0.24.0 and reject multi-step tasks #2806

Merged

Honor Harbor separate-verifier image and resource resolution #2807

Merged

Honor Harbor reward precedence and explicit scalar selection #2808

Merged

hallerite force-pushed the codex/split-execution branch 3 times, most recently from 76edaa1 to 83bd375 Compare October 9, 2026 08:05

hallerite marked this pull request as ready for review October 12, 2026 07:46

macroscopeapp Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/grading.py Resolved

Comment thread verifiers/v1/runtimes/compose.py Resolved

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/runtimes/container_target.py Resolved

Comment thread verifiers/v1/runtimes/compose.py Resolved

cursor Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/tasksets/harbor/taskset.py Resolved

hallerite force-pushed the codex/split-execution branch from 83bd375 to 8e2eeb5 Compare October 12, 2026 09:18

cursor Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/rollout.py Resolved

hallerite mentioned this pull request Oct 12, 2026

Separate public Runtime views from provider lifecycle backends #2827

Merged

hallerite force-pushed the codex/split-execution branch from 8e2eeb5 to 96ece3f Compare October 12, 2026 09:29

hallerite changed the base branch from main to codex/runtime-handles October 12, 2026 09:29

macroscopeapp Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/compose.py Resolved

hallerite force-pushed the codex/split-execution branch from 96ece3f to 4a8cf93 Compare October 12, 2026 09:42

cursor Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/rollout.py Resolved

Comment thread verifiers/v1/runtimes/deployment.py Resolved

hallerite marked this pull request as draft October 12, 2026 09:58

hallerite force-pushed the codex/split-execution branch from 4a8cf93 to 7546c1b Compare October 12, 2026 13:07

hallerite marked this pull request as ready for review October 12, 2026 14:29

cursor Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/grading.py Resolved

cursor Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/rollout.py Resolved

macroscopeapp Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/strategies.py Resolved

hallerite force-pushed the codex/split-execution branch 2 times, most recently from 002c973 to ecb006b Compare October 14, 2026 16:24

This was referenced Oct 15, 2026

Run task tool servers in a named Compose service #2872

Merged

Run tool servers on Prime runtimes of their own as VMs on the rollout LAN #2876

Merged

Nodes: one placement model for a rollout (RFC 0001, task side implemented) #2890

Open

macroscopeapp Bot reviewed Oct 29, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/deployment.py Resolved

Comment thread verifiers/v1/runtimes/strategies.py Resolved

cursor Bot reviewed Oct 29, 2026

View reviewed changes

Comment thread verifiers/v1/agent.py Resolved

macroscopeapp Bot previously approved these changes Oct 30, 2026

View reviewed changes

macroscopeapp Bot reviewed Oct 30, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/deployment.py Resolved

macroscopeapp Bot previously approved these changes Oct 30, 2026

View reviewed changes

mikasenghaas previously approved these changes Oct 31, 2026

View reviewed changes

hallerite and others added 6 commits October 31, 2026 01:05

cursor Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/deployment.py Resolved

Pin nano-rlm to its split-execution head

8528fbf

The head stops native shell commands with their turn and session, and names the worker's history in compaction notices under split execution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

macroscopeapp Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved

hallerite and others added 3 commits October 31, 2026 01:38

Pin nano-rlm to its split-execution head

4c1fc8a

The head syncs the session ledger to the execution worker incrementally, so long split sessions no longer outgrow one 16 MiB frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop the execution worker's unit test

4a9c28a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

macroscopeapp Bot previously approved these changes Oct 31, 2026

View reviewed changes

Pin nano-rlm to its split-execution head

95fab65

The head runs the supervisor's shell jobs, services and path watches in the workspace, so split execution refuses none of them and keeps the runtime prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor Bot reviewed Oct 31, 2026

View reviewed changes

Conversation

hallerite commented Oct 7, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validation

Add split deployment support: run harness on a separate Prime VM beside the workspace

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

macroscopeapp Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 31, 2026

Choose a reason for hiding this comment

Worker cleanup requires bash

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

hallerite commented Oct 7, 2026 •

edited by macroscopeapp Bot

Loading

cursor Bot left a comment •

edited

Loading

macroscopeapp Bot commented Oct 12, 2026 •

edited

Loading

macroscopeapp Bot commented Oct 12, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading