[faresobeid]

Problem

Branch.mm_token_type_ids marks image tokens by token id (Trace.mm_token_type_id_map). _project_prompt_attribution aligns renderer and vLLM tokens the same way. A sampled <|image_pad|>, or one in tool text, becomes an extra image run. prime-rl then fails with images=2, runs=3, or #3949 drops the branch.

Design

• MessageNode.mm_placeholders: {modality: [(offset, length)]} stores node-relative ranges over the node's (vLLM-expanded) token_ids. It is filled at commit from vLLM's returned ranges, or from the renderer's ranges when nothing was expanded. These are errors:
  • a range that crosses a message boundary;
  • a reused prefix node whose stored ranges disagree with this turn's ranges.
• Branch.mm_placeholders concatenates the node ranges. Branch.mm_token_type_ids is derived from it (1 = image, 2 = video).
• _project_prompt_attribution uses the renderer's logical placeholder ranges (TurnTokens.renderer_mm_placeholders) instead of np.isin(ids, map).
• Bridges pass the prefix's logical placeholder ranges (PendingTurn.previous_mm_placeholders) to the renderer as previous_multi_modal_data.
• Trace.mm_token_type_id_map, TurnTokens.mm_token_type_id_map and the delta scalar field are removed.

Validation

• tests/unit/test_graph.py: new test where a sampled pad in the completion and a literal pad in tool text before the tool's image leave only the placed image as a placeholder. The existing expansion test was updated.
• pytest tests/unit: all pass except test_api::test_config_tree_imports_without_the_runtime, which also fails on the base commit in the prime-rl venv (it picks up sitecustomize / nvidia_cutlass_dsl from that environment).
• Ruff is clean.

Notes

• The branch is based on the commit prime-rl pins (60290d4), because verifiers main now requires prime-runs 0.1.5, which prime-rl's lock does not admit yet. It merges cleanly into main: none of the touched files changed since the pin.
• Requires renderers feat: report media placeholder positions in logical prompts; drop mm_token_type_id_map renderers#211.

Merge order: renderers PrimeIntellect-ai/renderers#211, then this PR, then prime-rl PrimeIntellect-ai/prime-rl#4343.