Repository navigation
Replies: 7 comments
|
Thanks @President, this is a clear write-up, and I am happy to support the two hooks. One small note from my side, mostly to say where I think the strongest case is: the hook as a record of where the assembled context goes and how big it is. model.provider and server.address on modelCallRequest are what no existing hook carries today, and the example from my earlier comment on #122, customer data only to the in-house model, needs just those two fields. So content off by default seems the right default to me, and cases 1, 2, 3 and 5 all work with it off. On the gateway question in the open points, perhaps one argument helps: a gateway sees the request but not the steps that came before it, and the Guardian sees the steps but not the request. The hook is what lets one place see both. The OTel GenAI names and leaving the provider mapping to the harness both make sense to me. Thanks for putting this together! |
|
@President, I checked this against the spec on ACS already has three checks the post doesn't count:
The post is right about these gaps:
Before this turns into a spec PR, I'd want these changes:
The proposal should also address these risks:
A few smaller points came out of the schema check:
The smallest version I'd back is No issue tracks this hook pair yet. Issue #122 asks for inspecting and rewriting the prompt, and issue #124 covers per-turn usage. Could you open a new proposal issue for the narrowed hook pair and link this discussion, issue #122 and issue #124? A proposal issue gives the leads one place to triage it. |
|
Thanks for the careful check! The smallest version at the end, Effectively, whether it gets a hook of its own or is simply included as additional info in the existing |
|
I agree with @rocklambros, I'm not persuaded here that query rewrite, embeddings, judge are not tool calls. Moreover, like @rocklambros mentioned, we do already have compaction. For memory and skills, they might merit their own hooks, but many harnesses don't support those as individual hooks and instead label them as tool calls. For a threat scenario where multiple models are used, I think allowing optional fields for In short, I don't think there's a sufficiently strong case made here that justifies the tradeoffs of the large spec changes necessary to introduce a new, separate hook to be supported, at a coarser grain than other lifecycle hooks. I believe ACS has a flexible enough hook vocabulary to cover the proposed lifecycle events. |
|
Thanks @President for the detailed write-up, and @MKaptein for the use case. I agree the gap is real, but I wouldn't make this required, for two reasons:
So here's what I'd suggest, with no new hook:
@rocklambros @afogel @President @MKaptein would this work for you? |
|
@bar-capsule yes, that shape works for me. I backed a narrow Four things I'd want in the proposal issue:
Landing it with the usage object from issue #124 on @President, since you wrote this up, would you open the proposal issue with Bar's shape and these four, and link this discussion, issue #122 and issue #124? That gives the leads one thing to triage. |
|
@bar-capsule yes, this shape works for me. |
Uh oh!
There was an error while loading. Please reload this page.
In short:
steps/modelCallRequestfires before a model call.steps/modelCallResultfires after it. Field names follow OpenTelemetry (OTel) GenAI. Message text is off by default.Where this comes from: #122, NVIDIA #402, the AGT reference implementation, #123 / #124
[Proposal] No preLLMCall hook to inspect/rewrite the assembled message context before LLM dispatch #122 (14 Sep, still open) asked for a hook before the model call so the Guardian could rewrite the prompt. @rocklambros explained why not. @rocklambros left [Proposal] No preLLMCall hook to inspect/rewrite the assembled message context before LLM dispatch #122 open for a reply and set a condition ("per-source provenance" is the record of where each input came from):
@MKaptein answered both parts in [Proposal] No preLLMCall hook to inspect/rewrite the assembled message context before LLM dispatch #122:
NVIDIA-labs Object Oriented Agents (NOOA) #402 (29 Sep, opened by @rocklambros):
The ACS reference implementation with Microsoft's Agent Governance Toolkit (AGT) lists the same gap (README):
In
mapping.yaml, both points are "unexpressed at every verdict".Cost ceilings were requested in discussion #123 and [Proposal] Per-turn model usage on the wire, so a Guardian can enforce cost ceilings #124. Both propose usage after the turn (
turnEnd). [Proposal] Per-turn model usage on the wire, so a Guardian can enforce cost ceilings #124 also notes: "The Guardian cannot estimate usage itself because it never sees the assembled prompt ([Proposal] No preLLMCall hook to inspect/rewrite the assembled message context before LLM dispatch #122)."Where hooks fire today
The picture shows one agent at work. ✅ marks a place where an ACS hook fires. ❌ marks a place where data leaves the agent and no ACS hook fires.
flowchart LR U1["User"] -->|"userMessage ✅"| H["Harness<br/>builds each request"] K[("Knowledge base / memory")] -->|"knowledgeRetrieval ✅<br/>memoryContextRetrieval ✅"| H H -->|"main call and its answer ❌"| M["Model provider<br/>external or in-house"] H -->|"helper calls ❌"| M2["Same or another provider:<br/>title, compaction, memory,<br/>query rewrite, embeddings, judge"] H -->|"toolCallRequest ✅<br/>toolCallResult ✅"| T["Tools"] H -->|"agentResponse ✅"| U2["User"]The 5 cases below are about the ❌ arrows.
5 cases
1. Where the data goes
Rule: "customer data goes only to our own model". ACS cannot check this rule: it does not see which model gets each call.
Details and code
The model can change on every call, not only once per session:
memories.extract_model, or else the provider's preferred extraction model (code).judge_model(code).What ACS has today:
sessionStarthas no model field.subagentStartnames a model per subagent, not per call.acs.model_idin the trace identifies the Guardian's own evaluator model.Missing: where each call goes, and a way for the Guardian to block the call. A request to an external provider crosses a trust boundary. Under principle 3(b), a field belongs on the wire when it "marks a real trust-boundary crossing".
2. Cost before the call
Rule: "stop the call that goes over budget". ACS cannot check this rule: it sees a turn only after the turn ends, when the money is already spent.
Details and code
max_tokenslimits only the answer.turnEndalready exists. Thestep_countfield is described as "useful for per-turn rate-limit policies and [...] resource accounting". BUTturnEndfires after the money is spent.estimated_input_tokens.preCompactalready uses that name for the region about to be compacted. In the proposed hook,estimated_input_tokensdescribes the outgoing request size.3. Hidden calls
Helper calls (title, search query, memory note, embedding, LLM judge) also send data to a model. No ACS hook fires for them.
Details and code
Some hooks see the results later:
knowledgeRetrievalfires with retrieval results. The query has already gone to the embedding provider.postCompactreceives the summary after the summarizer call.Missing: a check on the helper call itself.
4. Tool text in the system prompt
Some frameworks put text from a tool or from memory into the system prompt or a developer message. The model trusts these parts most. No ACS hook sees this.
Details and code
Policy: "text from tools or retrieval never goes into the system or developer role".
{key}placeholders in the agent instruction from session state (code). When the agent has nostatic_instruction, the completed instruction is added to the system instruction (code). A tool can write session state (ToolContext.state). A tool output can therefore reach the system instruction.toolCallResultsees the tool result.toolCallResultdoes not see the state write or the instruction.memoryContextRetrievalcarries the text and provenance. No hook identifies the role assigned to the text. ThememoryContextRetrievalpayload has onlyquery,memory_storeandresults.The text keeps its provenance, but it enters the instruction role. In ACS terms, untrusted content is then "treated as if it were attested" (trust.md). This happens inside the harness, where no hook sees the role. Principle 2, item 6, asks whether the wire matches "where boundary crossing and trust-laundering actually occur". This is one such place.
Existing protections:
preCompact/postCompact.The proposed hook covers case 4 only when content is on. With content on, the Guardian can deny the call or remove the text with
redactions. With content off, which is the default, the Guardian never gets the text.5. Which model answered
A provider can answer with a different model than the one the agent asked for. ACS does not record which model answered, and cannot block that answer.
Details
gen_ai.response.model.otel-mapping.json,ocsf-mapping.json) have no field for the answering model.Who has which hook: 13 frameworks, checked in source
Checked in source on 20-21 Oct 2026. Every code link is pinned to a commit.
llm_callmiddlewareclient+params)wrap_model_call,before_modelafter_modelbefore_model_callbackafter_model_callbackon_llm_start,call_model_input_filteron_llm_endagent.model(RunConfigcan override it)before_model_request,wrap_model_requestafter_model_requesttransformParams,wrapGenerate,wrapStreambefore_llm_callFalseblocks)after_llm_callOnPromptRenderAsyncIFunctionInvocationFilter(per function, not per model call)pre_model_callpost_model_callasync_pre_call_hookasync_post_call_success_hookA hook gives a framework a place to run a check. A hook does not automatically catch every model call:
llm_callmiddleware. The middleware sees the compaction call but cannot rewrite the compaction prompt (code).The NOOA example shows that a harness can route helper calls through a model-call hook.
Why the check belongs in ACS
Model calls can also be checked outside ACS: by an LLM gateway in the network, or by settings inside the agent framework (for example, a spending limit). But then ACS checks the other actions of the agent, and not model calls. Model calls get their own rules, somewhere else.
All places compared
sessionStart, prompt inuserMessage)userMessagewould carry instructions, history and tool output, not only user input.sessionStarthas no model; AgBOM is an inventory)server.addresscan be a proxy. The final provider behind it needs a defined way to learn itturnEnd)estimated_input_tokensor with the contentknowledgeRetrieval,postCompact)A compromised harness skips any hook. An LLM gateway is still useful as a second layer for that case, but it cannot replace the hook, because it does not receive the framework's context.
The proposal
steps/modelCallRequestbefore every model call,steps/modelCallResultafter it. Helper calls too.modificationsobject ACS already has.The details: decisions, fields, and the 2 messages in JSON
methods_implementedMUST fire both hooks for every model call, including helper calls.methods_evaluatedlist as allowed (handshake.json).toolCallRequest. "Modify" uses the existingmodificationsobject. No new wire element is needed:parameter_overrideschanges a call field. For example, an override can send the call to the in-house model or lowermax_tokens.redactionsreplaces text in a message part at a JSON pointer, a path to a value in JSON. For example,/content/input_messages/1/parts/0/responseidentifies the tool output in the example below. Redactions work only when content is on.operation: embeddings, field meanings come from the embeddings span. Each field takes its meaning from OTel or from an existing ACS hook, and messages use OTel's JSON format. The long OTel names appear only in the ACS trace mapping. The existing mapping fromtool.nametogen_ai.tool.namefollows the same pattern.steps/modelCallRequest, before the call{ "jsonrpc": "2.0", "method": "steps/modelCallRequest", "id": "42", "params": { "acs_version": "0.1.0", "request_id": "7c1e0000-0000-4000-8000-000000000001", "timestamp": "2026-10-21T15:04:05Z", "metadata": { "agent_id": "coding-agent", "session_id": "9f3a0000-0000-4000-8000-000000000003", "turn_id": "t-7" }, "payload": { "operation": "chat", "model": { "name": "gpt-4.1-mini", "provider": "openai" }, "server": { "address": "api.openai.com", "port": 443 }, "max_tokens": 1024, "temperature": 0.2, "stream": true, "estimated_input_tokens": 18234, "content": { "system_instructions": [ { "type": "text", "content": "Summarize the conversation..." } ], "input_messages": [ { "role": "user", "parts": [ { "type": "text", "content": "..." } ] }, { "role": "tool", "parts": [ { "type": "tool_call_response", "id": "call_1", "response": "..." } ] } ], "tool_definitions": [ { "type": "function", "name": "read_file" } ] } }, "signature": { "algorithm": "HMAC-SHA256", "value": "...", "key_id": "..." } } }steps/modelCallResult, after the call{ "jsonrpc": "2.0", "method": "steps/modelCallResult", "id": "43", "params": { "acs_version": "0.1.0", "request_id": "8d2f0000-0000-4000-8000-000000000002", "timestamp": "2026-10-21T15:04:07Z", "metadata": { "agent_id": "coding-agent", "session_id": "9f3a0000-0000-4000-8000-000000000003", "turn_id": "t-7" }, "payload": { "request_id_ref": "7c1e0000-0000-4000-8000-000000000001", "exit_status": "success", "model": { "name": "gpt-4.1-mini-2025-04-14" }, "response_id": "chatcmpl-123", "finish_reasons": ["stop"], "usage": { "input_tokens": 18190, "output_tokens": 412, "cache_read_input_tokens": 16000, "cache_write_input_tokens": 0, "reasoning_output_tokens": 0 }, "content": { "output_messages": [ { "role": "assistant", "parts": [ { "type": "text", "content": "..." } ] } ] } }, "signature": { "algorithm": "HMAC-SHA256", "value": "...", "key_id": "..." } } }request_id_reflinks the result to the request, as ontoolCallResult.model.namein the result identifies the model that actually answered (case 5)."exit_status": "failure"and anerror_type.Fields
operation(chat, embeddings, ...)gen_ai.operation.namemodel.providergen_ai.provider.namemodel.namegen_ai.request.modelserver.address,server.portserver.address,server.portmax_tokens,temperature,streamgen_ai.request.max_tokens,gen_ai.request.temperature,gen_ai.request.streamestimated_input_tokenspreCompact, applied to the outgoing requestcontent.system_instructions,content.input_messages,content.tool_definitionsgen_ai.system_instructions,gen_ai.input.messages,gen_ai.tool.definitionsrequest_id_ref,exit_statustoolCallResulterror_typeerror.typemodel.namegen_ai.response.modelresponse_id,finish_reasonsgen_ai.response.id,gen_ai.response.finish_reasonsusage.input_tokens,usage.output_tokensgen_ai.usage.input_tokens,gen_ai.usage.output_tokensusage.cache_read_input_tokens,usage.cache_write_input_tokens,usage.reasoning_output_tokensgen_ai.usage.cache_read.input_tokens,gen_ai.usage.cache_write.input_tokens,gen_ai.usage.reasoning.output_tokenscontent.output_messagesgen_ai.output.messagesQuestions we expect, with answers
sessionStartis enough." A fixed model is common for the main call. Helper calls and per-call routing can use different models (case 1).operationfield follows the pattern oftoolCallRequest, which covers every tool. The format for stating a call's purpose remains an open question below.userMessage."userMessage"Fires when external user input arrives", with "Provenance:origin: user_input". The assembled request also contains instructions, history and tool output. Treating the assembled request asuserMessagewould change the boundary between user input and the full model request.toolCallRequest?"toolCallRequestfires "after the framework has parsed a tool call from the LLM's output". A model call is not a tool call parsed from model output. Tool policies would also have to tell model calls apart from tool calls.methods_evaluated. The harness treats those methods as allowed. As @MKaptein noted in [Proposal] No preLLMCall hook to inspect/rewrite the assembled message context before LLM dispatch #122, a tool call already has 2 round trips. A model call itself usually takes seconds.Open questions
preCompactormemoryStore?provenance_producer: deterministic, where do the source records for each message part go?parameter_overridesis "keyed by argument name" of a tool.parameter_overridesneeds a rule for model call fields. A change early in the prompt also breaks the provider's prompt cache. Breaking the prompt cache increases the call's cost.Questions to decide
All reactions