EvalPort: a portable format for eval/judge.py's JudgeVerdict output (external adapter idea)

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange standard (Apache 2.0) for portable LLM evaluation test cases, graders, suites, and results, with a Python/TS SDK (evalport-sdk) and 37 real framework adapters shipped so far (CrewAI, AutoGen, Ragas, LangSmith, MLflow, DeepEval, and others). Came across src/kiro_crew/eval/ while looking at how persistent-workspace agents do cross-session eval, and the fit with your LLMJudge/JudgeVerdict is close enough that I wanted to raise it as an idea rather than just quietly build something.

The fit

src/kiro_crew/eval/judge.py has:

@dataclass
class JudgeVerdict:
    score: float
    reason: str

produced by LLMJudge.judge_turn(description, criteria, user_msg, assistant_msg), parsed from a judge LLM's {"score": ..., "reason": ...} JSON response.

EvalPort's llm_judge grader type is defined with exactly that shape as its structured-output schema:

{"type": "llm_judge", "params": {
  "schema": {
    "type": "object",
    "properties": {
      "score": {"type": "number"},
      "reason": {"type": "string"}
    },
    "required": ["score", "reason"]
  }
}}

(from docs/grader-reference/README.md). That's not a coincidence of naming — it's the same judge pattern (prompt a model, parse score + reason/reasoning back out) that shows up across most LLM-as-judge implementations, which is part of why EvalPort standardized on it.

On the scenario side, src/kiro_crew/eval/scenario.py's Scenario → Session → Turn → Assertion model (with AssertionType.CONTAINS/NOT_CONTAINS/REGEX/EQUALS/JUDGE and a judge_criteria field) maps fairly directly onto an EvalPort suite's test_cases + graders — contains/regex/equals are graders EvalPort already defines natively, and judge_criteria is the llm_judge grader's rubric.

Why I'm raising it, not just shipping an adapter

I could build a standalone kirocrew-openeval-adapter package the same way I've done for other frameworks — working against your public dataclasses from the outside, no dependency on this repo, no PR needed against kirocrew itself (that's the pattern in, e.g., crewai-openeval-adapter and ragas-openeval-adapter — both convert against the target library's public result shapes, no upstream merge required). But your eval harness is internal tooling for a specific purpose (multi-session memory/context-recall benchmarking), not a general-purpose SDK meant for external consumption, so before spending time on an adapter nobody asked for I wanted to check: is there any interest in kirocrew eval scenario results being exportable as a portable, spec-validated format — e.g. for comparing memory-recall scores across KiroCrew versions over time, or letting someone outside this repo replay a scenario against a different judge model without gluing anything together by hand?

Rough sketch (not a PR — just to make the mapping concrete)

# hypothetical kirocrew_openeval_adapter, using the real dataclasses
from kiro_crew.eval.scenario import Scenario, AssertionType
from kiro_crew.eval.judge import JudgeVerdict

def scenario_to_suite(scenario: Scenario) -> dict:
    graders = []
    if scenario.judge_criteria:
        graders.append({
            "id": "gr_judge",
            "type": "llm_judge",
            "params": {"prompt": scenario.judge_criteria},
        })
    test_cases = []
    for session in scenario.sessions:
        for i, turn in enumerate(session.turns):
            case_graders = [
                {"id": f"gr_{i}_{a.type.value}", "type": a.type.value, "params": {"value": a.value}}
                for a in turn.assertions if a.type != AssertionType.JUDGE
            ]
            if any(a.type == AssertionType.JUDGE for a in turn.assertions):
                case_graders.append("gr_judge")
            test_cases.append({
                "id": f"{scenario.name}.{session.name}.{i}",
                "input": turn.user,
                "graders": case_graders,
            })
    return {"version": "1.0.0", "id": scenario.name, "graders": graders, "test_cases": test_cases}

def verdict_to_result(verdict: JudgeVerdict, test_case_id: str) -> dict:
    # EvalPort grader scores are normalized to [0.0, 1.0]; JudgeVerdict.score is 1-5
    return {
        "test_case_id": test_case_id,
        "grader_id": "gr_judge",
        "score": (verdict.score - 1) / 4,
        "passed": verdict.score >= 3.0,  # matches LLMJudge's default pass_threshold
        "reason": verdict.reason,
    }

Every field name above (score, reason, judge_criteria, Scenario/Session/Turn/Assertion/AssertionType, pass_threshold) is read straight from judge.py and scenario.py as they exist in main today — I checked before posting this rather than guessing.

Happy to build the adapter as an independent package (no PR against this repo, no dependency added here) if there's any appetite for it, or to just leave this here as a "someone looked at this and the interop path is open" note if not. No obligation either way — genuinely just flagging the fit since it was unusually clean.

— Sahi, independent contributor (not affiliated with this project)