Side-Eye

Draft cheap, review expensive — a cheap model does the work; its output is sampled to an expensive Claude judge, asynchronously, outside the request path. On demand, /sideeye:ask sends the session you're in (or part of it) to the strong judge for an independent second opinion. The scoreboard answers "how much can we save at held quality?" (Pitch: docs/judge-sampling-pitch.md.)

Install (Claude Code)

/plugin marketplace add redhat-et/sideeye
/plugin install sideeye@sideeye

That's the whole install. The first time you run /sideeye:ask, the skill installs the sideeye engine itself (one-time, ~30s, via uv or pipx if present) — no terminal, no files to copy.

If you also want the CLI on PATH outside Claude Code (or your machine has no uv/pipx and you'd rather install once by hand):

uv tool install git+https://github.com/redhat-et/sideeye   # or pipx / pip install
pip install -e .                                             # development checkout

One-command setup

After installing the CLI, run:

sideeye setup

Setup asks once for the judge credential, stores it in the Side-Eye config with mode 600, detects Claude Code, Codex, and OpenCode, and configures the bundled OpenCode plugin when Node/npm and a supported OpenCode config are available. It preserves unrelated JSON and JSONC settings.

Use sideeye setup --no-opencode to configure only the judge route.

Runtime requirements (judge route) are in the section below.

Configure the judge route

The judge must hit a real Claude route — never the cheap model that drafted the session (a judge sharing the drafted-with model is the self-grading failure Side-Eye exists to catch). Resolution order:

1. --base-url flag
2. SIDEEYE_JUDGE_BASE_URL + SIDEEYE_JUDGE_API_KEY env vars
3. config file — ~/.config/sideeye/config.json (or $SIDEEYE_CONFIG), chmod 600:

   { "judge": { "base_url": "https://your-claude-route", "api_key": "sk-…" } }

4. ambient ANTHROPIC_API_KEY (direct Anthropic) or ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN — accepted only when the host is really Anthropic's; a gateway route needs "judge": {"trust_ambient_route": true} to opt in.

A key alone is enough: api_key without base_url (in the config file or as SIDEEYE_JUDGE_API_KEY) means "talk to Anthropic direct" (https://api.anthropic.com) — the common case for a user with an API key and no gateway. Set base_url only when the judge sits behind a gateway.

Check what you'd get, without spending: sideeye route (exit 0 = ready).

First run

sideeye route                      # must say "verdict: OK"

/sideeye:ask --validate                            # free: is the route ready? ($0)
/sideeye:ask is my last change sound?              # cents, scoped to recent turns
/sideeye:ask --all                                 # full-session review, median ~$0.75

Both spend real money. Observed on 20 dogfood reviews: advise ≈ $0.02–$1, review ≈ median $0.75, up to ~$3.50 on big sessions (defaults: Opus 4.8 for review, Fable 5 for advise; --model fable|opus|sonnet|haiku to choose, --max-cost N as a ceiling).

The engine is still POC-grade in places, but it runs for real: 165 offline tests, and the capture → judge → verdict → cost pipeline is verified on real sessions. It is plain Python glue — deliberately: the perf-critical hot path is the gateway (Rust/praxis); the judge worker is I/O-bound (one multi-second LLM call), so language choice is irrelevant off-path. See docs/sideeye-phase-c-questions.md for the design decisions this implements.

Layout

sideeye/
  rubric/rubric_v3.md            pair-grading rubric (Phase B)
  rubric/rubric_session_v2.md    session-grading rubric — default for `sideeye review`
  rubric/rubric_advice_v1.md     default for `sideeye advise`
  config.py                      judge-route resolution chain + cheap-route guards
  judge/schema.py                verdict schema + validation + is_flagged()
  judge/transcript.py            capture-agnostic SessionTranscript (the pipe)
  judge/judge.py                 forced-tool-call structured verdict; judge_session()
  adapters/codex_rollout.py      adapter #1: Codex rollout log -> SessionTranscript
  adapters/claude_code.py        adapter #2: Claude Code transcript -> SessionTranscript
  adapters/opencode.py           adapter #3: OpenCode V2 export -> SessionTranscript
  integrations/opencode/         native OpenCode server/TUI plugin
  configure.py                   one-command setup and harness detection
  skills/                        Claude Code plugin skill (the /sideeye:ask router)
  docs/                          pitch, MVP scope, phase-C design questions, cost distillation
  record.py                      shared session-verdict record shape
  run_judge.py                   Phase B: grade a JSONL of (prompt, answer) pairs
  sampler.py                     Phase C: RANDOM stream — scan rollouts, judge, verdicts/sampled.jsonl
  escalate.py                    Phase C: HUMAN stream — "ask the expensive model", verdicts/escalated.jsonl
  cost_report.py                 the money story: counterfactual savings + quality, CLI + HTML
  tests/                         unit tests (no network): 165 passing
  data/                          seed_pairs, hard_tasks, ground_truth (fixtures)
  verdicts/                      run output (gitignored)

The capture-agnostic pipe (the key design decision)

Every capture source produces a SessionTranscript (judge/transcript.py); the judge, sampler, and cost report only ever see that shape. So capture options are just adapters into one pipe:

• codex_rollout — reads Codex's own session logs. In-scope, no praxis changes, full session transcript. Caveat: it's the client's view — if the gateway mutated the request, the log wouldn't show it.
• claude_code — reads Claude Code's session transcripts (~/.claude/projects/). What the skills and the default sampler use.
• gateway sampling tap (production, not built): a praxis filter that samples at request time and emits events (Rust, lives in the proxy repo); the schema is designed so results carry over unchanged.

Phase B — pair judging (falsification gate) — DONE

pytest tests -q                                         # 165 tests, no network
python -m sideeye.run_judge --pairs data/seed_pairs.jsonl \
    --ground-truth data/ground_truth.json               # needs a judge route

Result: judge caught 2/2 planted defects, 0/4 clean false-positives; on hard field-evidence tasks it correctly passed GLM's safe answers and flagged the truncated one. ~$0.008/pair on Sonnet 5.

Phase C — automatic capture + escalation

Two verdict streams, kept separate (escalations are an adversarial sample and must never skew the random-sample scoreboard):

# RANDOM stream — unbiased quality/savings estimate (Sonnet 5; --client claude|codex)
python -m sideeye.sampler --idle-min 10 --sample-rate 1.0

# HUMAN stream — "I'm not sure, ask the expensive model" (Opus 4.8 by default)
sideeye review                        # == python -m sideeye.escalate, tier-1
sideeye review --model fable          # the nasty ones

# The money story (counterfactual savings + held quality; CLI + HTML)
python -m sideeye.cost_report --html verdicts/cost-report.html

Escalation is review, not re-answer; it carries a tier knob (tier-1 transcript review = default; tier-2 agentic "run the code" = the only tier that catches execution-dependent defects, and is a sandboxed worker, not a script — not built in this POC, and it refuses honestly if asked).

The whole pipeline (capture → judge → verdict → cost) is verified end-to-end on real Codex rollouts and Claude Code sessions. On the dogfood deployment the judge route points at the metered gateway, so judge spend shows on the dashboard; for any other user it's just their Claude route (see Configure).

Rules honored

• Secrets from env or a local (untracked) key file only — never hardcoded, committed, or logged.
• In the reference deployment the judge routes through the metered gateway, so judge spend is visible on the dashboard; the draft model is self-hosted and costs $0.
• Counterfactual savings are labeled estimates, not measurements.