recon-agent

An intake-and-reconciliation agent for regulated goods transfer paperwork, built as a forward-deployed-engineering portfolio piece: deterministic where the work is arithmetic, an LLM at exactly one judgment point, an eval harness with a seeded golden dataset, and a human-in-the-loop review queue that feeds corrections back into the evals.

flowchart LR
    A[Transfer form<br/>CSV intake] --> B[Deterministic<br/>reconcile]
    B -->|verified findings| C{LLM<br/>judgment point}
    C -->|schema-valid JSON| D[Draft decision]
    C -.->|malformed / CLI down /<br/>offline| E[Template fallback]
    E --> D
    D --> F[[Human review queue<br/>approve or correct]]
    F -->|corrections| G[(feedback.jsonl<br/>new eval cases)]
    B -.-> H[(trace.jsonl<br/>audit trail)]
    C -.-> H
    D -.-> H

Loading

The agent finalizes nothing. Every decision lands at pending_review for a person, and every step is written to an audit trail.

Everything here is synthetic. Fake vendors, fake lots, fake products. The defect classes are modeled on failure modes observed in real regulated paperwork, including a signed production transfer form whose printed unit totals were +7 against the sum of its own rows.

Quickstart (30 seconds, no API keys)

git clone https://github.com/ohhkaneda/recon-agent && cd recon-agent
python3 generate_golden.py --n 50   # build a synthetic 50-form dataset
python3 evals.py --offline          # run the full eval harness -> 50/50 detection

Only Python 3 (stdlib) is needed. That's the whole pipeline, offline, on zero credits.

What that prints (actual output, click to expand)

$ python3 generate_golden.py --n 50
wrote 50 forms to golden/forms · distribution: {'clean': 22, 'subtotal_drift': 8,
'duplicate_container': 7, 'weight_mismatch': 6, 'total_drift': 6, 'missing_container': 1}

$ python3 evals.py --offline

=== recon-agent eval report ===
DETECTION: 50 runs, 50 passed, 0 failed.
JUDGMENT:  50 scored, 50 correct severity, 0 wrong.

A single form, and the audit trail it leaves:

$ python3 agent.py golden/forms/TF-0004.csv --offline
TF-0004:  flag_blocking  [fallback]  2 finding(s)

$ cat runs/TF-0004/trace.jsonl
{"step": "parse", "rows": 41, "meta": {"vendor": "Harborview Naturals", "lot": "265060",
  "product": "10mg THC Softgel", "printed_units_subtotal": "383341", ...}}
{"step": "reconcile", "findings": [
  {"check": "duplicate_container", "detail": "container ids duplicated: [4]"},
  {"check": "missing_container", "detail": "gap in sequence, missing: [5]"}],
  "stats": {"row_count": 41, "unit_sum": 383341}}
{"step": "judge", "source": "fallback", "judgment": {"severity": "flag_blocking",
  "memo": "Automated reconciliation found: container ids duplicated: [4]; gap in sequence, missing: [5].",
  "receiving_instruction": "Hold receipt and verify physical containers before signing."}}
{"step": "queued", "status": "pending_review"}

Note the last line. The agent stops at pending_review; a human decides. Timestamps elided above for width; they are in the real trace.

Tests

The pipeline itself is stdlib only; the test suite needs pytest.

pip install pytest
python3 -m pytest -q     # 29 tests

Evals score model behaviour on the golden set. The tests pin the parts that must not drift regardless of the model: each deterministic check, the severity policy (physical uncertainty always blocks), the JSON contract, and the unhappy paths (malformed model output, CLI missing, timeout, non-zero exit, wrong schema). CI runs both on every push.

Run the full loop

python3 generate_golden.py --n 50        # golden dataset + labels
python3 evals.py --offline               # full eval, no LLM, no credits
python3 agent.py golden/forms/TF-0006.csv    # one live run (claude -p)
python3 hitl.py                          # review queue: approve or correct
python3 evals.py                         # evals now include human feedback

No API keys required. The LLM step shells out to the claude CLI; without it, a deterministic fallback produces the same JSON contract so the pipeline runs end to end offline.

The engineer version (architecture and decisions)

• Pipeline: parse -> reconcile -> judge -> queue. Each step logs to runs/<form>/trace.jsonl, so every decision has a full audit trail. If the client cannot see what the agent did, they will not trust it.
• Deterministic core. Parsing, container-sequence checks, duplicate detection, weight-consistency, and totals reconciliation are plain code. Spending tokens on arithmetic adds cost and nondeterminism to steps that have exactly one right answer.
• One LLM judgment point. Given verified findings, the model classifies severity (accept / flag_minor / flag_blocking) and drafts the memo a human receiver reads. Output is schema-validated JSON; malformed output falls back to the deterministic template. The model is never asked to find or verify discrepancies, only to judge and communicate ones already proven.
• Unhappy paths. CLI missing, CLI timeout, malformed JSON, wrong schema, offline mode: all degrade to the template fallback with the same contract. There is one way this goes right and many ways it goes wrong; the fallback covers the wrong ways.
• Nothing is final without a human. Decisions land at pending_review. The queue (hitl.py) is where a person approves or corrects. Corrections append to feedback.jsonl, which the eval harness folds in as new cases. That is the feedback mechanism that improves the harness over time.

The VP version (problem, outcome, evidence, risk)

• Problem: inbound transfer paperwork gets signed with internal inconsistencies. A real signed form was 7 units off against its own rows; nobody caught it before signatures. Every such miss becomes a reconciliation problem downstream, on regulated goods.
• Outcome: every inbound form is reconciled automatically in seconds, and a human receives a drafted decision with the discrepancy already itemized, instead of finding it (or not) by hand.
• Evidence: on a 50-form golden dataset with five seeded defect classes, detection is 50/50 with zero false positives on clean forms. Judgment severity on the live model path matched the expected class on the seeded cases exercised. Offline judgment scoring is trivially perfect by construction (the fallback implements the policy), which is why the number that matters is the live-model one, and why human overrides keep being collected as the harder eval set.
• Risk: the agent finalizes nothing. Physical-goods uncertainty is always flag_blocking and always lands on a human. The failure mode of the automation is a false flag that costs a minute of review, not a wrong acceptance.
• The three buckets: cost (receiving-desk minutes per form), risk (signed paperwork that disagrees with itself, caught before acknowledgement), revenue n/a for this workflow. Measured, not asserted: the eval report is the measurement.

Honest limits

• The golden set is synthetic and its defects are seeded to be detectable; detection at 50/50 proves the harness logic, not real-world recall. The next step of the eval story is unseeded, real-shaped forms and the human-override log as ground truth.
• Judgment eval on the offline path is circular by design (fallback == policy). Only live-model runs and human feedback produce non-trivial judgment scores.
• Single-file CSV intake. The real-world version parses the messy stuff (PDF, XLSX, forwarded email bodies), which is an intake problem this architecture isolates in parse_form() by design.

License

MIT — see LICENSE.