Skip to content

feat: default to BM25 retrieval with bounded search output - #192

Draft
aeneasr wants to merge 8 commits into
mainfrom
aeneasr/bm25
Draft

aeneasr wants to merge 8 commits into
mainfrom
aeneasr/bm25

Conversation

@aeneasr

@aeneasr aeneasr commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Lumen can index and search code locally without an embedding service. This PR makes code-aware BM25 the default, retains semantic retrieval as an opt-in, and bounds search responses across both modes.

Changes

  • Add weighted SQLite FTS5 retrieval over paths, symbols, signatures, comments and bodies, with IDF-weighted query coverage, shared worktree collections and freshness checks. Expand proven calls/imports into at most two related snippets; lexical_only exposes the direct retrieval baseline.
  • Extend extraction and retrieval coverage across source languages, structured data and Markdown. Fix changed/deleted-file recovery, canonical project identity, chunk spans and structural expansion, with a shared source-judged retrieval catalog.
  • Add max_tokens (MCP) / --max-tokens (CLI): default 4000 estimated tokens, a hard 16,000-byte rendered-text ceiling; -1 disables it. Preserve ranked locations first, share remaining space across snippets, and report truncation/omission while keeping original line ranges. Include headers, warnings, metadata and the CLI newline in the cap; normalize invalid UTF-8 before sizing to preserve the cap through MCP JSON.
  • Integrate retrieval selection with CLI, MCP, hooks, documentation and mirrored plugin skills. Add offline BM25 E2E coverage, budget evaluation, a checked list of known retrieval misses, and SWE benchmark tooling/reports.

Evaluation

The catalog contains 445 outcomes: 417 supported and 28 diagnostic. Unlimited BM25 output passes 397/417 supported and 13/28 diagnostic outcomes. The 20 supported misses remain explicit failures in reports; regression checks reject new misses and require removing entries that start passing. See retrieval parity.

Budget evaluation used 445 executable MCP searches per configuration:

Configuration Supported pass Maximum response bytes
10 hits, unlimited 397/417 19,338
10 hits, default budget 397/417 15,997
10 hits, 1000 estimated tokens 343/417 4,000
20 hits, default budget 389/417 15,999

The default budget at 10 hits changed no case outcomes; all bounded responses stayed within their byte ceiling and parsed successfully. Tighter budgets or more hits can hide evidence even when its location remains visible. This is an output-size guardrail, not evidence of better ranking or lexical–semantic equivalence. Token counts are estimates; the byte ceiling is exact.

Validation and remaining work

  • make test, make lint, E2E-tag lint, and make e2e-bm25 passed locally during implementation.
  • After the final invalid-UTF-8 fix, all ./cmd tests, focused executable CLI/MCP budget and parser tests, and both lint configurations passed again. The full suite and valid-text catalog were not rerun after that correction.
  • git diff --check passed. Opus 5.5 independently assessed the budget design and implemented it, including the MCP serialization regression fix.
  • Keep this PR draft pending fresh CI and semantic snapshot review. The recorded local semantic language run matches 33/58 historical snapshots (48/58 at the original BM25 handoff); extraction changes and local model/runtime drift require review on the pinned CI runtime. Snapshots were not regenerated. The full real-embedding catalog was not completed, so this PR does not claim semantic parity.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

aeneasr and others added 7 commits October 11, 2026 11:52
The lexical indexing path chunked files but never applied
splitOversizedChunks, so generated files became single BM25 documents of
up to 9,411 lines. One MCP search hit then returned 763k characters and
overflowed the agent's tool output budget. Apply the same
LUMEN_MAX_CHUNK_TOKENS split the semantic path uses; no merging so
lexical documents stay symbol-scoped.

IndexVersion 5 -> 6: existing lexical indexes contain the oversized
documents and must be rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude Code defers every MCP tool behind ToolSearch by default, so the
model only saw the tool name and had to load it before use. In bench-swe
runs Sonnet 5.5 called the search tool in 1 of 12 tasks and used Bash
grep instead. Advertise anthropic/alwaysLoad on semantic_search so it is
in context with its description from the first turn (Claude Code >=
2.1.285); index_status and health_check stay deferred.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- add --retrieval (bm25|semantic, default bm25); results dir is named
  after the retrieval mode and preflight only checks Ollama for semantic
- build lumen with -tags fts5 in preflight
- register the dynamic MCP server as plugin_lumen_lumen so the tool name
  matches what the plugin hooks tell Claude to call; since #179 the
  nudge referenced a tool that did not exist in bench sessions
- accept both tool names in analysis and reports; show retrieval mode in
  the summary header

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@aeneasr aeneasr changed the title feat: default to code-aware BM25 retrieval feat: default to BM25 retrieval with bounded search output Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant