Inference-time compute scheduler for reasoning models.
Meridian is a scheduling layer for vLLM that treats the two token-generation phases of reasoning models as distinct workloads. It intercepts the vLLM scheduler at runtime — no fork required — and applies separate SLOs, eviction policies, and entropy-driven budget control to each phase.
Reasoning models (DeepSeek-R1, Qwen3, o3, Granite 3.2) emit two structurally different sequences in a single request:
[prompt] → <think> ... N reasoning tokens ... </think> → [output tokens]
| Phase | User-visible? | Latency tolerance | Cost profile |
|---|---|---|---|
| Think-decode | No | High | Throughput-bound |
| Output-decode | Yes | Zero tolerance | Latency-bound |
Standard continuous-batching schedulers process both phases from the same priority queue with the same inter-token latency target. This leaves output-phase latency unnecessarily constrained by think-phase batch dynamics. Meridian exploits the asymmetry.
| Component | Verified in CI | Verified on GPU runner |
|---|---|---|
PhaseRouter state machine |
✓ | — |
MeridianScheduler dual-queue dispatch |
✓ | — |
PhaseAwareBlockManager three-tier eviction |
✓ | — |
EntropyProbe CPU backend (NumPy) |
✓ | — |
dspark_bridge acceptance bounds, hook, ledger |
✓ | — |
| Phase-segmented draft acceptance — measured | — | — (deferred, protocol) |
meridian-kernels CUDA kernels — EAT/entropy correctness vs CPU |
— | ✓ |
| vLLM plugin — attach / reorder / inject | ✓ | ✓ |
Disagg block manager surface (offload_block, ingest_block) |
✓ | — |
| NIXL fabric wire protocol (synthetic mock) | ✓ | — |
| Benchmark harness — synthetic replay | ✓ | ✓ |
Benchmark harness — real-vLLM with Qwen2.5-0.5B |
— | — |
| SLSA L2 provenance attestation | ✓ (on every v* tag) |
— |
cargo deny supply-chain audit |
✓ | — |
Incoming requests
│
▼
Phase Router ──── token stream state machine, O(1) per token
│
├── ExitThink / ForceBudget ─▶ EAT + RPDI entropy signals
│
├── Think-Decode Scheduler (TPOT-relaxed, 2.5× batch budget)
│
└── Output-Decode Scheduler (TTOT-strict, stream priority)
│
▼
Phase-Aware KV Block Manager
ThinkComplete ← evicted first
ThinkActive
OutputCritical ← never evicted without alerting
-
Dual-queue scheduling — output-phase requests drain the GPU batch before think-phase requests fill remaining capacity. Backed by
crates/meridian-core/src/scheduler.rs. -
Phase-aware KV eviction — three tiers with strict priority ordering.
OutputCriticaleviction fires an observable counter; the target rate is zero. Backed bycrates/meridian-core/src/block_manager.rs. -
Entropy-driven budget forcing — EAT (arXiv:2509.26522) and RPDI (arXiv:2603.14251) signals inject
</think>when the model signals convergence, not on a static timer. CUDA kernels run on a dedicated secondary stream. Backed bycrates/meridian-kernels/andpython/meridian/entropy_probe.py. -
Drop-in vLLM plugin — wraps the existing scheduler via attribute delegation; fully reversible; no vLLM source modification. Backed by
python/meridian/vllm_plugin.py. -
Disagg KV transfer —
offload_block/ingest_blockhooks support prefill-decode disaggregation fabrics (NIXL, Mooncake-compatible). Documented in ADR-0006. -
Phase-conditioned speculative decoding — the same phase signal, applied one level down the stack to draft-model scheduling. Backed by
crates/meridian-core/src/dspark_bridge/and ADR-0009. Off by default, and half-inert by design — see below.
DeepSeek's DeepSpec ships trained
speculative-decoding drafters for Qwen3. Its README states each released
checkpoint was trained on data its target generated in non-thinking mode —
while Qwen3's enable_thinking toggle is what production reasoning traffic uses.
That is a documented training/serving mismatch, and it suggests speculative decoding's speedup may be silently phase-dependent for these checkpoints.
What is proved (note): Meridian's EAT signal is DSpark's confidence-head supervision target for a specific drafter, and target entropy provably ceilings the single-step acceptance of any deterministic drafter — a one-sided bound computable from a signal Meridian already has.
What is not measured: the phase gap itself. It needs a GPU and is not scheduled. So the shipped hook can only ever reduce draft depth, and only where the proved bound justifies it; conditioning on phase requires supplying measured rates together with the provenance of the run that produced them. A synthetic trace cannot be promoted to a published claim — that rule is enforced by the type system, not by documentation.
The deferred experiment is fully specified in the Phase 1 protocol: hypotheses, models, sample sizes, statistic, and a decision rule fixed before any data exists.
Requires Linux (or WSL2), Rust 1.85+, Python 3.11+. GPU + CUDA 12.6 for the CUDA backend; not required for synthetic benchmarks or unit tests.
git clone https://github.com/angelnicolasc/meridian.git
cd meridian
# Install Python deps and build the native extension.
uv sync --project python
maturin develop --release -m crates/meridian-python/Cargo.toml
# Run all tests (no GPU required).
cargo nextest run --workspace
uv run --project python pytest -m "not gpu and not vllm"
# Run the synthetic A/B benchmark (no GPU, no vLLM).
uv --project python run python -m benchmarks.meridian_bench synthetic-replay \
--baseline stock --duration-s 30 --arrival-rate 8 --out-dir bench-out/For CUDA kernel support, append --cargo-extra-args="--features cuda" to the
maturin develop call (requires nvcc + CUDA 12.6).
crates/meridian-core/ Rust scheduler core — no CUDA, no Python dep
crates/meridian-kernels/ CUDA entropy and EAT kernels + C FFI + NIXL wrappers
crates/meridian-python/ pyo3 bindings (built via maturin)
python/meridian/ Python package — EntropyProbe, vLLM plugin, config
docs/ mdBook documentation + 9 ADRs + research notes
models/ Per-model token boundary configs (DeepSeek-R1, Qwen3, ...)
benchmarks/ Replay harness — synthetic + real-vLLM + A/B mode
Meridian is not a throughput optimiser, accuracy guarantor, or full serving engine. See Non-goals for explicit scope boundaries.
| Artifact | Link |
|---|---|
| CI results | GitHub Actions |
| Release provenance (SLSA L2) | Releases |
| Supply-chain audit | cargo deny check in ci.yml |
| Architecture & ADRs | docs/src/ |
| Benchmark methodology | ADR-0005 |
Contributions welcome under DCO sign-off. Read CONTRIBUTING.md and CODE_OF_CONDUCT.md before opening a PR. Security disclosures: SECURITY.md.