[willccbb]

Each proxied request holds a client socket and a worker socket, so a router started under the common 1024 soft RLIMIT_NOFILE runs out of descriptors at roughly 500 concurrent requests. It then can't accept clients (axum::serve::listener: accept error: Too many open files (os error 24), hidden by the default vllm_router_rs= log filter) or connect to its workers (Failed to send typed request ... error sending request for url). Those 500s open the circuit breaker, and while the load holds the descriptors, every half-open probe fails the same way, so a healthy worker stays unavailable until the load stops.

This came up in a prime-rl RL run. After a burst of about 100 send errors, every request for the rest of the run failed with No available workers (all circuits open or unhealthy). prime-rl spawns the router from a shell with soft limit 1024 (vLLM raises only its own limit), and the replay below reproduces that pattern. That run's logs didn't record descriptor counts or the OS error, so descriptor exhaustion there is the leading explanation rather than an observed one.

The fix raises the soft limit to the hard limit in server::startup, which both the Python Router.start and the Rust binary go through before opening any socket, and logs the result, e.g. Open file limit: 1048576 (hard 1048576). A low hard limit stays visible in that line.

Replay on a Linux box, v0.2.3 router in front of a fake uvicorn worker, open-loop 200 req/s with ~8s requests (~1600 concurrent):

router	start limit (soft/hard)	result
v0.2.3	1024 / 1048576	fds pinned at 1024, EMFILE accept errors, breaker opens, ~97% 503 for the whole run
v0.2.3	65536 / 65536	peak ~3080 fds, no errors, 100% 200
this PR	1024 / 65536	logs Open file limit: 65536 (hard 65536), no errors, 100% 200

cargo fmt --check and cargo clippy --release --lib --bins -- -D warnings pass. --all-targets fails on examples/simulate_consistent_hash.rs (for_kv_map), the same as on main with clippy 1.97.

🤖 Generated with Claude Code

---

Note

Low Risk
Startup-only Unix setrlimit with graceful logging on failure; no change to request routing or auth logic.

Overview
Raises the process open file descriptor limit at router startup so high concurrency proxying does not hit the default soft RLIMIT_NOFILE (~1024), which effectively caps the router around ~500 in-flight requests (client + worker sockets per request).

Adds the rlimit dependency and, in server::startup on Unix (after logging init, before sockets/metrics), bumps the soft limit toward the hard limit and logs Open file limit: {soft} (hard {hard}) or warns if the raise fails. Non-Unix builds are unchanged (#[cfg(unix)]).

^{Reviewed by Cursor Bugbot for commit 9a10824. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Raise process open file limit at startup on Unix

Adds the rlimit crate and, on Unix, server startup now raises the process open-file limit to the maximum available value in server.rs. The resulting NOFILE soft and hard limits are logged on success; a warning is logged if either operation fails. Non-Unix platforms and the rest of the startup order are unchanged.

^{Macroscope summarized 9a10824.}