Skip to content

perf: load characterization and performance rework, CI and release automation - #19

Merged
kevinsbarnard merged 19 commits into
developfrom
feat/load-performance
Nov 19, 2026
Merged

kevinsbarnard merged 19 commits into
developfrom
feat/load-performance

Conversation

@kevinsbarnard

@kevinsbarnard kevinsbarnard commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Load characterization and performance rework of the server, plus CI, release automation and deployment hygiene.

The analysis found that most loss under load came from design rather than Python itself:

  • A 100 ms poll sleep combined with a 64-message queue dropped messages above about 640 msg/s per client, even with a single client.
  • Clients requesting the same frame at the same time each encoded it separately (9.3 JPEG encodes per frame with 10 Dial clients).
  • Per-client deflate ran on the event loop, including on incompressible JPEGs.
  • The spy channel read from a lossy queue, so it under-reported rates.
  • A slow JSON encoder.

Results

Same harness and machine (24 cores, Python 3.13), v0.8.0 defaults vs this branch. Details in benchmarks/README.md.

Scenario v0.8.0 This branch
JSON, 1 client, 1.2 k msg/s 66% delivered, p50 32 ms 100%, p50 0.5 ms
JSON, 50 clients 36%, CPU 112% 99.9%, CPU 76%
JSON, 100 clients (compression off) 12% 100%, p50 4.4 ms
Dial, 10 clients: JPEG encodes per frame 9.3 1.00
Dial, 10 clients: CPU 162% 42%
Spy Hz for a 100 Hz channel (streaming / periodic) 64 / 12 100 / 100

Changes

  • Delivery: event-driven instead of polling. New ?conflate=1 keeps the latest message per channel; periodic mode always does this. A cached routing table means each regex runs once per channel. Invalid requests are closed with code 1008.
  • Encoding: each message is encoded once and shared by all clients; cache hits skip the thread hop. Separate light (JSON) and heavy (image) pools. JSON encoder is 3–6x faster, about 2x more with the optional orjson extra. Text is pre-encoded to UTF-8.
  • Spy: a shared counter on the LCM thread, with per-connection views.
  • CLI: shared lib/cli.py; --compression, --uvloop, --processes, --ping-interval, --ping-timeout; Dial adds --max-fps and --image-workers.
  • Ops: /healthz and /metrics (Prometheus) on the server port. Docker installs with uv and includes the extras, an ARGS env var and a HEALTHCHECK.
  • Fixes: shutdown no longer hangs on a quiet bus; exceptions can't kill the LCM thread; closed connections stop work immediately; -v controls drop warnings.
  • Project:
    • Python ≥3.11 (the old floor made uv sync fail) and uv.lock committed.
    • Ruff config added and existing lint fixed; pre-commit hooks.
    • Tests are self-contained (vendored LCM types, memq://) and run in CI on 3.11–3.14.
    • python-semantic-release now handles versioning: develop produces -rc.N prereleases and main produces releases, as GitHub releases only with no PyPI publish.

This branch is based on main, so it also brings develop up to date with the v0.8.0 commit.

Behavior changes

  • JSON output is compact and strict: NaN and ±inf become null.
  • The Dial proxy defaults to --compression none.
  • --poll-interval is accepted but ignored.

Library API changes (only for code importing lcm_websocket_server.lib):

  • LCMObserver: get_nowait(), await get() and drain() replace get(block=...) and task_done().
  • LCMTimedObserver is removed.
  • LCMSpy is now a view over SpyAccumulator.
  • The encode cache is owned by the server, and handlers should be stateless (TextPayload, is_heavy()).
  • LCMWebSocketServer no longer takes empty_wait_seconds.
  • App run() functions take ServerOptions.

Before merging

  • The release job pushes a version commit and tag to develop/main using GITHUB_TOKEN. If those branches are protected, allow GitHub Actions to push or use a token that can bypass protection.
  • No commits carry BREAKING CHANGE footers; with major_on_zero = false the next version is 0.9.0 (rehearsed locally on a scratch develop: 0.9.0-rc.1).

Test plan

  • uv run pytest: 168 passed on Python 3.11, 3.12, 3.13 and 3.14 (108 originally)
  • uv run ruff check . clean; pre-commit hooks pass
  • benchmarks/load.py all before and after (table above)
  • Docker image built and run: /healthz OK, /metrics OK, container reports healthy, runs under --uvloop
  • --processes 2 end-to-end test (CLI subprocess serving the spy channel)
  • semantic-release rehearsed in a scratch clone (0.9.0-rc.1, lockfile and changelog updated, tag created)
  • CI green on this PR
  • Try the Dial proxy against live MolaRS traffic

🤖 Generated with Claude Code

kevinsbarnard and others added 19 commits April 20, 2026 10:53
websockets>=16 needs Python 3.10+ (17 needs 3.11+), so the previous
>=3.9 floor made the project unresolvable with uv. Also adds ruff as a
dev dependency and pins the pytest-asyncio fixture loop scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gging

- Break out of the update loops on ConnectionClosed instead of continuing
  to encode for a dead socket until its queue drains.
- Give the LCM daemon a handle timeout so stop() no longer hangs on a
  quiet bus, and always stop the republisher when serve() exits.
- get_logger() now uses the logging hierarchy, so pubsub drop warnings
  honour -v and propagate to standard handlers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the poll-and-sleep consumer loop. With a 100 ms sleep and a
64-entry queue, any client receiving more than ~640 msg/s dropped
messages, and every message waited up to 100 ms.

- LCMObserver uses a deque(maxlen) and an awaitable get(); the LCM
  thread wakes the consumer via call_soon_threadsafe only when it is
  waiting. --poll-interval is now a deprecated no-op.
- New conflate mode keeps the latest message per channel so busy
  channels cannot evict quiet ones. Enabled with ?conflate=1 and always
  used for ?update_interval_ms (periodic mode previously overflowed its
  queue between ticks). Periodic ticks use absolute deadlines.
- LCMRepublisher caches channel -> matching observers (copy-on-write,
  invalidated on (un)subscribe), so each regex runs once per channel.
  Adds taps for per-message hooks and contains exceptions so they
  cannot kill the LCM thread.
- Channel regexes are validated (and length-capped) at connect; invalid
  requests are closed with 1008 instead of silently matching nothing.
- Each connection ends as soon as the socket closes, even when idle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Previously every client matching LWS_LCM_SPY (including the default .*)
added a second all-traffic observer, doubling LCM-thread dispatch, and
its stats were computed from a bounded queue that dropped under load
(a 100 Hz channel was reported as 64 Hz, or 12 Hz in periodic mode).

SpyAccumulator now counts every message once on the LCM thread via a
republisher tap. LCMSpy is a per-connection view that keeps the
since-connect semantics by snapshotting counters at creation; rates are
computed from deltas between emissions. Jitter is measured over fixed
windows of the emission interval. Emission runs as its own task.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-handler EncodeCache checked then computed without tracking
in-flight work, so concurrent clients all missed and encoded the same
frame (9.3 JPEG encodes per frame with 10 Dial clients).

- EncodeCache moves to the server and is confined to the event loop:
  the first request starts the encode and concurrent requests await the
  same shielded future. Cache hits return without a thread hop. It keeps
  the last few messages per channel so slightly lagging clients still
  hit, and caches None so unencodable messages are not retried.
- Handlers are now pure and stateless; JSONHandler and DialHandler drop
  their internal caches.
- Handlers mark CPU-heavy messages via is_heavy(); these run on a
  separate heavy pool so JPEG bursts cannot starve JSON work.

Also removes unused test imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
encode_value() used isinstance() against a runtime_checkable Protocol,
which costs ~1 us per value and dominated encoding (1.75 ms of 2 ms for
a 512-beam multibeam message).

- Dispatch on exact type with per-type slot caching and a fast path
  for homogeneous primitive arrays: odometry_t 32 -> 12 us, multibeam
  1.99 ms -> 0.31 ms.
- When the optional orjson extra is installed it serializes the LCM
  objects directly via a default hook: 5.8 us and 0.14 ms respectively.
- Output is compact, strict JSON on both paths: NaN and +/-inf become
  null (previously inf produced an invalid Infinity literal).
- Handlers can return TextPayload (UTF-8 bytes) which the server sends
  as a text frame with text=True, so text is encoded once rather than
  per connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New lib/cli.py consolidates argument parsing, LCM type registry setup,
  SIGTERM handling and process launching that json_proxy and dial_proxy
  each duplicated.
- --compression {deflate,none}: deflate now uses compression level 1
  (about half the CPU of level 6 for a similar ratio on JSON). Defaults:
  deflate for the JSON proxy, none for the Dial proxy, since per-client
  deflate of incompressible JPEG frames cost more CPU than encoding them.
- --uvloop runs on uvloop (optional extra).
- --processes N runs N workers on one port with SO_REUSEPORT to scale
  past one core; each has its own LCM subscription, encode cache and
  spy stats.
- Dial proxy gains --image-workers for the heavy (image) pool.
- Fix the __main__ blocks, which passed the result of main() to
  asyncio.run().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DownsamplingMJPEGEncoder no longer calls cv2.resize at scale 1.0,
  which was a 0.4 ms full-frame copy per 3 MP image.
- DialHandler accepts max_image_fps (--max-fps) to cap transcoded
  frames per image channel. The limiter is deadline based with 10%
  tolerance so source jitter does not halve the rate, and skipped
  frames are cached as None so every client skips the same frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also fixes the spy client example, which parsed ev.event instead of
ev.data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copies the lcm-gen output for stdlcm.header_t, senlcm.image_t and
senlcm.image_metadata_t from MolaRS into tests/lcmtypes and puts it on
the pytest path, so tests no longer need MolaRS on PYTHONPATH (and can
run in CI). Removes a NaN test that never exercised NaN; that behavior
is covered by test_types.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins ruff 0.17 and targets Python 3.11 so local runs, pre-commit and
CI agree; lcm-gen output is excluded. Remaining broad excepts in the
handlers are intentional (malformed input must not propagate) and are
annotated. Example scripts with shebangs are now executable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CI runs ruff and pytest on Python 3.11-3.14 using the locked
  dependencies.
- Tests default LCM_DEFAULT_URL to memq:// so they never need a
  multicast route.
- pre-commit runs ruff plus basic file hygiene hooks; this commit also
  applies their whitespace/EOF fixes across the repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ServerMetrics tracks clients, connections, rejected requests, LCM
  messages received, frames and bytes sent, queue drops, encode
  requests/encodes/errors/time, and worst event-loop lag.
- Plain HTTP GET /healthz and /metrics (Prometheus text format) are
  answered on the WebSocket port via process_request; WebSocket
  upgrades on any path are unaffected, so channel names are not
  reserved.
- --ping-interval / --ping-timeout (0 disables) expose the websockets
  keepalive, which can close connections on slow links carrying large
  frames.
- server_factory test fixture moves to conftest for reuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Install the wheel with uv pip, including the orjson and uvloop extras
  (and image for the Dial image).
- New ARGS env var appends extra flags (e.g. --uvloop --processes 4).
- HEALTHCHECK polls /healthz.
- Use the lcm-websocket-json-proxy entry point name.
- Keep .venv, caches, tests and benchmarks out of the build context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers /healthz and /metrics, LCM UDP receive buffer sizing, tuning
for constrained links, scaling with --processes, TLS via a reverse
proxy, Docker ARGS/HEALTHCHECK, and the uv/pre-commit/benchmark
development workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pushes to main create releases and pushes to develop create -rc.N
prereleases: python-semantic-release derives the version from
conventional commits since the last tag, updates pyproject.toml,
uv.lock and CHANGELOG.md, tags, and creates a GitHub release with the
wheel and sdist. Releases run only after lint, tests and build pass.
Nothing is published to PyPI.

Also aligns the workflow with the team template (action versions,
uv cache, workflow_dispatch, read-only default permissions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevinsbarnard
kevinsbarnard merged commit b01e5e1 into develop Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant