Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,13 +1,10 @@
.dockerignore
docker/

.git
.gitignore

poetry.lock
uv.lock
dist/
scripts/

*__pycache__/
.venv/
Expand Down
91 changes: 89 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ jobs:
concurrency: release
permissions:
contents: write
outputs:
released: ${{ steps.release.outputs.released }}
version: ${{ steps.release.outputs.version }}
prerelease: ${{ steps.release.outputs.prerelease }}
steps:
- uses: actions/checkout@v7
with:
Expand All @@ -66,7 +70,90 @@ jobs:
with:
enable-cache: true

- name: Release
run: uv run --group release semantic-release version
- name: Version, tag and GitHub release
id: release
run: |
BEFORE=$(git tag -l | sort)
uv run --group release semantic-release version
AFTER=$(git tag -l | sort)
NEW_TAG=$(comm -13 <(echo "$BEFORE") <(echo "$AFTER") | head -1)
if [ -n "$NEW_TAG" ]; then
VERSION=${NEW_TAG#v}
echo "released=true" >> "$GITHUB_OUTPUT"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
if [[ "$VERSION" == *-* ]]; then PRE=true; else PRE=false; fi
echo "prerelease=$PRE" >> "$GITHUB_OUTPUT"
else
echo "released=false" >> "$GITHUB_OUTPUT"
fi
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Upload release assets
if: steps.release.outputs.released == 'true'
run: uv run --group release semantic-release publish
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# Builds and smoke-tests the Docker image. On pull requests the image is
# only tested. After a release it is built from the release tag and pushed
# to Docker Hub:
# release on main: X.Y.Z, X.Y, latest
# prerelease on develop: X.Y.Z-rc.N, develop
# (Tags pushed by GITHUB_TOKEN do not trigger workflows, so this runs here
# rather than in a separate on-tag workflow.)
docker:
needs: [build, release]
if: >-
!cancelled() && needs.build.result == 'success' &&
(github.event_name == 'pull_request' || needs.release.outputs.released == 'true')
runs-on: ubuntu-latest
env:
PUBLISH: ${{ needs.release.outputs.released == 'true' }}
VERSION: ${{ needs.release.outputs.version }}
PRERELEASE: ${{ needs.release.outputs.prerelease == 'true' }}
steps:
- uses: actions/checkout@v7
with:
ref: ${{ env.PUBLISH == 'true' && format('v{0}', env.VERSION) || '' }}
- uses: docker/setup-buildx-action@v3

- name: Image metadata
id: meta
uses: docker/metadata-action@v5
with:
images: mbari/lcm-websocket-server
flavor: latest=false
tags: |
type=semver,pattern={{version}},value=${{ env.VERSION }},enable=${{ env.PUBLISH }}
type=semver,pattern={{major}}.{{minor}},value=${{ env.VERSION }},enable=${{ env.PUBLISH == 'true' && env.PRERELEASE == 'false' }}
type=raw,value=latest,enable=${{ env.PUBLISH == 'true' && env.PRERELEASE == 'false' }}
type=raw,value=develop,enable=${{ env.PUBLISH == 'true' && env.PRERELEASE == 'true' }}
type=raw,value=ci,enable=${{ env.PUBLISH != 'true' }}

- name: Build
uses: docker/bake-action@v6
with:
source: .
files: |
./docker-bake.hcl
cwd://${{ steps.meta.outputs.bake-file }}
targets: image
load: true
set: |
*.cache-from=type=gha
*.cache-to=type=gha,mode=max

- name: Smoke test
run: docker/smoke_test.sh "mbari/lcm-websocket-server:${{ env.PUBLISH == 'true' && env.VERSION || 'ci' }}"

- name: Log in to Docker Hub
if: env.PUBLISH == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Push
if: env.PUBLISH == 'true'
run: echo "${{ steps.meta.outputs.tags }}" | xargs -n1 docker push
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
exclude: ^(tests/lcmtypes/|lcm_websocket_server/lib/lcm_utils/channel_stats(_list)?\.py$)
exclude: ^(lcm_websocket_server/lib/lcm_utils/channel_stats(_list)?\.py$)
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.17.6
Expand Down
77 changes: 52 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ Or with optional extras:
```bash
pip install ".[image]" # Dial proxy (OpenCV)
pip install ".[orjson,uvloop]" # faster JSON encoding and event loop
pip install ".[lcmtypes]" # CoMPAS and MOLARS LCM types
```

The `lcmtypes` extra installs [compas-lcmtypes](https://github.com/CoMPASLab/compas_lcmtypes) and [molars-lcmtypes](https://github.com/CoMPASLab/molars_lcmtypes), the default types for the Dial proxy and the Docker image. Any other Python package of lcm-gen types works as well (see `--lcm-packages`).

See [Development](#wrench-development) for working on the project itself.

## :rocket: Usage
Expand Down Expand Up @@ -111,9 +114,7 @@ To run the server locally on port 8765 and republish messages on all channels (J
lcm-websocket-dial-proxy --host localhost --port 8765 --channel '.*' --quality 75 --scale 1.0
```

The Dial proxy depends on the `molars-lcmtypes` Python package to be installed. This package is not available on PyPI, so it must be built and installed manually; see the [MolaRS repository](https://github.com/CoMPASLab/molars/) for more info.

The `Dockerfile.dial` file can be used to build the image with the `molars-lcmtypes` package installed. For this, the Python wheel `molars_lcmtypes-0.0.0-py3-none-any.whl` must be placed at the repository root before building the image.
By default the Dial proxy discovers types from `compas_lcmtypes` and `molars_lcmtypes` (install the `lcmtypes` extra) and transcodes messages of the LCM type `senlcm.image_t`. Use `--lcm-packages` and `--image-type` to change either; the image type is matched by its LCM name regardless of the Python package prefix it was generated with.

### Connecting to the Server

Expand Down Expand Up @@ -228,41 +229,61 @@ The server has no authentication or encryption. When it is reachable beyond a tr

## :whale: Docker

### Build
One image, [`mbari/lcm-websocket-server`](https://hub.docker.com/r/mbari/lcm-websocket-server), runs either proxy. It includes the `image`, `orjson`, `uvloop` and `lcmtypes` extras, so the CoMPAS and MOLARS types are available without mounting anything.

### Tags

Images are built and pushed by CI after each release:

A Docker image to run the `lcm-websocket-server` can be built with:
| Tag | Source |
|-----|--------|
| `X.Y.Z`, `X.Y`, `latest` | Releases from `main` |
| `X.Y.Z-rc.N`, `develop` | Prereleases from `develop` |

### Run

```bash
./scripts/docker_build.sh
# JSON proxy for all channels
docker run -d --name lcm-websocket-server --network=host mbari/lcm-websocket-server

# Dial proxy, downscaled JPEGs capped at 5 fps per camera
docker run -d --name lcm-websocket-dial --network=host \
-e PROXY=dial -e ARGS="--scale 0.5 --quality 60 --max-fps 5" \
mbari/lcm-websocket-server
```

This will create the `mbari/lcm-websocket-server` image.
Run with `--network=host` so LCM's UDP multicast reaches the container.

### Run
| Variable | Default | Description |
|----------|---------|-------------|
| `PROXY` | `json` | `json` or `dial` |
| `HOST` | `0.0.0.0` | Address to listen on |
| `PORT` | `8765` | Port to listen on |
| `CHANNEL` | `.*` | LCM channel regex to subscribe to |
| `LCM_PACKAGES` | `compas_lcmtypes,molars_lcmtypes` | Python packages to discover LCM types from |
| `ARGS` | | Extra flags, e.g. `--uvloop --processes 4 -vv` |

Flags can also be passed as container arguments (`docker run ... mbari/lcm-websocket-server --uvloop`), and any other command runs as given (`docker run -it ... mbari/lcm-websocket-server bash`).

The container can be run with:
To use other LCM types, mount the package under `/app`, which is on `PYTHONPATH`, and list it in `LCM_PACKAGES`:

```bash
docker run \
--name lcm-websocket-server \
--rm \
-e HOST=0.0.0.0 \
-e PORT=8765 \
-e CHANNEL=".*" \
-v /path/to/your_lcm_types_package:/app/your_lcm_types_package \
-e LCM_PACKAGES=your_lcm_types_package \
--network=host \
-d \
docker run -d --network=host \
-v /path/to/my_lcmtypes:/app/my_lcmtypes:ro \
-e LCM_PACKAGES=my_lcmtypes \
mbari/lcm-websocket-server
```

Note that the `HOST`, `PORT`, and `CHANNEL` environment variables specified above are the defaults for the `mbari/lcm-websocket-server` image. These can be omitted if the defaults are acceptable.
The image has a `HEALTHCHECK` on `/healthz`; it cannot reach the server if `HOST` is set to an address other than `0.0.0.0` or `127.0.0.1`.

Extra command-line flags can be passed with the `ARGS` environment variable, e.g. `-e ARGS="--uvloop --processes 4 -vv"`. The image includes the `orjson` and `uvloop` extras and a `HEALTHCHECK` that polls `/healthz`; if `HOST` is set to an address other than `0.0.0.0` or `127.0.0.1`, the health check cannot reach it.
### Build

The `LCM_PACKAGES` environment variable should be set to the name of the package (or comma-separated list of packages) that contains the LCM Python message definitions. The `/app` directory is included in the `PYTHONPATH` so that any packages mounted there (as shown with `-v` above) can be imported.
The build is defined in `docker-bake.hcl` and installs exactly the dependencies in `uv.lock`:

It's recommended to run with `--network=host` to avoid issues with LCM over UDP. This will allow the container to use the host's network stack.
```bash
docker buildx bake # builds mbari/lcm-websocket-server:local
docker/smoke_test.sh mbari/lcm-websocket-server:local # checks both proxies start and report healthy
```

## :wrench: Development

Expand All @@ -273,10 +294,16 @@ uv run pytest
uv run ruff check .
```

The tests are self-contained: they use LCM types vendored in `tests/lcmtypes` and the in-process `memq://` LCM provider, so no LCM network or MolaRS install is needed. CI runs lint and tests on Python 3.11 to 3.14.
The tests use the `compas-lcmtypes` package and the in-process `memq://` LCM provider, so no LCM network is needed. CI runs lint and tests on Python 3.11 to 3.14 and smoke-tests the Docker image.

Before a release, run the load benchmarks and compare against [benchmarks/README.md](benchmarks/README.md) to catch performance regressions. They need the full CoMPAS LCM types on `PYTHONPATH`:
Before a release, run the load benchmarks and compare against [benchmarks/README.md](benchmarks/README.md) to catch performance regressions:

```bash
uv run python benchmarks/load.py all
```

### Releases

[python-semantic-release](https://python-semantic-release.readthedocs.io/) runs in CI after lint, tests and build pass. It derives the version from [conventional commits](https://www.conventionalcommits.org/), updates `pyproject.toml`, `uv.lock` and `CHANGELOG.md`, tags the release, and creates a GitHub release with the wheel and sdist. Pushes to `main` make releases and pushes to `develop` make `-rc.N` prereleases; each release also publishes the Docker image. Nothing is published to PyPI.

Publishing the image needs two repository secrets: `DOCKERHUB_USERNAME`, and `DOCKERHUB_TOKEN`, a Docker Hub access token with read and write access to `mbari/lcm-websocket-server`.
4 changes: 2 additions & 2 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

Load and microbenchmarks used to characterize the server under load.

Both scripts need the CoMPAS LCM Python types (`stdlcm`, `senlcm`, `navlcm`, `geolcm`) on `PYTHONPATH`.
Both scripts use the CoMPAS LCM types from the `lcmtypes` extra.

```bash
export PYTHONPATH=/path/to/lcmtypes/python
uv sync --all-extras
uv run python benchmarks/load.py all # or: latency, json-scale, json-rate, dial, slow, periodic
uv run python benchmarks/encode.py
```
Expand Down
12 changes: 6 additions & 6 deletions benchmarks/encode.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@
Microbenchmark of JSONHandler.handle (decode + JSON encode) on the stdlib
and orjson paths.

Usage (requires stdlcm/senlcm/navlcm/geolcm on PYTHONPATH):
Usage (requires the lcmtypes extra: uv sync --all-extras):
uv run python benchmarks/encode.py
"""
import time

from geolcm import point_t
from compas_lcmtypes.geolcm import point_t
from compas_lcmtypes.navlcm import odometry_t
from compas_lcmtypes.senlcm import multibeam_euclidean_t
from compas_lcmtypes.stdlcm import header_t
from lcmutils import LCMTypeRegistry
from navlcm import odometry_t
from senlcm import multibeam_euclidean_t
from stdlcm import header_t

from lcm_websocket_server.apps.json_proxy import JSONHandler
from lcm_websocket_server.lib.lcm_utils import types as T

reg = LCMTypeRegistry()
for p in ("stdlcm", "senlcm", "navlcm", "geolcm"): reg.discover(p)
reg.discover("compas_lcmtypes")
hdr = header_t(); hdr.frame_id = "base"
odo = odometry_t(); odo.header = hdr
mb = multibeam_euclidean_t(); mb.header = hdr; mb.n_beams = 512
Expand Down
15 changes: 8 additions & 7 deletions benchmarks/load.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
end-to-end latency (from the header_t timestamp stamped at publish) and the
spy channel's reported rates.

Usage (requires stdlcm/senlcm/navlcm/geolcm on PYTHONPATH):
Usage (requires the lcmtypes extra: uv sync --all-extras):
uv run python benchmarks/load.py SUITE [--duration S]

Suites: latency, json-scale, json-rate, dial, slow, periodic, all
Expand Down Expand Up @@ -37,10 +37,10 @@ def _cpu_seconds() -> float:

def _message_builders():
import numpy as np
from geolcm import point_t
from navlcm import odometry_t
from senlcm import image_t, multibeam_euclidean_t
from stdlcm import header_t
from compas_lcmtypes.geolcm import point_t
from compas_lcmtypes.navlcm import odometry_t
from compas_lcmtypes.senlcm import image_t, multibeam_euclidean_t
from compas_lcmtypes.stdlcm import header_t

from lcm_websocket_server.lib.image import PixelFormat

Expand Down Expand Up @@ -92,6 +92,7 @@ def run_server(port, mode, server_kwargs, streams, ready, go, window_end, stop,
import logging

import cv2
from compas_lcmtypes.senlcm import image_t

from lcm_websocket_server.apps.dial_proxy import (
DialHandler,
Expand All @@ -113,11 +114,11 @@ def emit(self, record):

pubsub._logger.addHandler(DropCounter())

registry = build_registry(["stdlcm", "senlcm", "navlcm", "geolcm"], logging.getLogger("bench"))
registry = build_registry(["compas_lcmtypes"], logging.getLogger("bench"))
json_handler = JSONHandler(registry)
if mode == "dial":
encoder = DownsamplingMJPEGEncoder(scale=1.0, params=[cv2.IMWRITE_JPEG_QUALITY, 75])
inner = DialHandler(ImageMessageToJPEGHandler(encoder), json_handler)
inner = DialHandler(ImageMessageToJPEGHandler(encoder, image_t), json_handler)
else:
inner = json_handler

Expand Down
22 changes: 22 additions & 0 deletions docker-bake.hcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Image build definition shared by CI and local builds.
#
# docker buildx bake # build mbari/lcm-websocket-server:local
# docker/smoke_test.sh mbari/lcm-websocket-server:local
#
# In CI, docker/metadata-action overrides the tags and labels of the
# "docker-metadata-action" target.

target "docker-metadata-action" {
tags = ["mbari/lcm-websocket-server:local"]
}

target "image" {
inherits = ["docker-metadata-action"]
context = "."
dockerfile = "docker/Dockerfile"
platforms = ["linux/amd64"]
}

group "default" {
targets = ["image"]
}
Loading
Loading