Repository navigation
pull in changes from upstream - #3
Merged
Anthony Riley II (anth1y) merged 73 commits intoOct 28, 2026
Merged
Conversation
With TCP window scaling (RFC 1323), 4 MiB rx/tx ring buffers lift the Bandwidth-Delay Product ceiling from ~2 MB/s (at the previous 64 KiB) to ~70 MB/s at 50 ms RTT. This change is a prerequisite for the BDP gain from the upcoming event-driven poll loop; in isolation it does not move throughput because the 1 ms thread::sleep poll loop bounds the smoltcp \xe2\x86\x94 guest internal RTT before the receive window matters.
The smoltcp poll loop previously slept on a 1 ms timer between
iterations, which inflated the smoltcp <-> guest internal RTT and capped
in-VM throughput at ~1 MB/s regardless of buffer sizes. Real network
RTT to a CDN is tens of milliseconds, but the sleep added a ~1 ms floor
that prevented smoltcp's congestion window from opening past a single
MTU per round.
Replace the dedicated thread's blocking sleep loop with a Tokio
current-thread runtime hosting an AsyncFd-based select! over:
1. Host AF_UNIX FD readability (frame from guest)
2. cmd_rx.recv() (StackCommand from proxy engine)
3. sleep_until(poll_delay) (smoltcp's own retransmit / delayed ACK)
This keeps the netstack on its own OS thread (so a busy smoltcp poll
cannot starve the proxy's TLS / DNS / MITM workers on the shared
multi-thread runtime) while making it react in microseconds instead of
milliseconds.
The 1 ms floor and 50 ms clamp on poll_delay are removed; the loop now
honors smoltcp's timer exactly, including Some(0) which means continue
immediately. Burst commands are coalesced via try_recv after each
recv().await wake.
device.rs surfaces a DrainStatus { WouldBlock | HitCap | Error } so the
loop can avoid awaiting when the kernel buffer still has frames (cap
hit) and can correctly hand off readiness to AsyncFd otherwise. The
host FD remains in blocking mode for sends; reads continue to use
MSG_DONTWAIT so AsyncFd's readable() never resolves into a blocking
read.
Measured throughput on a 90 MB Cloudflare speed-test download
(speed.cloudflare.com/__down?bytes=90000000): ~1.12 MB/s -> ~65 MB/s
(~58x), reaching ~66% of host-native throughput on the same network.
Proxy CPU rises from ~3% (sleep-bound) to ~15% (actual work).
With the sleep-driven poll loop replaced by an event-driven select loop,
the 256-frame ingress cap no longer prevents kernel buffer drops (the
loop wakes immediately on FD readability), but it still fragments a full
4 MiB BDP burst (~2 800 frames) across ~11 mini-poll cycles. Each cycle
runs a complete iface.poll, socket walk, drain_pending_sends and DNS
poll over what is effectively the same data window.
Raising the cap to 4096 lets a typical BDP-sized burst fit in a single
batch:
- one smoltcp poll instead of ~11
- one ACK round (better cwnd growth) instead of ~11
- less HashMap churn in inspect_pending_frames
- HitCap returns to being a real backstop for runaway flows rather
than the steady-state path
Memory is bounded by MAX_PENDING_FRAMES * MTU = 4096 * 1514 ~ 6 MiB
transient, freed each poll cycle, well below the existing kernel
SO_RCVBUF (4 MiB) ceiling.
Measured impact at this stage is small (within trial-to-trial noise on
the 90 MB Cloudflare workload) because the next binding constraint is
per-frame syscall and allocation overhead in the RX path. The real
value of this commit is correctness/clarity: HitCap is now a genuine
backstop for runaway flows rather than the steady-state path.
Add focused unit tests for the new DrainStatus return type added with the event-driven poll loop: - drain_recv_returns_wouldblock_on_empty_fd - drain_recv_accumulates_non_icmp_frames - drain_recv_discards_icmp_frames - drain_recv_returns_hit_cap_when_pending_rx_fills A small RAII Pair helper wraps an AF_UNIX SOCK_DGRAM socketpair with generous send/recv buffers so the cap test can queue >MAX_PENDING_FRAMES datagrams without blocking. Tests run synchronously (drain_recv is sync, relies on MSG_DONTWAIT) so they need no tokio runtime.
VZFileHandleNetworkDeviceAttachment defaults its MTU to 1500 unless
setMaximumTransmissionUnit: is called. We never called it, so every
guest TCP segment was capped at the wire MTU and host smoltcp paid
per-1.5-KiB-frame syscall + checksum + TCP processing costs.
Apple's API supports IP-MTUs up to 65535. Setting it to the maximum
hands smoltcp ~64 KiB super-frames per syscall, collapsing per-frame
costs ~40x. In our host<->guest topology there is no wire NIC to
fragment past, so jumbo frames are strictly free.
Coordinated changes:
- shuru-darwin: FileHandleNetworkAttachment::new(fd, mtu) now calls
setMaximumTransmissionUnit:.
- shuru-vm: pass NET_IP_MTU = 65535 from sandbox.
- shuru-proxy: device::MTU = 65549 (14 + 65535) so smoltcp matches
the attachment frame size; MAX_PENDING_FRAMES 4096 -> 256 to keep
transient buffer ~16 MiB at the new frame size.
- shuru-proxy: SO_RCVBUF 4 -> 8 MiB, SO_SNDBUF 1 -> 2 MiB to hold
~128 in-flight super-frames per Apple's 4x ratio guidance.
- shuru-linux: signature parity for the non-Darwin stub.
Cloudflare 90 MB download in-VM: ~65 MB/s -> ~89.9 MB/s mean (4 trials,
cold-start dropped). Now ~91% of host-native (~99 MB/s). All 20
shuru-proxy unit tests pass; --allow-host enforcement still blocks
non-allowed domains at the DNS gate.
Adds cgroup v2 controllers, BPF for cgroup device filtering, seccomp filter, veth, bridge netfilter and the xtables/nftables pieces Docker needs. Everything stays built in, no modules.
Size capped tmpfs for /run (64m) and /dev/shm (256m). cgroup2 is mounted with nsdelegate and memory_recursiveprot and every available controller is enabled in the root subtree so container runtimes get memory, cpu, pids and io limits.
Lets builds and tests use an isolated set of kernel, initramfs, rootfs and checkpoints without touching the installed ones.
prepare-docker-checkpoint.sh creates a checkpoint with pinned docker-ce, containerd.io and a digest pinned nginx-unprivileged image, with the trixie security and updates archives enabled. build-kernel-hf.sh cross compiles the kernel on Hugging Face Jobs for hosts without the disk for a local build.
Nine ignored integration tests cover non-root execution with default isolation, HTTP through port forwarding, egress policy, host isolation, resource limits and volume persistence across daemon and VM restarts.
An image declares long running processes in /etc/shuru/services/<name>.json. Init starts them after networking, gates exec requests on their readiness check, and on SHUTDOWN_REQ sends SIGTERM, waits out each stop timeout, SIGKILLs what remains, syncs, and replies. The host Sandbox::stop() now performs that handshake before power off and kill() is the old hard stop, so checkpoints are cloned from a consistent disk on every stop path. The separate sync exec before checkpoint creation goes away.
The checkpoint build now installs Docker and declares the service in a first step, then boots from that checkpoint, where dockerd is already up, to pull the pinned image. with-dockerd is gone; the tests and README call docker directly. Nine tests still pass, in 79s instead of 125s because the daemon starts once per boot.
C string literals instead of hand terminated byte strings, pointer casts through cast() so they stay correct where c_char is signed, a while let for the vsock read loop, io::Error::other, and a literal header line in checkpoint list. No behaviour change.
Shows a container served through port forwarding from the docker checkpoint and the service file that starts dockerd at boot.
A guest that already died cannot answer, and the vsock connect would retry for ten seconds before falling through to the hard stop.
A recipe is a directory with recipe.json and shell steps, fetched from GitHub at an exact commit or read from a local path. shuru recipe list shows the official catalog in superhq-ai/recipes, shuru recipe show prints one, shuru checkpoint create NAME --recipe REF builds it, and shuru run --recipe REF builds on first use and reuses the checkpoint until the recipe changes. Each step runs in a fresh boot from the previous step, services declared by the recipe are in place from step one, and the checkpoint records its source and revision in a sidecar shown by checkpoint list. Third-party recipes ask for confirmation or --yes. The CLI moves to 0.8.0 because the docker recipe declares that as its minimum.
The two step script moves to the docker recipe in superhq-ai/recipes, where a pull by digest is tagged afterwards so docker run by tag works offline. README, changelog, RFC 0003 and 0004 and the website show shuru checkpoint create docker --recipe docker and the shuru run --recipe docker one-liner. The RFC 0003 suite evidence is regenerated against a recipe-built checkpoint.
proto, vm and guest move to 0.4.0 for the shutdown frames and service model, the sdk to 0.5.0 for the new vm and proto, and the lockfile follows the cli at 0.8.0.
shuru checkpoint update NAME takes the same arguments as create and replaces an existing checkpoint, or creates it when missing. The new checkpoint is built under NAME.update and swapped in only when the command exits 0, so a failed update leaves the old one in place. --from NAME layers the command onto the existing checkpoint and --recipe REF rebuilds from a recipe at the current revision. create now points at update instead of asking for a delete. Closes #44
SHURU_DATA_DIR, SHURU_KERNEL, SHURU_INITRD, SHURU_ROOTFS, SHURU_STORAGE and SHURU_BOOT_TRACE were undocumented. They are the per-machine layer; permissions stay on flags and in shuru.json.
Two recipes that share a name, such as a fork and the official one, would otherwise rebuild over each other on every switch. The flag requires --recipe; the default stays the recipe name.
v0.8.0-rc.1 and the like are rehearsals of the workflow: marked pre-release so shuru upgrade and the GitHub latest release ignore them, and the Homebrew tap is left alone. The changelog heading moves from Unreleased to 0.8.0.
parse_reference probed ./NAME/recipe.json for any reference, so a cloned repository could shadow an official recipe, inherit its trust and run its own steps as root in the build VM without the prompt. Only ./ ../ / ~/ . and .. are local now. A bare name that also exists locally prints a note pointing at ./NAME. Found by the Superagent scan on PR #45.
The browser recipe in superhq-ai/recipes installs Chromium from Debian and runs it headless as a guest service under an unprivileged user, DevTools on port 9222. The README section and the Browser Use example switch to shuru run --recipe browser, with the example keeping the VM alive instead of launching Chromium as root with --no-sandbox. A gated test builds the recipe and checks CDP inside the VM and through a forwarded host port.
Docker inside the VM, guest services and recipes (RFC 0003, 0004, 0005)
The hero section carried a section label like the ones that sit on the dashed borders further down, but there is no border above it, so the label floated in empty space and the padding that made room for it pushed the heading down. The label and its style block are gone and the top padding now only clears the fixed nav.
Browser recipe: headless Chromium as a guest service
www: drop the Hero label and pull the heading up
The spec lists stats, free page hints and reporting as queues 2, 3 and 4, but each exists only when its feature is negotiated and a queue that does not exist takes no index. The Linux driver, Cloud Hypervisor, crosvm and QEMU all number the optional queues that way, in spec order, skipping the absent ones. The transport now hands the backend the feature set the driver accepted at FEATURES_OK, and the balloon builds its queue layout from it: inflate and deflate first, then each optional queue in spec order when its feature is on. Every index maps to a named role; nothing goes through a wildcard, and the device declares three queues instead of five. Re-measured on a1.metal with this layout: 1109 MB to 87 MB resident in nine seconds after the guest freed 1 GB, target 1 GB and back to 2 GB move guest MemFree 1975 to 951 to 1887 MB, reporting worker idle after.
Memory balloon with free page reporting
Every Linux VM asked vhost-vsock for guest CID 3. vhost CIDs are global to the host kernel, so a second VM got EADDRINUSE, the error was printed and ignored, and its host side connected to the first VM. On a1.metal a second `shuru run` executed inside the first sandbox. The Linux backend now emulates virtio-vsock in the VMM. Each VM owns its device and CID space, the host end of a connection is a socketpair in the shuru process, and guest-initiated connections are refused. Host data stays in the socket until the guest posts an rx buffer and is read straight into it, sized to that buffer and to the guest credit; control messages are per-connection flags, so nothing queues in the VMM. Sockets are registered once, edge-triggered. Every boot also gets a random token and instance id on the kernel command line. The guest requires the token as the first frame of every control and forward connection and answers with the instance id, which the host checks. The token is redacted from serial output. Protocol version 2. CI runs the proto, linux, vm and guest tests on a Linux arm64 runner.
…guest memory Every virtio device sat behind one bus lock and disk and shared folder requests ran on the kicking vCPU while holding it. Each device now has its own lock and a thread of its own (vsock keeps its worker). Queue kicks are ioeventfds registered with KVM, interrupts fire only when buffers were used and the guest did not suppress them, and the interrupt status and line change under one lock so an ack cannot lower the line over a new reason. Avail and used index accesses get the barriers they were missing. vsock reads host data with one readv straight into the payloads of as many guest rx buffers as the credit allows, and sends guest data with one sendmsg from guest memory, copying only what the socket does not take. Rx buffers are read from the ring only as packets need them. Guest kicks are off while the worker runs, and it sleeps until the next deadline instead of waking every 100 ms. a1.metal, three runs against 0.9.3 vhost: guest to host 434 to 451 MB/s (vhost 117 to 191), host to guest 315 to 360 (353 to 366), exec p50 1.8 ms (1.6), shared folder read 343 to 384 MB/s (241 to 245).
The guest agent is now /init in an uncompressed initramfs. It mounts the root itself, configures networking and replays the journal through the kernel, so the initramfs gunzip, e2fsck and offline resize2fs are gone from the boot path and the rootfs no longer carries a copy of the agent. A larger disk grows online with EXT4_IOC_RESIZE_FS. The initramfs carries /dev/console and /dev/null, without which Rust std aborts the agent before main on a kernel that has no console for it. The kernel tick goes to 1000 Hz: virtio_pci_modern sleeps a millisecond at a time during the asynchronous Apple device reset, and each sleep cost two jiffies at 100 Hz. On APFS the base image is rewritten sequentially and extended to 4096 MB once, so cloning it costs well under a millisecond instead of 0.5 to 1.5 s. Disks default to 4096 MB, which is also the minimum. `shuru run -- true` on an M-series Mac goes from about 300 ms to about 160 ms. Every checkpoint path freezes ext4 in the guest before the host copies the disk. A seal carries an id and a lease, the guest thaws itself when the lease runs out, and the host publishes the copy only after a matching thaw, so a failed save keeps the previous checkpoint. The root mounts errors=remount-ro and a disk with recorded errors refuses to boot or seal. Seals run FITRIM, and trim is available from the stdio protocol and both SDKs. The Linux block device supports discard and write zeroes as hole punching, with every range and descriptor chain bounds checked. Protocol version 3. Fixed on the way: watch events carry their names again, files of 1 MB and up travel in 256 KiB frames, a macOS VM that powers off during boot no longer hangs the connect, and checkpoint saves report stopped services. RFC 0010.
… the guest does A guest could grow the host shuru process without limit: by writing to a socket the remote never read, by flooding a socket the guest never read, or by printing to a spawned process stdout the client never consumed. The old build reached 2 GB in five seconds on the last one. The proxy holds a 128 KiB window per connection each way and reads from the guest only as the upstream takes data. Closed and unanswered sockets are reaped on timeouts, connection ids carry a serial so a reused socket handle cannot get the data of a dead connection, DNS has 64 queries in flight, and chunk size lines stop at 4 KiB. smoltcp buffers are halved to 512 KiB and 256 KiB at the same measured throughput. stdio and SDK event queues are bounded, and buffered exec output and file reads fail past 64 MiB with an error that points at streaming. On Linux the VMM validates every virtqueue size, ring address and descriptor chain, bounds net TX frames and balloon inflates, and caps virtio-fs handles, inodes, names and request sizes. Stdin input is now a request answered once the data reaches the guest; the TypeScript SDK keeps 1 MiB in flight and the stdio server refuses more than 4 MiB of unanswered stdin per process. A kill sends SIGTERM and, if the process is still running two seconds later, a KILL_REQ on its own connection, so it no longer waits behind stdin nobody reads. The guest also kills the process group when the host hangs up on a piped exec. smoltcp could read a guest SYN straight from the socketpair while it polled, skipping the pass that adds a listener, and answered with a reset. It now only sees inspected frames; 600 back to back connections succeed where 150 to 475 did before. macOS, footprint of the shuru process: host flood 1.3 GB -> 6 MB, stdout flood 2 GB -> 5 MB. a1.metal: 6 to 11 MB in every scenario but 400 stalled connections (200 MB), and a sandbox in a 1280 MB cgroup with 1024 MB of guest RAM survived filling its RAM plus that flood. RFC 0011.
The proxy connection handlers take the guest side and the shared engine state as two structs instead of ten and eleven arguments. The rest is what clippy asked for: div_ceil, io::Error::other, is_multiple_of, struct init over field assignment, collapsed ifs, casts to the same type, unused imports, and FUSE structs and a method nothing used. shuru-linux had never been linted, since it does not build on macOS. Clippy is clean on macOS, on Linux arm64 and for the musl guest.
linux, vm, guest, proto: per-VM userspace vsock and a connection hello
The watch that kills the process group when the host hangs up polled every 200 ms, and the exec joined it before sending the exit frame, so every exec took up to 200 ms longer than its command. It now waits on a pipe as well as the connection and returns the moment the pipe closes. The exec closes it once the child has exited but before it is reaped, so a late hangup still cannot reach a reused pid. shuru run -- true on an M-series Mac: 370 to 400 ms -> 140 to 160 ms.
The Linux backend emulates a PL011 and boots with console=ttyAMA0, but the kernel was built without the PL011 driver, so a Linux guest printed nothing and --verbose showed no kernel log. The driver is built in now. The Linux bootloader also stripped quiet from every command line. With a working console that pushes the whole kernel log through the emulated UART on every boot, 830 ms against 600 ms on a1.metal, for output that goes to /dev/null. Linux keeps quiet unless --verbose is set, as macOS does.
shuru-proto 0.6.0, shuru-guest 0.6.0, shuru-proxy 0.5.0, shuru-store 0.2.0, shuru-darwin 0.2.3, shuru-linux 0.4.1, shuru-vm 0.7.0, shuru-sdk 0.9.0, shuru-cli 0.11.0, @superhq/shuru 0.5.0.
fast boot, sealed checkpoints and bounded host memory
The Linux node runtime (0008) and Linux VM snapshots (0009), updated for what shipped in 0.10.0 and 0.11.0: the vsock device and hello as built, measured vsock numbers, the cgroup envelope and hosted limits from RFC 0011, and the 0.11.0 cold boot as the restore baseline. How the hosted service runs, including where snapshots are stored off the host, is kept out of this repository, so the RFCs refer to it by name rather than by number.
A snapshot holds the running kernel and agent of the release that took it, so refusing every snapshot from an earlier release would break every template and paused sandbox on each upgrade. The manifest records the release and guest protocol version, the VMM reads device state from the last few releases, and the host keeps their guest protocols answerable. Past that window a resume is a reported reboot from the snapshot disk.
A jailed VMM cannot open paths, so the KVM layer now takes /dev/kvm, kernel, initrd and disk as open files. VirtualMachine::new opens them from the configuration as before; from_descriptors takes them ready. serve_vsock accepts host connects on a listening unix socket: the client writes "CONNECT <port>\n" and reads "OK <host port>\n" once the guest accepts, as in Firecracker. The accepted socket becomes the host end of the connection directly. The connection cap is per VM now, 1024 by default. Also drops the deprecated time_t alias in virtio-fs, which warned on musl.
The first piece of RFC 0008. shuru-vmm runs one VM. It inherits only a listening control socket, gets its configuration and every descriptor in one message on it, builds the VM and installs a seccomp allowlist on all threads before the vCPUs start. clone3 gets ENOSYS so thread creation falls back to clone, whose flags the filter checks. shuru-node run is the root side for now: it opens everything, creates shuru.slice/sbx-<slot> with the RFC 0011 envelope, and starts the VMM in new mount, pid, net, ipc, uts and cgroup namespaces, on an empty read only root, with no capabilities, uid 100000 + slot, no_new_privs and rlimits. The binary is executed from a descriptor, so the root stays empty. On a1.metal the jail costs 6 to 17 ms and run -- true takes 372 to 416 ms against 414 to 427 ms for the CLI. In kill mode an exec, disk writes, 64 MiB over vsock and stop ran without a kill.
smoltcp counts a socket in SYN-RECEIVED as active, so the stack handed the proxy a connection before the guest acknowledged the SYN-ACK. A connection the proxy refused right away, such as an address outside the allowlist, was closed in that state and never sent its FIN, and the guest waited for its own timeout. It now gets an empty reply at once. A listener stuck in SYN-RECEIVED is reaped like one still listening.
With --allow-net, shuru-node starts shuru-netd for the sandbox: the shuru-proxy stack in the same jail as the VMM, but with the host network, its own uid (200000 + slot) and a cgroup next to the VMM. The node opens the frame socketpair and hands netd one end; netd answers with the secret placeholders and its CA, which the node installs in the guest before the command runs. What netd can reach: - the proxy refuses any destination that is not a public address (public_only), which also turns off the host exposure path; - the host firewall drops non-public IPv4 and all IPv6 sent by the netd uids, in an nftables table of its own; - upstream TLS is checked against roots the node sends, so nothing is read from disk. Seccomp for netd is a deny list: no exec, no namespaces, no tracing, no module or key calls, IP sockets only. Both processes now refuse openat with EPERM instead of killing. glibc malloc opens /sys/devices/system/cpu/online when a thread adds an arena, and whether that happens before or after the filter is timing; the root is empty, so the open could never succeed anyway. Tested on a1.metal with both filters in kill mode: https through netd, metadata, host private and gateway addresses refused, allowlist, a secret substituted only on its host, netd in /proc as designed, nft drops by uid, and nothing left behind.
The acceptor read every CONNECT line itself, so a client that connected and sent nothing held up all others for the 2 s timeout. Each client now gets a short-lived thread, at most 16 at a time; past that a client is closed at once. On a1.metal under kill-mode seccomp: a connect behind a silent client answers in under a millisecond, behind 21 silent clients it is closed, and once they time out connects work again.
rustc links crt-static aarch64 Linux executables with -static -no-pie, since its target specs do not enable static-pie, on gnu and musl alike. Adding -static-pie to the link alone gives a PIE that crashes at start: gcc still picks crt1.o, which does not relocate. scripts/static-pie-cc drops both flags and links with -static-pie, so the binaries start from rcrt1.o and relocate themselves. just build-node uses it. On a1.metal, shuru-vmm and shuru-netd load their code at a different address on every run, under kill-mode seccomp.
node: jailed shuru-vmm (RFC 0008)
The vsock listener answers OK <host port>, the VMM runs from a descriptor in an empty root as static PIE, openat gets EPERM instead of a kill, shuru-netd has a deny-list filter and public-only egress enforced by the proxy and nftables, and the a1.metal measurements are in.
docs: RFC 0008 node runtime and RFC 0009 Linux VM snapshots
Anthony Riley II (anth1y)
merged commit Oct 28, 2026
f21559d
into
xbow-engineering:main
5 checks passed
This was referenced Oct 28, 2026
Proxy enforces the host allowlist only on port 443 (SNI); HTTP Host and other protocols bypass it
#4
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.