Skip to content

feat(node): a Firecracker backend, behind --backend firecracker - #370

Merged
WaylandYang merged 3 commits into
devfrom
feat/firecracker-backend
Oct 29, 2026
Merged

WaylandYang merged 3 commits into
devfrom
feat/firecracker-backend

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Not yet run on KVM. None of bm, bj or hk has /dev/kvm. This is compiled and unit-tested on macOS only. DESIGN-v0.6-FIRECRACKER.md lists what to verify first on a KVM host; until then every behaviour here is an assumption.

The optional backend from DESIGN-v0.6 §2, on upstream Firecracker (no fork), built on the Backend trait from #368.

What it keeps from v0.5

Shared page cache across forks every child of a tag maps one memory file (File backend, MAP_PRIVATE)
Diff snapshot chains a restored sandbox snapshots next as only its dirtied pages; chains are hard-linked into each image, assembled once into a cached dot file
Private disk per sandbox reflinked; refuses to start where reflink does not work (DESIGN-v0.6 §7.5)
Clock after restore date -s through the agent

Not included: live fork (needs the vendored fork; upstream #6270 is not merged), hugepages (file-backed hugetlbfs restore is upstream #6099), user volumes (refused with an error for now).

How

  • The trait now covers preparation (template_root, prepare, layer, unmount, teardown), because a VM needs a disk image, not an overlay. runsc's implementations call bundle.rs exactly as before.
  • Memory and disk travel together. Every image carries the disk as it was while the VM was paused. The VM's disk path is the relative rootfs.ext4, and Firecracker runs in the sandbox's own disk directory, so a vmstate opens the loading sandbox's copy, never the parent's. Then resume_vm:false → PATCH /drives/rootfs → resume, as in feat(vmm): give every restored child its own rootfs backing #321; any failure kills it unresumed.
  • Fixed guest address 10.42.0.2 behind a tap in each netns, DNATed from the netns address, so snapshots restore into any namespace.
  • Same cgroup as runsc, joined before exec; own session. Freeze, accounting, cgroup.kill and restart adoption are unchanged code.
  • Guest: the agent starts the template's program from sandstorm.app= on the kernel command line. Port 8888 belongs to the agent.

Testing

  • macOS: cargo test -p sandstorm-core (173 lib + integration) and cargo clippy --all-targets -D warnings pass. New tests: the API client against a fake socket, load ordering (no resume before the re-point, none after a failed one), chain reading/assembly, sparse diff application, boot args and the 2048-byte limit, prepare. Agent: rootfs-init/tests/test_agent_app.py.
  • The Linux-only code (FICLONE, SEEK_DATA) is compiled for the first time by this PR's CI. CI's ext4 cannot reflink, so the two reflink tests skip there; apply_diff has its own test that runs.
  • Unrelated and pre-existing on dev: wake::tests::a_wake_behind_a_full_queue_is_refused_at_once fails whenever it runs alone, and passes in the full suite.

First things to check on a KVM host

  1. That a Diff after snapshot/load with dirty tracking holds exactly the pages dirtied since the load. If not, chains lose pages.
  2. Boot, freeze/thaw, hot ↔ snapshot, branch + fork, detach + reattach, destroy, and a sandstormd restart with VMs running.
  3. The four tiers' numbers; runsc's do not carry over.

🤖 Generated with Claude Code

@WaylandYang
WaylandYang force-pushed the feat/firecracker-backend branch from 0024ccc to 7196c54 Compare October 29, 2026 10:36
WaylandYang and others added 3 commits October 29, 2026 19:13
NOT YET RUN ON KVM: none of bm, bj or hk has /dev/kvm. Compiled and
unit-tested on macOS only; DESIGN-v0.6-FIRECRACKER.md lists what to
verify first.

Upstream Firecracker, no fork. From v0.5 it keeps the three things v0.6
kept: forks of one tag share a page cache (File backend, MAP_PRIVATE on
one memory file); a restored sandbox snapshots next time as a diff of the
pages it dirtied, chained onto its image; and every sandbox has its own
reflinked disk. Live fork stays out -- it needs the vendored fork and
upstream #6270 is not merged.

- The Backend trait now prepares the sandbox too (template_root, prepare,
  layer, unmount, teardown): a VM needs a disk image, not an overlay.
  runsc's implementations call bundle.rs exactly as before.
- Every image carries the disk as it was while the VM was paused, since
  the guest's page cache refers to disk blocks. The disk's path in the VM
  is the relative rootfs.ext4 and Firecracker runs in the sandbox's own
  disk directory, so a vmstate opens the loading sandbox's copy, never
  the parent's; then load with resume_vm:false, PATCH /drives/rootfs,
  resume, as in #321. Any failure kills it unresumed.
- Diff chains are hard-linked into each image, so dropping a tag never
  breaks one derived from it; assembled once into a dot file that tag
  transfer leaves out.
- Every guest is 10.42.0.2 behind a tap in its netns, DNATed from the
  netns address, so a snapshot restores into any namespace.
- Firecracker joins the sandbox's cgroup before exec and runs in its own
  session: freeze, accounting, cgroup.kill and restart adoption are the
  runsc code unchanged.
- Refuses to start without /dev/kvm, the kernel, or a work dir that can
  reflink.
- The agent starts the template's program from sandstorm.app= on the
  kernel command line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
@WaylandYang
WaylandYang force-pushed the feat/firecracker-backend branch from 7196c54 to bc030dd Compare October 29, 2026 11:13
@WaylandYang
WaylandYang merged commit 5f05568 into dev Oct 29, 2026
11 checks passed
@WaylandYang
WaylandYang deleted the feat/firecracker-backend branch October 29, 2026 11:19
WaylandYang added a commit to Floating-Y/sandstorm that referenced this pull request Oct 29, 2026
dev now runs the archive through the Backend trait (deeplethe#368, deeplethe#370). Resolving
the conflict in detach_with_commands keeps this PR's two-process pipeline
and adds what dev needed:

- tar gets -S again. A Firecracker layer is a sparse rootfs.ext4; without
  it tar reads every hole and the restore writes them all out.
- The archive directory is fsynced after the rename. The local layer and
  snapshot are deleted right after, so the rename must be durable first.
- Stdio is imported, and the test fixture builds its Runtime with
  Runtime::new now that Runtime has an engine and a fingerprint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
WaylandYang added a commit to Floating-Y/sandstorm that referenced this pull request Oct 29, 2026
dev now drives checkpoints through the Backend trait (deeplethe#368) and has a
Firecracker backend (deeplethe#370), so this PR's fix is redone on top of them.
The idea is unchanged: an attempt writes into a directory of its own and
is renamed into place, without replacing anything, only once it checks
out. What changed in the reworking:

- Validation is `Backend::validate`. The runsc check keeps this PR's
  statefile-header and pages-file rules; Firecracker checks vmstate,
  rootfs.ext4 and an unbroken mem chain. Before, every Firecracker image
  would have failed to publish.
- `observe` runs no subprocess: `runsc statefile` and the validation cache
  are gone. A published image carries a seal listing its files' lengths
  (dot names excluded, so tag listings skip it and backend caches such as
  .mem.assembled do not break it); an image from before sealing is judged
  by `validate`. The cgroup is read first, so a running sandbox is never
  reported as Snapshot.
- Failed attempts are removed at once, or by the next attempt or a
  destroy if a writer is still holding one, rather than kept until
  someone destroys the sandbox by hand. `create` no longer refuses.
- renameat2 falls back to check-then-rename on EINVAL; one sandbox's
  transitions never run concurrently, so that holds.
- `branch` publishes its tag the same way, so a failed branch no longer
  leaves a tag that blocks the name forever.
- Nothing is fsynced, as before: this guards against a failed checkpoint,
  not a power cut, and syncing whole images would sit on the snapshot
  tier's critical path.

Tests use the in-memory Recorder backend, so they run on any OS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant