Repository navigation
subscribe-sdk: Add the Rust SDK package - #37483
Closed
bobbyiliev wants to merge 4 commits into
Closed
bobbyiliev wants to merge 4 commits into
bobbyiliev wants to merge 4 commits into
Conversation
…ier engine, and cohorts
bobbyiliev
changed the title
bobbyiliev
added a commit
that referenced
this pull request
Oct 21, 2026
…37463) Proposes the Materialize SDK: a Rust protocol core with no I/O, compiled into thin per-language packages that use their native drivers, with `subscribe` and `sink` modules. It answers the Sink SDK product requirements, includes the subscribe semantics review checked against the code, plans the move onto durable subscriptions (#38468), and makes turbopuffer the first sink. Prototypes: #37483 (Rust), #37517 (Python). Diagrams (Materialize-internal): https://claude.ai/artifact/Q6193A4KskJpfWgUuYNY6v --- **TL;DR** - **What:** an official SDK for building your own sink from Materialize to anything: search indexes, caches, your own Postgres, webhooks. Rust first, with Python and Node to follow. - **Why customers care:** no lost or duplicate data after a crash or restart. Several views can be written at one consistent point in time, which log-based CDC tools cannot do from offsets alone. No Kafka needed. - **How it's built:** every hand-written sink we found had a different bug in the same logic: when a batch is safe to write, how to resume without gaps, how to line up several views. So that logic lives once, in a small Rust library (the protocol core) compiled into each language package. It runs inside the customer's app, not as a service, and never opens a connection. Each language keeps its usual Postgres driver. Python, Node and later languages therefore behave the same, and one fix reaches every language in one release. - **What a sink author writes:** open the target, write a batch, commit. The SDK handles the snapshot, resume after restart, retries, dead-lettering and clear errors. - **First proof:** a turbopuffer sink running on our internal context graph in October. - **Input wanted:** the name (deferred question 1), whether the SDK can rely on subscribing by id (question 2), and what a sink should do when a blue/green deploy swaps the view it reads (question 3).
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Rust package (
mz-subscribe) of the Subscribe SDK undermisc/subscribe-sdk(its own isolated Cargo workspace), plus the shared top-level README: the layered, correct-by-construction client (raw stream, one shared min-frontier consistency engine, single-view consistent batches and multi-view cohorts, and a bounded background drain that never backpressures Materialize) with an opaque resume token that bakes in theAS OF frontier-1arithmetic. The Python package is a companion PR (#37517); design in #37463.