Skip to content

Commit 858303f

Browse files
kalaspuffclaude
andauthored
perf: cheaper per-message consume and publish paths (#7)
**Stack** (merge in order; each PR is based on the previous one): 1. #2 fix(consumer): wake the dispatcher and the event pump immediately when stopping 2. #3 perf(consumer): free handler slots when settlement starts and bound the settling backlog 3. #4 perf(consumer): batch acks and retries (full batches at once, 5 ms linger) 4. #5 feat(consumer): full receive batches (min_receive_batch, receive_batch_wait, default prefetch) 5. #6 feat!: lean defaults - raw envelope, opt-in CloudEvents and MD5 checks, immediate stop 6. #7 perf: cheaper per-message consume and publish paths 👈 this PR --- Part 6 of 6 of the **fast defaults** work for 0.2.0 (overview in #2), driven by profiles of the consume and publish paths (single CPU, py-spy and cProfile). - Settlement awaitables are resolved in batches (`runtime::Completions`: one interpreter attach and one `call_soon_threadsafe` for everything that completed meanwhile) and run as tracked engine tasks (`close()` waits for them; cancelling the awaitable no longer stops a started settlement). - Automatic acks are not awaited per message: `Delivery.ack_in_background()` queues the delete and the settle loop finishes it (failures: `message.ack_failed` events and metrics). - Acks are queued synchronously when a settling slot is free, so they reach the settle loop in request order; the send-at-once rule now counts the awaited settlements in a batch (`mark_awaited`), which keeps explicit-ack batches full. - Python: cached event enablement and log targets, cached type names and encoders (built-in codecs only), no route-list copies, guarded event emission, telemetry fast paths without spans. - Rust: decoding no longer copies messages; interned dict keys. **Tests:** `a_batch_every_handler_waits_for_is_sent_at_once`, `background_acks_are_tracked_until_close`, `background_acks_wait_for_a_settling_slot` (Rust); `test_stop_completes_automatic_acks`, handlers do not wait for automatic acks, encoder cache, event enablement, routing caches. **Measured** (fantest, all scenarios, the whole stack against 0.1.0; all 360 trials independently verified: every message delivered exactly once with an identical body, every consumed message deleted): | | 0.1.0 defaults | 0.1.0 tuned | defaults | auto-ack | tuned config | |---|---:|---:|---:|---:|---:| | consume, 1 poller: client CPU µs/msg | 171.5 | 85.7 | 64.2 | 44.8 | 50.4 | | consume, 8 pollers: client CPU µs/msg | 126.4 | 93.9 | 60.4 | 52.6 | 54.8 | | consume, 1 poller: MiniStack CPU µs/msg | 335 | 130 | 135 | 125 | 135 | | messages per receive (1 / 8 pollers) | 2.9 / 3.8 | 10.0 / 9.6 | 10.0 / 9.6 | 10.0 / 9.6 | 10.0 / 9.6 | | publish, 1 caller: client CPU µs/msg | 221.8 | 217.6 | 186.9 | 196.2 | 176.7 | | publish_batch ×10, 1 caller: client CPU µs/msg | 45.5 | 35.0 | 37.0 | 38.3 | 30.5 | | publish_batch ×10, 16 callers: client CPU µs/msg | 45.8 | 32.1 | 36.0 | 35.9 | 29.3 | | teardown ms | 528 | 534 | 10 | 11 | 10 | | threads (max) | 49 | 93 | 15 | 14 | 17 | Compared with #6: consuming costs 119.7 → 64.2 µs/msg with `await message.ack()` and 102.7 → 44.8 µs/msg with returning handlers; publishing 220.9 → 186.9 (single) and 41.6 → 37.0 µs/msg (batch). A probe with this build (2,000 messages, 1 poller, MiniStack) sends 200 `DeleteMessageBatch` calls with either handler style, i.e. 100 per 1,000 messages. Against the plan's acceptance checks (targets measured on a slower machine): | check | target | result | |---|---|---| | messages per receive, 1 and 8 pollers | ≥ 9.5 | 10.0 / 9.6 | | client CPU per consumed message, 1 poller | ≤ 117 µs, goal ≤ 80 µs | 64 µs (`await message.ack()`), 45 µs (auto-ack) | | MiniStack CPU per consumed message, 1 poller | ≈ tuned | 135 µs (tuned 130, 0.1.0 defaults 335) | | `DeleteMessageBatch` calls per 1,000 messages | ≤ 120 | 100 | | publish CPU per message | within 10 % of tuned | single −14 %; batch +6 % (16 callers +12 %: the defaults JSON-encode `dict`s, tuned sends bytes) | | `consumer.stop()` with no running handlers | < 50 ms | 10 ms process teardown | <sub>Benchmark: the fantest SNS/SQS client benchmark (not part of this repo) against MiniStack 1.5.27, run without Docker on an OrbStack VM on Apple Silicon (MiniStack pinned to 2 vCPUs, client to 4, CPython 3.13, `--profile dist` builds). Every run measured 0.1.0 next to the build under test; medians over 3 passes × 3 rounds of 2,000 consumed messages. *defaults*: the build's defaults with handlers that `await message.ack()` (the harness's drivers); *auto-ack*: the same with returning handlers; *tuned*: the harness's hand-tuned configuration (pre-encoded bytes, no attributes, 50 slots per poller). Across runs, 0.1.0 tuned stayed within 85.7-89.4 µs per consumed message, so the runs compare.</sub> **Checked locally on this commit:** cargo fmt, clippy `-D warnings`, ruff, pyrefly, `scripts/check_consistency.py`, 147 Rust tests plus the 4 Rust e2e tests against MiniStack, 223 pytest tests (memory broker and MiniStack) on Python 3.12, 3.13, 3.14 and 3.15. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 13d0d36 commit 858303f

25 files changed

Lines changed: 803 additions & 208 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,8 @@ docstrings everywhere. README stays short and links to the docs.
147147

148148
* Calling into Python from Rust threads (events are pulled; see ADR 0004).
149149
* Threads or Tokio runtimes outside `crates/fanferro-py/src/runtime.rs`, or awaitables not created
150-
with `runtime::future_into_py`: they would outlive the `atexit` shutdown of the runtime.
150+
with `runtime::future_into_py` (or `runtime::Completions` for settlements): they would outlive the
151+
`atexit` shutdown of the runtime.
151152
* Holding a `parking_lot` lock across `.await`.
152153
* Long `wait_time` in tests; `time.sleep` in async code; global state in Python modules.
153154
* Logging message bodies (they may contain personal data) - log ids and types.

‎CHANGELOG.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,18 @@ Without any tuning, clients and consumers now perform like a hand-tuned 0.1 conf
2828
- **Slots are freed when settlement starts**: a handler that returns hands its slot to the next
2929
message while the delete is in flight; settlements in progress are bounded separately.
3030
- **Batched acks**: deletes and visibility changes wait up to `settle_linger` (5 ms) to fill a
31-
batch of ten, and are sent at once when every running handler awaits an explicit settlement.
31+
batch of ten, and are sent at once when every running handler awaits an explicit settlement
32+
in that batch.
33+
- **Automatic acks are not awaited per message**: the engine deletes the message in the
34+
background (`consumer.stop()` waits for it). `consumer.stats()["processed"]` and the
35+
`message.processed` event count handlers that succeeded; `stats()["engine"]["messages_acked"]`
36+
counts confirmed deletes, and a failed delete is reported as a `message.ack_failed` event. Explicit
37+
settlements (`await message.ack()`...) still resolve once SQS confirmed them.
3238
- `consumer.stop()` and `client.close()` no longer wait up to half a second for the dispatcher or
3339
the event pump, and events emitted just before `close()` are no longer lost.
40+
- Per-message work in the consumer and publisher is lighter: settlement awaitables are resolved
41+
in batches, event checks, type names and encoders are cached, and messages are no longer copied
42+
while being decoded.
3443

3544
Migrating from 0.1:
3645

‎crates/fanferro-core/src/consumer/config.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,8 +67,8 @@ impl Default for HeartbeatConfig {
6767
#[serde(default, deny_unknown_fields)]
6868
pub struct SettleConfig {
6969
/// Seconds a batch waits for more settlements (at most 1). A batch is sent as soon as it
70-
/// holds ten deletes or ten visibility changes, or when a settlement is urgent; 0 sends
71-
/// whatever is queued immediately.
70+
/// holds ten deletes or ten visibility changes, or when no other settlement can join it;
71+
/// 0 sends whatever is queued immediately.
7272
pub linger: f64,
7373
/// Retries for failed delete/visibility calls.
7474
pub retry: RetrySettings,

‎crates/fanferro-core/src/consumer/lease.rs‎

Lines changed: 79 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,18 @@
1010
1111
use std::collections::BTreeMap;
1212
use std::sync::Arc;
13-
use std::sync::atomic::{AtomicBool, Ordering};
13+
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
1414
use std::time::Duration;
1515

1616
use parking_lot::Mutex;
1717
use time::OffsetDateTime;
1818
use time::format_description::well_known::Rfc3339;
19-
use tokio::sync::{OwnedSemaphorePermit, oneshot};
19+
use tokio::sync::{OwnedSemaphorePermit, mpsc, oneshot};
2020
use tokio::time::Instant;
2121

2222
use super::EngineInner;
2323
use super::config::MAX_VISIBILITY;
24-
use super::settle::{SettleCommand, SettleKind};
24+
use super::settle::{Reply, SettleCommand, SettleKind};
2525
use crate::backend::{ReceivedMessage, SendRequest};
2626
use crate::error::{AwsErrorKind, Error, Result};
2727
use crate::events::{Event, EventKind};
@@ -92,8 +92,10 @@ pub struct Lease {
9292
/// Settling slot: held while the settlement is in progress.
9393
settling: Mutex<Option<OwnedSemaphorePermit>>,
9494
settled: AtomicBool,
95-
/// The next settlement command is urgent (see [`Lease::expedite`]).
96-
expedite: AtomicBool,
95+
/// The next settlement is awaited by a handler (see [`Lease::mark_awaited`]).
96+
awaited: AtomicBool,
97+
/// How many handlers wait, when all of them do (0: unknown; see [`Lease::mark_awaited`]).
98+
waiting_handlers: AtomicUsize,
9799
}
98100

99101
impl std::fmt::Debug for Lease {
@@ -129,7 +131,8 @@ impl Lease {
129131
permit: Mutex::new(Some(permit)),
130132
settling: Mutex::new(None),
131133
settled: AtomicBool::new(false),
132-
expedite: AtomicBool::new(false),
134+
awaited: AtomicBool::new(false),
135+
waiting_handlers: AtomicUsize::new(0),
133136
}
134137
}
135138

@@ -170,19 +173,35 @@ impl Lease {
170173
/// meanwhile), then hands the in-flight slot back so that receiving continues while the
171174
/// settlement completes. The lease stays tracked until [`Lease::finish`].
172175
async fn begin(&self) -> Result<()> {
176+
self.claim()?;
177+
self.start_settling().await;
178+
Ok(())
179+
}
180+
181+
/// Marks the lease as being settled; fails if a settlement was requested already.
182+
pub(crate) fn claim(&self) -> Result<()> {
173183
if self.settled.swap(true, Ordering::SeqCst) {
174184
return Err(Error::Lease(format!("message {} was already settled", self.message.message_id)));
175185
}
186+
Ok(())
187+
}
188+
189+
pub(crate) async fn start_settling(&self) {
176190
// The semaphore is never closed, so this only fails if the engine is gone.
177-
if let Ok(permit) = self.inner.settling.clone().acquire_owned().await {
191+
let permit = self.inner.settling.clone().acquire_owned().await.ok();
192+
self.enter_settling(permit);
193+
}
194+
195+
/// Takes the settling slot, then hands the in-flight slot back.
196+
fn enter_settling(&self, permit: Option<OwnedSemaphorePermit>) {
197+
if let Some(permit) = permit {
178198
*self.settling.lock() = Some(permit);
179199
self.inner.metrics.settling.add(1);
180200
}
181201
if let Some(entry) = self.inner.leases.lock().get_mut(&self.id) {
182202
entry.status = LeaseStatus::Settling;
183203
}
184204
self.permit.lock().take();
185-
Ok(())
186205
}
187206

188207
/// Returns the settling slot (if one is held).
@@ -229,19 +248,38 @@ impl Lease {
229248
}
230249

231250
async fn settle(&self, kind: SettleKind) -> Result<()> {
232-
let sender = self.inner.settle_sender().ok_or_else(|| Error::Shutdown("the consumer is closed".into()))?;
233251
let (reply, receiver) = oneshot::channel();
234-
let mut command = SettleCommand::new(self.message.receipt_handle.clone(), kind, Some(reply));
235-
command.urgent = self.expedite.swap(false, Ordering::SeqCst);
236-
sender.send(command).map_err(|_| Error::Shutdown("the consumer is closed".into()))?;
252+
self.queue(kind, Reply::Channel(reply));
237253
receiver.await.map_err(|_| Error::Shutdown("the consumer closed before settling".into()))?
238254
}
239255

240-
/// Sends this lease's next settlement (or explicit extension) right away instead of letting
241-
/// it wait to fill a batch. Use it when the caller is blocked on that settlement and no
242-
/// other settlement is expected soon (e.g. every handler awaits an explicit ack).
243-
pub fn expedite(&self) {
244-
self.expedite.store(true, Ordering::SeqCst);
256+
/// Hands a settlement command to the settle loop, which sends the outcome to `reply` (right
257+
/// away if the consumer is closed).
258+
pub(crate) fn queue(&self, kind: SettleKind, reply: Reply) {
259+
let Some(sender) = self.inner.settle_sender() else {
260+
reply.send(Err(Error::Shutdown("the consumer is closed".into())));
261+
return;
262+
};
263+
let mut command = SettleCommand::new(self.message.receipt_handle.clone(), kind, Some(reply));
264+
command.awaited = self.awaited.swap(false, Ordering::SeqCst);
265+
command.waiting_handlers = Some(self.waiting_handlers.swap(0, Ordering::SeqCst)).filter(|&n| n > 0);
266+
if let Err(mpsc::error::SendError(command)) = sender.send(command)
267+
&& let Some(reply) = command.reply
268+
{
269+
reply.send(Err(Error::Shutdown("the consumer is closed".into())));
270+
}
271+
}
272+
273+
/// Notes that a handler waits for this lease's next settlement (or explicit extension), so
274+
/// that the batch it joins is not held back for settlements that cannot come.
275+
///
276+
/// `waiting_handlers`: when every handler that could request another settlement waits for
277+
/// one, how many handlers wait. The settle loop sends the batch as soon as it holds that many
278+
/// awaited settlements; while fewer are pending, some handlers wait for settlements that were
279+
/// sent already, and their slots will soon go to handlers that settle more.
280+
pub fn mark_awaited(&self, waiting_handlers: Option<usize>) {
281+
self.awaited.store(true, Ordering::SeqCst);
282+
self.waiting_handlers.store(waiting_handlers.unwrap_or(0), Ordering::SeqCst);
245283
}
246284

247285
fn cap_visibility(&self, timeout: Duration) -> Duration {
@@ -252,12 +290,35 @@ impl Lease {
252290

253291
/// Deletes the message (successful processing).
254292
pub async fn ack(&self) -> Result<()> {
255-
self.begin().await?;
293+
self.claim()?;
294+
self.ack_claimed().await
295+
}
296+
297+
/// [`Lease::ack`] after [`Lease::claim`] succeeded.
298+
pub(crate) async fn ack_claimed(&self) -> Result<()> {
299+
self.start_settling().await;
256300
let result = self.settle(SettleKind::Delete).await;
257301
self.finish(Outcome::Acked, &result);
258302
result
259303
}
260304

305+
/// The fast path of an ack after [`Lease::claim`]: when a settling slot is free (the usual
306+
/// case), takes it and queues the delete with `reply` right away, so acks requested one after
307+
/// another reach the settle loop in that order. Hands `reply` back when no slot is free.
308+
pub(crate) fn try_queue_ack(&self, reply: Reply) -> std::result::Result<(), Reply> {
309+
let Ok(permit) = self.inner.settling.clone().try_acquire_owned() else {
310+
return Err(reply);
311+
};
312+
self.enter_settling(Some(permit));
313+
self.queue(SettleKind::Delete, reply);
314+
Ok(())
315+
}
316+
317+
/// Records the outcome of an ack whose delete was queued with [`Lease::queue`].
318+
pub(crate) fn finish_ack(&self, result: &Result<()>) {
319+
self.finish(Outcome::Acked, result);
320+
}
321+
261322
/// Makes the message visible again after `delay` (retry with backoff).
262323
pub async fn nack(&self, delay: Duration) -> Result<()> {
263324
self.begin().await?;

‎crates/fanferro-core/src/consumer/mod.rs‎

Lines changed: 80 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,9 @@ mod lease;
2727
mod poller;
2828
mod settle;
2929

30+
use std::borrow::Cow;
3031
use std::collections::{BTreeMap, HashMap, VecDeque};
32+
use std::future::Future;
3133
use std::sync::Arc;
3234
use std::sync::atomic::{AtomicU64, Ordering};
3335
use std::time::Duration;
@@ -55,7 +57,7 @@ use crate::types::attributes::{md5_of_attributes, md5_of_body};
5557
use crate::types::reserved;
5658
use config::Settings;
5759
use lease::LeaseEntry;
58-
use settle::{SettleCommand, SettleKind};
60+
use settle::{Reply, SettleCommand, SettleKind};
5961

6062
/// A received message handed to the application.
6163
#[derive(Debug)]
@@ -75,6 +77,73 @@ impl Delivery {
7577
pub fn message(&self) -> &ReceivedMessage {
7678
self.lease.message()
7779
}
80+
81+
/// Acks the message without waiting for the delete. The settlement runs on the engine's
82+
/// runtime and is tracked ([`ConsumerEngine::close`] waits for it); its outcome is reported
83+
/// through metrics (`messages_acked`, `ack_failures`) and events (`message.acked`,
84+
/// `message.ack_failed`).
85+
///
86+
/// # Errors
87+
///
88+
/// [`Error::Lease`] if the message was settled (or its settlement started) already.
89+
pub fn ack_in_background(self: &Arc<Self>) -> Result<()> {
90+
self.lease.claim()?;
91+
// Failures are reported by the lease (metrics and events).
92+
self.continue_ack(|_| {});
93+
Ok(())
94+
}
95+
96+
/// Acks the message and hands the outcome to `done`. The delete is queued before this
97+
/// returns (when a settling slot is free), so acks requested one after another reach the
98+
/// settle loop in that order; `done` runs once the delete completed.
99+
pub fn spawn_ack<Done>(self: &Arc<Self>, done: Done)
100+
where
101+
Done: FnOnce(Result<()>) + Send + 'static,
102+
{
103+
match self.lease.claim() {
104+
Ok(()) => self.continue_ack(done),
105+
Err(error) => done(Err(error)),
106+
}
107+
}
108+
109+
/// An ack after its lease was claimed. Usually the delete is queued right away and finished by
110+
/// the settle loop (no task per ack); without a free settling slot, a tracked task waits for
111+
/// one.
112+
fn continue_ack<Done>(self: &Arc<Self>, done: Done)
113+
where
114+
Done: FnOnce(Result<()>) + Send + 'static,
115+
{
116+
let delivery = self.clone();
117+
let finish = Reply::Callback(Box::new(move |result: Result<()>| {
118+
delivery.lease.finish_ack(&result);
119+
done(result);
120+
}));
121+
if let Err(finish) = self.lease.try_queue_ack(finish) {
122+
let delivery = self.clone();
123+
let inner = &self.lease.inner;
124+
inner.settlements.spawn_on(
125+
async move {
126+
delivery.lease.start_settling().await;
127+
delivery.lease.queue(SettleKind::Delete, finish);
128+
},
129+
&inner.runtime,
130+
);
131+
}
132+
}
133+
134+
/// Runs `settle` (an ack, retry, release... of this delivery) on the engine's runtime and
135+
/// hands its outcome to `done`. Like background acks, the settlement is tracked:
136+
/// [`ConsumerEngine::close`] waits for it, and it completes even if nobody waits for `done`.
137+
pub fn spawn_settlement<Settle, Fut, Done>(self: &Arc<Self>, settle: Settle, done: Done)
138+
where
139+
Settle: FnOnce(Arc<Self>) -> Fut + Send + 'static,
140+
Fut: Future<Output = Result<()>> + Send + 'static,
141+
Done: FnOnce(Result<()>) + Send + 'static,
142+
{
143+
let delivery = self.clone();
144+
let inner = &self.lease.inner;
145+
inner.settlements.spawn_on(async move { done(settle(delivery).await) }, &inner.runtime);
146+
}
78147
}
79148

80149
/// Lifecycle state of the engine.
@@ -122,6 +191,8 @@ pub(crate) struct EngineInner {
122191
pollers: Mutex<Vec<JoinHandle<()>>>,
123192
background: Mutex<Vec<JoinHandle<()>>>,
124193
decoders: TaskTracker,
194+
/// Settlements running in the background (see [`Delivery::ack_in_background`]).
195+
settlements: TaskTracker,
125196
rate_limiter: Option<RateLimiter>,
126197
runtime: tokio::runtime::Handle,
127198
/// Until when an aborted long poll may still be served by the queue (see
@@ -231,7 +302,7 @@ impl EngineInner {
231302

232303
async fn decode(self: &Arc<Self>, lease: Lease) -> Option<Delivery> {
233304
let received_at = OffsetDateTime::now_utc();
234-
let message = lease.message().clone();
305+
let message = lease.message.clone();
235306
if self.settings.verify_md5 {
236307
let body_ok =
237308
message.md5_of_body.as_ref().is_none_or(|md5| md5.eq_ignore_ascii_case(&md5_of_body(&message.body)));
@@ -256,9 +327,9 @@ impl EngineInner {
256327
self.handle_sns_control(lease, unwrapped.info).await;
257328
return None;
258329
}
259-
(unwrapped.message, unwrapped.attributes, Some(unwrapped.info))
330+
(Cow::Owned(unwrapped.message), unwrapped.attributes, Some(unwrapped.info))
260331
}
261-
Ok(None) => (message.body.clone(), message.attributes.clone(), None),
332+
Ok(None) => (Cow::Borrowed(message.body.as_str()), message.attributes.clone(), None),
262333
Err(error) => {
263334
self.metrics.decode_failures.inc();
264335
return Some(Delivery { lease, sns: None, decoded: Err(error), received_at });
@@ -418,6 +489,7 @@ impl ConsumerEngine {
418489
pollers: Mutex::new(Vec::new()),
419490
background: Mutex::new(Vec::new()),
420491
decoders: TaskTracker::new(),
492+
settlements: TaskTracker::new(),
421493
rate_limiter,
422494
runtime,
423495
orphaned_receives_until: Mutex::new(None),
@@ -576,6 +648,9 @@ impl ConsumerEngine {
576648
/// and after in-flight handlers finished or were cancelled.
577649
pub async fn close(&self, release_unsettled: bool) {
578650
self.stop_receiving(false).await;
651+
// Background acks complete first: they need the settle loop, which stops below.
652+
self.inner.settlements.close();
653+
self.inner.settlements.wait().await;
579654
for delivery in self.drain_buffered() {
580655
if release_unsettled {
581656
let _ = delivery.lease.release().await;
@@ -591,7 +666,7 @@ impl ConsumerEngine {
591666
let mut command = SettleCommand::new(
592667
entry.receipt_handle,
593668
SettleKind::Visibility(self.inner.visibility_floor(Duration::ZERO)),
594-
Some(reply),
669+
Some(Reply::Channel(reply)),
595670
);
596671
command.urgent = true;
597672
if sender.send(command).is_ok() {

0 commit comments

Comments
 (0)