chainsource: escalate unrecoverable block epoch loss to the daemon by Roasbeef · Pull Request #762 · lightninglabs/wavelength

Roasbeef · 2026-06-18T21:22:56Z

The problem

The wallet daemon uses block notifications to advance chain-dependent work. If its backend repeatedly closes the notification stream or rejects reconnect attempts, the daemon currently retries forever. Its RPC listener can stay available while block consumers make no progress. Restarting the process can recover by constructing a fresh backend connection.

The fix

Track continuous failure at both reconnect boundaries: the shared LND upstream reader and each chainsource subscription. The shared reader normally keeps subscriber queues open while it reconnects, so its own failure budget is essential to detect a sustained LND outage.

Only a delivered block with a changed height or hash clears an outage. Repeated registration, or replaying the same cached tip before closing again, does not count as recovery. Reorg replacements still count as progress and all epochs continue to be forwarded. The failure budget pauses while a stream is open. A long Bitcoin block interval therefore cannot turn a later transient stream closure into a fatal error.

After a five-minute outage budget plus up to one minute of jitter, the daemon's fatal callback cancels its run context. Escalation is checked between reconnect attempts. The normal shutdown path drains services and returns the fatal cause, causing the command to exit nonzero so a supervisor can restart it. The fatal marker preserves that outcome even when the backend error wraps context.Canceled or cancellation interrupts startup.

flowchart LR
    A[Open stream; budget paused] -->|Closes| B[Retry; budget runs]
    B -->|Reopens| A
    A -->|New block identity| C[Clear spent retry time]
    C --> A
    B -->|Budget expires with fatal hook| D[Cancel daemon and fail health]
    D --> E[Drain services and exit nonzero]

Safety rule

A daemon must not remain alive forever against continuously closed or failing block streams. Transient reconnection still recovers in process. Callers without a fatal callback keep their existing indefinite retry contract. Normal cancellation wins over a later fatal report.

Health and scope

The daemon’s existing separate, opt-in health listener serves /livez and /readyz. Both immediately fail when the run context is cancelled. Fatal escalation uses that same context, so health withdrawal follows the shutdown decision directly.

Open but silent streams remain a separate follow-up. No Bitcoin inter-block idle deadline is introduced: a long interval without a block does not prove backend failure. Initial registration errors still return directly to the subscriber.

Tests

Repeated immediately closed streams, including streams that replay the same tip, exhaust the budget and escalate exactly once.
Repeated registration errors exhaust the same budget and preserve the backend cause.
The real notifier adapter’s shared reader escalates raw upstream failures while its subscriber queues remain open during reconnects. Hookless notifiers keep those queues alive and resume delivery.
Hookless subscriptions continue retrying past the fatal budget.
A changed block identity resets the outage budget; duplicate tip replays do not. Same-height reorg replacements count as recovery. Open time is paused even when only a duplicate tip was delivered; a later transient close does not turn a long Bitcoin block interval into a fatal error.
Cancellation while reconnecting exits without fatal escalation.
A real in-process daemon serves gRPC, then drains on fatal or ordinary cancellation. Fatal errors, including wrapped cancellation, return as errors; ordinary cancellation followed by a fatal report returns cleanly. Both health endpoints withdraw immediately.

Local validation:

Focused daemon/chainsource/health tests: passed with dev nolog build tags.
Targeted race tests for block subscriptions, daemon shutdown, and health: passed.
make unit pkg=chainsource log="stdlog trace" timeout=5m: passed.
Real-LND TestLNDBlockFanoutReorg: passed with SQLite and PostgreSQL.
Module tidiness: no changes. Formatting and changed-code lint are checked before publication.
Shared-hub epoch recovery tests and notifier adapter unit tests cover the current LND path.
Go documentation audit: no new missing comments relative to the base.

Published head: 6a86b0d, rebased onto main at ccc97b2.

Rebase validation passed: focused race tests for chainsource, the LND hub, daemon exit/health/cancellation, the upstream authorization expiry test, and cancellation cleanup; changed-code lint including actorblock; formatting, module tidiness, and commit-message checks. Gateway reviewed this exact head: no findings, then approved. Fresh full CI, including the race suite, and Mobile Bindings passed. Final check summary: 25 passed, 7 skipped, no failures. All three published commits are GitHub Verified. GitHub still requires a qualifying maintainer approval; the Gateway App approval does not satisfy that gate.

The runtime patches are unchanged by the rebase. The expiry-test scheduling fix is now supplied by main. The remaining test-only repair isolates the process-wide goroutine-count assertion from parallel sibling HTTP workers, preserving its threshold, map cleanup assertions, and repeated-cancellation checks.

Compatibility and rollout

No RPC, wire-format, or database migration changes. Hookless chainsource and notifier consumers retain indefinite retries. The daemon opts into escalation, including when hosted through RunWithContext. Embedded daemon hosts must handle its terminal error, or the SDK’s Client.Wait() result, and create a fresh runtime when recovery is required. Making this policy separately configurable for embedded daemons is a bounded follow-up. A deployment of the merged source and supervision configured to restart failed processes are required to activate recovery. Production deployment is outside this PR.

gemini-code-assist

Code Review

This pull request introduces a fatal timeout mechanism for block epoch subscriptions to prevent the daemon from silently spinning when a backend connection is stuck. If a subscription remains down past the fatal timeout, the actor escalates the failure via an OnFatal callback, which triggers a non-zero exit of the daemon so it can be restarted. A potential issue was identified where a normal shutdown or unsubscribe could trigger a spurious fatal escalation if the subscription had already exceeded the timeout; checking for context cancellation before escalating resolves this.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b19d79ce9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Roasbeef · 2026-06-23T21:21:50Z

@claude review this

claude · 2026-06-23T21:22:07Z

Claude finished @Roasbeef's task in 2m 44s —— View job

Review: chainsource: escalate unrecoverable block epoch loss to the daemon

I read the full diff across block_epoch_actor.go, chainsource.go, server.go, and the gateway/health changes. This is a well-motivated, carefully-built fix — the down-streak tracking (stamp on first loss, clear only on a delivered block) correctly defeats the "storming backend hands back streams that immediately close" false-healthy case, the once-sampled jitter is the right call for fleet de-correlation, and bubbling the failure through context.CancelCause instead of os.Exit keeps the shutdown path clean. The health/readiness backstop is a sensible companion. Comments are excellent.

Two correctness issues stand out, one of which I'd treat as blocking.

🔴 1. Without an `OnFatal` hook, the subscription is silently killed instead of retried (agree with Codex)

In monitorBlocks (chainsource/block_epoch_actor.go:430-438):

if a.reconnectExhausted(downSince, fatalTimeout) {
    a.escalateFatal(log, downSince, lastErr)
    return   // <-- exits the monitor unconditionally
}

escalateFatal is a no-op when OnFatal is None, but the return fires regardless. So for any caller that doesn't install the hook, a >5m backend outage now permanently closes the block subscription (and the iterator channel) rather than retrying forever as before #698. This is a behavioral regression, and it's reachable today:

systest/systest.go:209 builds ChainSourceConfig with no OnFatal — confirmed.
SDK/embedded and external BlockEpochConfig consumers, since the hook is documented as optional.

The daemon path is fine (server.go:1925 wires fn.Some(s.signalFatal)), but the actor shouldn't degrade into a silent dead-subscription for everyone else. Only take the fatal return when a hook is actually installed; otherwise keep retrying (preserving the old unbounded-retry contract):

if a.reconnectExhausted(downSince, fatalTimeout) &&
    a.cfg.OnFatal.IsSome() {

    a.escalateFatal(log, downSince, lastErr)
    return
}

Either that, or make OnFatal required for bounded retries and document it. Fix this →

🟠 2. Spurious fatal escalation during shutdown (agree with Gemini)

At block_epoch_actor.go:430, reconnectExhausted is evaluated before any check of a.ctx.Err() (that check only lives inside waitForReconnect, one step later). If the actor is stopped (Stop/OnStop → a.cancel()) while the subscription is already down past the budget, the loop escalates on the way out.

Worth flagging that this is slightly worse than Gemini described: signalFatal unconditionally does s.fatalFlag.Store(true) (server.go:1254) before the fatalCancel call. Even though fatalCancel(err) is a no-op once the run ctx is already cancelled (first cause wins), the latched fatalFlag flips LivenessCheck/ReadinessCheck to failing — so a clean shutdown that happened to coincide with a long outage reports a fatal failure it didn't really have. Cheap guard:

if registration == nil {
    if a.ctx.Err() != nil {
        return
    }
    if a.reconnectExhausted(downSince, fatalTimeout) ... {

Fix this →

🟡 3. A stream that opens but never delivers (and never closes) won't escalate

reconnectExhausted is only consulted while registration == nil (the reconnect loop). If a reconnect succeeds but the new stream then stays open and silent forever, the goroutine blocks in the select on registration.Epochs (block_epoch_actor.go:464). downSince (stamped at the prior close) is never cleared — no block was delivered — but it's also never re-checked, so the fatal budget never fires. This is narrower than the symptom you fixed (closing streams), and a healthy backend delivers blocks well within the budget, so it may be acceptable — but it's a gap in the "continuously down" guarantee worth a sentence in the comment, or a bounded wait on the open-but-idle stream if you want full coverage.

Nits / observations (non-blocking)

server.go:1253 signalFatal concurrency: fatalCancel is written once in run() before any subsystem starts and read from the actor goroutine afterwards, so the happens-before holds and the no-mutex read is fine. The doc comment ("safe to call from any subsystem goroutine") is accurate. Just confirming it isn't a data race.
fatalReconnectTimeout jitter on the explicit-timeout path: the design (auto-jitter only on the default path, exact on an explicit timeout) is reasonable and well-documented, and production uses the default path (the daemon never sets FatalReconnectTimeout), so production gets jitter. 👍
Gateway health routes: in-process-only checks, %q-escaped reason, 503 on failure, warn-and-skip when checks are unavailable — all good. The unauthenticated-but-no-backend-I/O reasoning is sound.
Test coverage: TestBlockEpochActorEscalatesWhenBackendStaysDown and the jitter test are clean and clock-driven. Consider adding a case asserting the no-OnFatal behavior once lib: start adding lib helpers #1 is resolved (subscription keeps retrying rather than exiting), and one for the shutdown-during-down race in multI: add initial repo scaffolding #2.

Overall: solid, the architecture is right. I'd block on #1 (silent subscription death for hookless callers) and fold in #2 while you're there; #3 is a judgment call.

Read the PR diff and changed files
Analyze block_epoch_actor.go fatal reconnect logic
Analyze config wiring (chainsource.go, server.go, gateway)
Review tests
Evaluate prior reviewer feedback (Gemini, Codex)
Post review
· fail-fast-block-epoch-loss

Roasbeef · 2026-06-23T21:29:53Z

Thanks — addressed the two correctness issues, folded into the existing commits (force-pushed 54f70758).

#1 (🔴 silent subscription death without OnFatal) — fixed. The fatal return is now gated on a.cfg.OnFatal.IsSome(): only callers that opted into bounded escalation take the exit path; hookless callers (systest, SDK/embedded) keep the original unbounded-retry contract instead of having the subscription silently killed after the budget. New test TestBlockEpochActorRetriesWithoutOnFatal asserts a hookless actor against a permanently-dead backend keeps reconnecting (attempt count climbs well past where an escalating actor would have stopped) rather than exiting.

#2 (🟠 spurious escalation on shutdown) — fixed. Added an a.ctx.Err() != nil early-return at the top of the registration == nil block, before reconnectExhausted, so a stop that coincides with a long outage exits cleanly and never latches fatalFlag (which would otherwise flip liveness/readiness to failing). I didn't add a dedicated unit test for this one: the trigger is an inherent goroutine-timing boundary (cancellation observed precisely between the loop top and the exhaustion check), so any test would be racy/flaky; the guard itself is a cheap, obviously-correct ordering check.

#3 (🟡 open-but-silent stream) — documented, not implemented. Agreed it's narrower than the closing-stream symptom this PR targets, and downSince is only ever stamped on a close, so a healthy backend (which delivers blocks well within the budget) is unaffected. Added a comment on the downSince/budget block noting that an open stream that goes silent without closing is out of scope. Happy to add a bounded idle-wait on the open stream in a follow-up if you'd prefer full coverage.

Nits confirmed (no change needed): the fatalCancel happens-before is sound, and the jitter-on-default-path design is intentional. go test -race ./chainsource/... ./darepod/... green; make lint-changed-local and commitmsg-lint clean.

litbot-9000 · 2026-08-12T03:01:20Z