LMDB reader TLS cleanup can segfault during Engine test teardown #194

Problem

MBV’s unit-test job for PR #1865 intermittently aborts with SIGSEGV in post_delegation_aml_rejects_high_risk_signer. AML rejects the signer as expected; the crash follows the log sequencer is draining the in-flight work.

Linux reproduction captured the same native stack on both Engine 0.20.0 (bc6b5a96876d97760b4e42d7119357b10ae1cd5b) and 0.19.0 (6250465a26039c5fb9f80528960dbdeca0883a7a), so this predates the 0.20.0 update:

Thread "post_delegation":
#0  stop_on_segv                    # diagnostic handler installed after initial repro
#1  <signal handler called>
#2  mdb_env_reader_dest
#3  __nptl_deallocate_tsd
#4  start_thread
#5  clone3

The captured fault is the write clearing reader->mr_pid in LMDB’s TLS destructor:

mdb_env_reader_dest+45: movl $0x0,0x8(%rax)

This points to LMDB environment closure racing thread-exit cleanup of a reader slot, rather than the AML assertion or account commit-sequence change. Running under GDB from startup masked the failure across 1,200 attempts; a SIGSEGV handler that stopped the process after the fault allowed attaching and collecting the stacks.

Reproduction

On Linux, check out MBV 19774717cd7ffe806c08cc6faa36b48d29a4b98f (branch chore/engine-0.19.0-redsuite-stack) and build:

cargo test --locked -p magicblock-chainlink --test 10_aml_undelegation --no-run

Set test_bin to the executable printed by that build, then repeat the test in eight concurrent processes:

for worker in {1..8}; do
  (
    for attempt in {1..500}; do
      taskset -c 0-3 "$test_bin" \
        --exact post_delegation_aml_rejects_high_risk_signer --nocapture \
        > "/tmp/aml-undelegation-$worker.log" 2>&1 || break
    done
  ) &
done
wait

The initial 100 sequential Linux attempts passed. The first concurrent batch reproduced SIGSEGV after 84 attempts in one worker; the other seven completed 100 each. Repeating against MBV base e18a212563b1da58d929666ca9592f97f1d39bfb with Engine 0.19.0 also reproduced the crash, including the matching TLS-destructor stack.

Each test uses its own temporary database directories. Parallel processes add scheduling pressure; they do not share one LMDB environment or database directory.

Expected

Resolve the teardown/lifetime race while retaining TLS-backed read transactions. Switching to non-TLS transactions is not an acceptable fix.

Context

Ubuntu 24.04, x86-64, Rust 1.96.1; reproduction processes restricted to four CPUs. Dependencies: heed 0.22.1 and lmdb-master-sys 0.2.6.

Relevant inspected code at Engine 0.20.0:

• TLS transaction type and LMDB environment setup.
• ShutdownManager drop cancels services without awaiting them.
• TestEngine lifecycle and explicit close.

Observed under concurrent test execution; concurrency may be an important trigger. Production validators are not run concurrently on the same node, so this test workload may overstate production exposure. There is no production reproduction or evidence that every graceful shutdown is affected; production likelihood remains unestablished.