Tracking: commit, checkpoint and WAL recovery fixes found by crash and stress testing #1376

This work was produced with the help of language models.

This collects the fixes for commit, checkpoint and WAL recovery bugs, so they can be reviewed as a set. Each is its own PR against main with its own tests; the open ones stay drafts, since one may change without notice to stay compatible with the others. This is a broad experiment aimed at improving durability under pressure. Backward compatibility is out of scope until a working solution is found.

Process. Bugs come from a workload simulator and the stress loops below. Each gets an issue with a deterministic repro, checked on main and the last release, and a fix PR whose tests fail on main. Because one bug can hide another (a segfault after a refused BEGIN ends a workload before a lost commit shows), the open PRs are also merged onto one main commit and stressed against main alone, with the same seeds.

Merged from the same testing: #1195, #1333, #1338, #1355, and #1389, which fixes #1386, a segfault after a write or checkpoint is refused at max_db_size.

Merge order. Each PR is a fix of its own, but some depend on others or change the same lines:

• #1400 conflicts with main in WAL::logCommittedWAL, where #1380 landed; it needs main merged in.
• #1399, #1400, #1408, #1436 and #1437 at any point; each merges cleanly into current main. The last two both add tests at the end of mvcc_retention_test.cpp, so the second needs a rebase.

A safe sequence for the open PRs: #1399, #1400, #1408, #1436, #1437.

Compatibility. Every fix except #1377's keeps the WAL and data-file formats. #1378 changes only the content of the pinned-checkpoint marker file, which older builds check just for existence. A marker that is not exactly one of the two known bytes now counts as missing, so a 0.22 database that crashed while writing its marker is refused; opening it once with 0.22 recovers it. #1377's fix adds an ABORT WAL record that older builds cannot read; it lasts only until the next checkpoint retires the WAL.

Stress loop. A C program on the C API: writer threads add relationships tagged with unique ids, with a checkpoint after every commit, and rounds add a full 64 MB buffer pool, COPY, several writers, a write transaction held past the 5 s checkpoint wait, or a file-size limit that fails checkpoint writes. Each round ends with SIGKILL or a clean close; a fresh process then checks that every committed write is there exactly once.

Run 1, main 7bcc47d:

Run 2, #1378 on main 7bcc47d + #1358 + #1361 + #1375, 300 rounds with the same seeds: 2,884 committed writes lost in 42 rounds without it, none with it.

Run 3, #1380 on main 7bcc47d + #1358 + #1361 + #1375 + #1378, 300 rounds with the same seeds: rounds where a write reported as failed was there after reopening went from 4 to none, and no committed write was lost.

Run 4, all six PRs at their current heads merged on main 7bcc47d: 300 stress rounds without a violation, and the workload simulator (sessions, heavy, default and races, 40 workloads of 60 steps each) passed with the rules for these bugs enabled.

Run 5, the workload simulator with a rule for each of these bugs: a kill during the checkpoint after a pinned one, a file-size limit (a full disk), a max_db_size cap, a COMMIT on a full buffer pool followed by a crash, a reopen with the other checksum setting, and a named graph. Each rule fails on a build without its fix: #1316, #1357, #1199, #1396 and #1397 on main, #1304 without #1378, the named-graph case without #1379, #1386 without #1389. The exception is the full-pool COMMIT, which never failed in the simulator, so #1380 rests on its own tests and Run 3. With all eight PRs and #1389 at their current heads merged on main 4ca2237, the five simulator suites (sessions, heavy, default, races, recovery; 40 workloads of 60 steps) passed with every rule enabled, apart from #1406, an older bug (also in 0.21.2) that the recovery suite found. With its fix, #1408, merged too, the recovery suite passed with that rule enabled as well.

Run 6, #1436 and #1437, with a second loop, a Python script on the driver's C-API backend: one writer and up to three read-only transactions on a node table of 3,000 to 140,000 rows. Each round runs an update, insert, delete, write transaction, CHECKPOINT or reopen, or opens or closes a reader, and every read is checked against the snapshot its transaction began with. With the nine earlier PRs, #1389 and both fixes merged on main 4ca2237 and runtime checks on, 200 seeds of 300 rounds passed. A seed fails when the writer reads an update on another row (#1303) or a reader sees a row deleted after it began (#1305); the table counts each seed's first failure. On the same first 40 seeds, the control builds failed as follows:

Readers close newest-first, and a CHECKPOINT runs each time a reader opens with none open; that keeps the loop clear of #1438, which neither fix covers. On the same merge, 300 rounds of the C stress loop above passed. The simulator does not reach #1303 or #1305: the Run 5 merge with #1408 passes its sessions and races suites with their rules on. So here it only checked for regressions, and every failure it found also happens on main or on that merge.

Run 7, the recovery suite on the Run 6 merge with the review follow-ups of #1378, #1379 and #1380, on main 4ca2237. Its first run found a hole in #1378: after a crash between a checkpoint's marker removal and its record, recovery kept the earlier CHECKPOINT record without a marker, so a second crash refused to open. It also found a bug in the simulator's own storage-limit rule. With both fixed, 40 workloads of 60 steps passed with every rule enabled.

#1304: #1378 and #1379.

• A transaction committed after a pinned CHECKPOINT record proves that checkpoint completed. Recovery now skips through the last such record and discards any leftover shadow.
• The pinned-checkpoint marker now means "every CHECKPOINT record in the active WAL is applied". That covers a crash during the next checkpoint and both crash points from the #1304 comment.
• When only a later commit proves a CHECKPOINT applied, recovery writes the marker before it removes the shadow, so a second crash still reopens (Run 7).
• fix(storage): reopen named graphs after a crash around a pinned checkpoint #1379, on top of fix(storage): recover pinned checkpoints that a crash interrupts #1378, covers named graphs, which still failed to reopen (graph shadow file db.g.shadow is missing.). Under the marker they load without their shadows. A pinned checkpoint now also retires graph WALs, since a named graph's commits go to the main WAL. Older builds kept those graph WALs, so under an older build's marker graphs load that way only when no named graph has a WAL of its own.
• Implement WAL based replication #1366's standby replica would rely on the same pinned-WAL state.

#1377: approach. When applying a transaction fails after its COMMIT record is durable (a full buffer pool, or a named graph dropped by a concurrent DROP GRAPH):

• the fix appends and syncs an ABORT record naming that transaction's block in the WAL;
• recovery skips aborted blocks, so a failed COMMIT is not applied once its ABORT is durable, and the database stays writable;
• recovery ignores an ABORT that names no transaction committed after the last CHECKPOINT;
• only if the ABORT record itself cannot be written does the WAL refuse writes until restart.

#1377: rejected alternatives.

• Refuse writes until restart and report the transaction as committed, so the next reopen applies it once. Simpler and needs no new WAL record, but:
  • a transient full buffer pool would stop every writer until a restart;
  • the restart would apply a transaction the client saw fail, and readers would see the rolled-back state until then;
  • it breaks the existing DROP GRAPH race test, whose failed commit recovery already skips.
• Refuse writes only when recovery would apply the failed transaction: not safe, since one transaction can write to both a dropped graph and main.
• Reserve the memory before writing the COMMIT record: not possible; the amount is only known while applying.
• Let the buffer pool overcommit while a commit is applied: removes the out-of-memory cause, but changes pool policy and doesn't cover the DROP GRAPH race. Left as a possible follow-up.

Candidates, not fixed yet:

• Bug: after a CHECKPOINT, a read-only transaction sees an update committed after it began #1438: after a CHECKPOINT, a read-only transaction sees an update committed after it began; in the retention code fix(storage): keep updates on their rows when a CHECKPOINT runs with a reader open #1436 and fix(storage): keep rows created and deleted after a reader began hidden after CHECKPOINT #1437 change, also in 0.22.0, and 0.21.2 shows it too (Bug: a read-only transaction loses its snapshot when another connection runs CHECKPOINT #1184).
• Bug: dropping the built-in primary-key index is not rolled back and is lost after a crash #1317: dropping the built-in primary-key index is not rolled back and is lost after a crash.
• Bug: Rows replayed from the WAL after a crash are missing from the FTS index and cannot be deleted #1241: rows replayed from the WAL after a crash are missing from the FTS index and cannot be deleted.
• Bug: a partitioned table with a primary key segfaults on a later checkpoint #1407: a partitioned table with a primary key segfaults on a later checkpoint, and after a failed checkpoint accepts duplicate keys; partition children are never finalized after a checkpoint.

Next: narrow lenient WAL replay to a torn tail, as promised on ladybug-python#77, which makes lenient replay Python's default. Today it stops at the first record it cannot read for any reason, so intact commits after a corrupted record or a failed read are dropped too. The fix will open as a draft under this issue.

To do once these PRs settle:

• Update the user docs' transactions page (ladybug-docs cypher/transaction.md):
  • its Checkpoint section still says a checkpoint needs no active transactions and that CHECKPOINT fails otherwise;
  • it should say that a failed automatic checkpoint no longer fails the commit;
  • it should say what happens when the checkpoint COPY requires fails.
• Release notes:
  • fix(txn): don't fail a committed write when its automatic checkpoint fails #1361: a failed automatic checkpoint no longer fails the commit; it prints a warning on stderr, and a later commit retries it.
  • fix(wal): throw on a checksum-setting mismatch instead of dropping the WAL #1358: opening a database with the other enableChecksums setting is refused and keeps the WAL, where it used to delete it.
  • fix(txn): log an ABORT record when a committed transaction cannot be applied #1380: the ABORT record lasts until the next checkpoint, and older builds cannot read it, so checkpoint on the new build before downgrading.
  • fix(storage): recover pinned checkpoints that a crash interrupts #1378: a 0.22 database that crashed while writing its pinned-checkpoint marker is refused until it is opened once with 0.22.