Harden transfer contracts and isolate cold-cache failures

[loverustfs]

Change

A permanently denied cold write previously returned to the queue head on every drain and prevented healthy jobs from publishing. The sink now processes a bounded snapshot fairly, retries transient errors up to three total attempts, and isolates terminal failures for individual retrieval. drain_report exposes partial progress; drain returns the first error after processing the round. Pending and failed jobs share the same byte and job admission limits, with a hard ceiling of 4,096 retained jobs.

Cold publication and refill also enforce the following contracts:

• Object and manifest keys contain their respective content digests. Admission verifies the keys, complete object records, manifest ranges, identities and checksums. Interleaved retries in one namespace cannot overwrite another pack.
• A PUT error never triggers speculative object deletion: the manifest may already have committed before its acknowledgement was lost. Unreferenced objects require later explicit, reference-safe reclamation.
• Manifest v3 binds each range to the complete validated canonical CacheIdentity. All four public refill APIs require the expected identity from the upstream authenticated request. Tenant and identity mismatches return no payload and cause no payload-range download.
• Legacy v1/v2 manifests are clean misses and require recompute and repack. Malformed v3 data remains an error; changing a version header does not migrate an entry.
• Fault-domain isolation checks the epoch budget for the entire removal batch before changing membership, credits or pending fill notices.

The transfer-contract work remains in this PR: registered-slot reuse and atomic batch admission, overflow-safe fencing, bounded payload-free fill notices, explicit CPU TCP fallback receipts, and honest Mooncake comparison labels. Mooncake and libnixl are not linked. See ADR 0009, the data paths, and the test matrix.

Validation

Exact tested commit: 02178a55617391b273ff17021457596ee6b91e30, with a clean working tree.

• python3 scripts/ci/vm_regression.py --profile cpu --offline --output artifacts/pr8-regression-02178a5: 18/18 gates PASS locally.
• Default workspace: 218 Rust tests PASS; regression runner: 30 Python tests PASS.
• Formatting, default/all-feature Clippy with warnings denied, doc tests, optional/combined feature tests and all CPU demos passed through that runner.
• New regression cases cover permanent-error isolation, fair bounded retries, independent failed-job byte/job limits, cancellation, partial progress, object/manifest commit-before-error, same-namespace interleaving, malformed public jobs, all refill entry points, complete identity mismatches, legacy formats, corruption, key bounds and atomic batch isolation.
• GitHub CI for this exact head: all 6 checks PASS across Linux x86_64, native aarch64 and aarch64 QEMU. Both the push workflow and pull-request workflow passed.

Grok/VM reproduction from a clean checkout:

python3 scripts/ci/vm_regression.py --profile cpu --output artifacts/pr8-vm-regression

Keep environment.json, capabilities.json, results.json, summary.md, junit.xml, all logs and the process exit code. See the VM runbook.

Compatibility and failure behavior

Refill signatures now require expected: &CacheIdentity; tenant-only and identity-free payload access are removed. Raw ObjectStore access remains a trusted internal capability. Identity matching does not authenticate the caller or prove the producer's model computation. Upstream authentication and authorization remain required.

Manifest v3 and digest-derived keys require cold-cache repopulation. The hot protocol and object format remain unchanged. A full failed queue applies admission backpressure until take_failed or cancel releases its credits. A PUT failure can leave an orphan; the sink does not claim automatic reference-safe garbage collection. The byte budget bounds requested refill bodies at the trait boundary, not backend wire/RSS usage: the existing HTTP probe's head fetches a body to calculate its checksum.

Evidence limits and scope

This is CPU correctness evidence, not a VM backtest or a device performance result. GPU engine export/import, RDMA, GDS, DOCA, SPDK, BlueField offload and live RustFS remain NOT_RUN_HARDWARE. TCO_UNKNOWN and NOT_EVALUATED remain unchanged. ARM/QEMU CPU execution is not a DPU test.

The cold crate remains off the hot path. No hot-tier erasure coding or durability replicas are added. Callers fall back to recompute when cold refill is unavailable or incompatible; legacy entries are not rewritten or deleted by this migration.