Repository navigation
v1.91.300.0 test(flaky): 2026-10-28 free Linux timing pass: ubicloud fake ssh, coordinator clone, qa-fix-loop late clock (GSTA-139 S4) - #3543
Merged
Conversation
…on a '-O' in the checkout name (GSTA-139) The fake ssh's *-O* arm, meant for 'ssh -O exit' (whose last argument is the host, so it never matched), caught any remote command whose path held '-O': dest work/ubi-run-O????? whenever mkdtemp's first random character was O. Sync then failed with exit 2 (tar child 141) or every phase exited 0, in about 1.3 s, before UBI_RUN_TIMEOUT could fire: Free Tests 40534568946 (:417, 2) and 40638411165 (:454, 0). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…it, not by copying its objects directory (GSTA-139) Every push into the fixture origin leaves a detached 'git maintenance run --auto' there (local transport drops the shard's GIT_CONFIG_COUNT backstop), which creates and deletes objects/maintenance.lock after the push returns. A local clone copies origin.git/objects entry by entry, so an entry appearing or vanishing mid-copy fails it; --no-local never lists that directory. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… and registers the outer budget (GSTA-139) The #3413 qa-only template: the late scenario's real setup ran inside its 1000 ms mocked work budget and a 1160 ms native deadline, so under load the attempt never started (0/10 under 24 busy loops, attempts length 0). The clock now stands still from the body's start until the capture begins, the capture gets exactly 1000 ms, and the native deadline is the registered 6080 ms outer budget. Gate edit: captureBudgetMs is asserted equal to 1000. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
…ment (GSTA-139) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…-subtraction check the late clock freeze dropped) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…in TESTING_INTERNALS Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
qa-fix-loop late-timeout under 24 busy loops on 4 cores: 0 of 10 runs passed before, 20 of 20 after. The clock now stands still through setup, as #3413 did for qa-only.
Before/after as text
Why
"We should have a flaky test fix pass every day that tries to fix flaky tests" (Garry, 2026-10-19). Today's ledger had two Linux reds that each came back twice, the ubicloud watchdog case and the ship-queue coordinator clone. It also held one latent load failure. Each now fails only when the code under test is wrong.
Blast radius
Only test fixtures and one doc paragraph change, and no product code does. The two fixture helpers that change are the fake ssh inside test/ubicloud-runner.test.ts and coordinatorClone, which only test/ship-queue-train.test.ts and test/gstack-ship-queue.test.ts call. Both files pass 31/31 and 65/65 (script ran), and the remote free tier passes at this head with 10,053 tests and 0 failures.
Approvals
Source: gstack-approvals record for flaky-2026-10-28-free (display only; not an approval citation).
Summary
Three entries from handoffs/flaky-ledger.md (GSTA-139 lane S4), each repaired at its cause in the test fixture:
*-O*arm caught any remote command whose checkout path held-O, which happens whenever mkdtemp names the rootubi-run-O…(1 in 62). A replay with that prefix fails both watchdog cases every time with exit 2 in 1.3–1.5 s (tar child 141). The arm is deleted. It never matchedssh -O exit, whose last argument is the host. A regression test runs a checkout namedubi-run-O1x2y3: it fails 3/3 with the arm and passes 3/3 without it.git maintenance run --autothere. Trace2 and strace confirm it, because git's local transport drops the shard'sGIT_CONFIG_COUNTbackstop. A local clone copiesorigin.git/objectsentry by entry. Churningmaintenance.lockfails such a clone 3 times in 200, and--no-localfails 0 times in 200. coordinatorClone now clones through git's transport. Not claimed: the CI errors named a loose object, and that process is not identified (600 loaded push-then-clone runs did not reproduce it). The transport clone never lists that directory, so no entry appearing or vanishing there can fail it.The fourth entry (gstack-gbrain-mcp-verify, local only) stays Open: it passed 10 of 10 runs under 24 busy loops and has no captured cause.
Test Coverage
All new code paths have test coverage. The three regression tests all fail without their fix and pass with it: the ubicloud
holds -Otest (exit 2 with the arm), the coordinator-clone symlink seam (refusing to clone with --local), and qa-fix-loop slow-setup (Expected 600, Received 1000 when the workflow passes the whole work budget).Tests ran: remote_gate 10053
Pre-Landing Review
No blocking findings. Pass 1 (testing specialist, claude-opus-5-5) and a cross-family outside voice (openai/gpt-6.1-sol, run at the lane brief's request, because the review plan skips it for the tests class) found the same informational gap. The late-timeout freeze had dropped the check that the capture timeout subtracts setup time. It was fixed in pass 1 (slow-setup scenario), and pass 2 on the fix range found nothing.
REVIEW_LOOP: passes=2 blocking_found=0 deferred=0 noted=0 fixed=1 scope=frozen dispatches=2
Gate edits: 11 listed, 7 read, 0 findings (patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none). The edit to note: test/qa-fix-loop-fixture.test.ts late-timeout now asserts
captureBudgetMs === 1000(was 0 < x < 1000), and its test deadline is the registered 6080 ms outer budget (was budget − 5000 + 80). Negative control: with the workflow's outer budget reverted to the bare work budget, the scenario fails (afterFinalizationtrue).Outside review: completed (openai/gpt-6.1-sol subagent), no blocking findings. It notes that the two CI reds are attributed to the
-Oarm by signature (exit codes, durations, tar 141), because the CI logs carry no temp paths.Eval Results
No prompt-related files changed, so evals were skipped.
Design Review
No frontend files changed, so design review was skipped.
Issues
Refs #3413
TODOS
No TODO items completed in this PR.
Documentation
Status: current. No documentation changes needed; the reviewed docs match the branch.
Scope: re-audit
s4-docs-2offlaky-2026-10-28-free(base 4b8c03b, head b826dd4): 6 committed files (CHANGELOG.md, docs/TESTING_INTERNALS.md, test/helpers/ship-queue-fixture.ts, test/qa-fix-loop-fixture.test.ts, test/ship-queue-train.test.ts, test/ubicloud-runner.test.ts). Nothing staged, unstaged or untracked. Docs were found by searching every tracked Markdown file and template for the changed names.Documentation health:
GIT_CONFIG_*, andcoordinatorCloneuses--no-local.qa-fix-loop-fixturestill owns the repair contract.Coverage: all shipped changes are test fixtures with adequate documentation; no public surface changed. Diagram drift: none.
Test plan
gstack-pregate --stage tests --remote --spend-cap 2at b826dd4: 10,053 tests across 305 files, 0 failed ($0.10)bun test test/ubicloud-runner.test.ts31/31;bun test test/ship-queue-train.test.ts test/gstack-ship-queue.test.ts65/65;bun test test/qa-fix-loop-fixture.test.ts11/11bun run typecheckclean;bun run typecheck:testno new diagnostics; spawnsync tripwire, module-size ratchet, bun-entry isolation, git-ref tripwire green🤖 Generated with Claude Code