Skip to content

v1.91.300.0 test(flaky): 2026-10-28 free Linux timing pass: ubicloud fake ssh, coordinator clone, qa-fix-loop late clock (GSTA-139 S4) - #3543

Merged
capy-ai[bot] merged 7 commits into
mainfrom
flaky-2026-10-28-free
Oct 29, 2026
Merged

capy-ai[bot] merged 7 commits into
mainfrom
flaky-2026-10-28-free

Conversation

@garrytan

@garrytan garrytan commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3543,
  "head": "b826dd4cc7deffc60fef950dade8fc7912c3a98c",
  "version": "1.91.294.0",
  "base": "main",
  "gate": {
    "remote_gate": {
      "pass": 10053,
      "fail": 0,
      "skip": 0
    }
  },
  "session_kind": "unattended",
  "pregate": "requires-remote cleared @15041e36c9f0 (1 checks, current for b826dd4cc7de)",
  "requires_remote": [
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/40717603992",
    "windows-free-tests.yml/windows-free-shard: https://github.com/garrytan/gstack/actions/runs/40717612065",
    "pregate.json/remote_gate: remote-gate tree=15041e36c9f0 tests=10053"
  ],
  "issues": {
    "fixes": [],
    "refs": [
      3413
    ],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "docs/, README.md, CONTRIBUTING.md, AGENTS.md, CLAUDE.md (re-audit s4-docs-2 at b826dd4; s4-docs-1 edited docs/TESTING_INTERNALS.md)"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 0,
    "fixed": 1,
    "scope": "frozen",
    "dispatches": {
      "specialist": 2
    }
  }
}

What changed

qa-fix-loop late-timeout under 24 busy loops: 0/10 before, 20/20 after (clock frozen through setup, as #3413 did for qa-only)

qa-fix-loop late-timeout under 24 busy loops on 4 cores: 0 of 10 runs passed before, 20 of 20 after. The clock now stands still through setup, as #3413 did for qa-only.

Before/after as text
before (main 4b8c03b): busy=24 runs=10 pass=0  (every run: attempts length 0 at :118)
after:                 busy=24 runs=20 pass=20 (late-timeout)
after review fix:      busy=24 runs=10 pass=10 (slow-setup + late-timeout)

Why

"We should have a flaky test fix pass every day that tries to fix flaky tests" (Garry, 2026-10-19). Today's ledger had two Linux reds that each came back twice, the ubicloud watchdog case and the ship-queue coordinator clone. It also held one latent load failure. Each now fails only when the code under test is wrong.

Blast radius

Only test fixtures and one doc paragraph change, and no product code does. The two fixture helpers that change are the fake ssh inside test/ubicloud-runner.test.ts and coordinatorClone, which only test/ship-queue-train.test.ts and test/gstack-ship-queue.test.ts call. Both files pass 31/31 and 65/65 (script ran), and the remote free tier passes at this head with 10,053 tests and 0 failures.

Approvals

Date Asked Exact words Outcome Covers Still pending
2026-10-19 Daily flaky-test fix pass (crew rule 16) “you can just run the failed test again and note that it is flaky and we should have a flaky test fix pass every day that tries to fix flaky tests” (Garry (via GStack Minder CREW.md)) standing approval GSTA-139 2026-10-28 flaky pass repairs; plan step exempt per docs/captain.md none

Source: gstack-approvals record for flaky-2026-10-28-free (display only; not an approval citation).

Summary

Three entries from handoffs/flaky-ledger.md (GSTA-139 lane S4), each repaired at its cause in the test fixture:

  • ubicloud-runner watchdog (repeat ×2, exit 2 and exit 0 instead of 124). The cause was not timing. Both reds finished in 1.2 s and 2.2 s, before the 3 s UBI_RUN_TIMEOUT could fire. The fake ssh's *-O* arm caught any remote command whose checkout path held -O, which happens whenever mkdtemp names the root ubi-run-O… (1 in 62). A replay with that prefix fails both watchdog cases every time with exit 2 in 1.3–1.5 s (tar child 141). The arm is deleted. It never matched ssh -O exit, whose last argument is the host. A regression test runs a checkout named ubi-run-O1x2y3: it fails 3/3 with the arm and passes 3/3 without it.
  • ship-queue coordinatorClone "failed to copy file … No such file or directory" (×2). Every push into the fixture origin leaves a detached git maintenance run --auto there. Trace2 and strace confirm it, because git's local transport drops the shard's GIT_CONFIG_COUNT backstop. A local clone copies origin.git/objects entry by entry. Churning maintenance.lock fails such a clone 3 times in 200, and --no-local fails 0 times in 200. coordinatorClone now clones through git's transport. Not claimed: the CI errors named a loose object, and that process is not identified (600 loaded push-then-clone runs did not reproduce it). The transport clone never lists that directory, so no entry appearing or vanishing there can fail it.
  • qa-fix-loop late-timeout (latent). It applies v1.91.222.0 test: daily flaky-fix pass: qa-only late clock, fragment-match verdict, install-admission timeout, ff-update reap, judge parse diagnostics (GSTA-91) #3413's qa-only clock freeze. A new slow-setup scenario keeps the setup-subtraction check that the freeze removed.

The fourth entry (gstack-gbrain-mcp-verify, local only) stays Open: it passed 10 of 10 runs under 24 busy loops and has no captured cause.

Test Coverage

All new code paths have test coverage. The three regression tests all fail without their fix and pass with it: the ubicloud holds -O test (exit 2 with the arm), the coordinator-clone symlink seam (refusing to clone with --local), and qa-fix-loop slow-setup (Expected 600, Received 1000 when the workflow passes the whole work budget).
Tests ran: remote_gate 10053

Pre-Landing Review

No blocking findings. Pass 1 (testing specialist, claude-opus-5-5) and a cross-family outside voice (openai/gpt-6.1-sol, run at the lane brief's request, because the review plan skips it for the tests class) found the same informational gap. The late-timeout freeze had dropped the check that the capture timeout subtracts setup time. It was fixed in pass 1 (slow-setup scenario), and pass 2 on the fix range found nothing.
REVIEW_LOOP: passes=2 blocking_found=0 deferred=0 noted=0 fixed=1 scope=frozen dispatches=2
Gate edits: 11 listed, 7 read, 0 findings (patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none). The edit to note: test/qa-fix-loop-fixture.test.ts late-timeout now asserts captureBudgetMs === 1000 (was 0 < x < 1000), and its test deadline is the registered 6080 ms outer budget (was budget − 5000 + 80). Negative control: with the workflow's outer budget reverted to the bare work budget, the scenario fails (afterFinalization true).
Outside review: completed (openai/gpt-6.1-sol subagent), no blocking findings. It notes that the two CI reds are attributed to the -O arm by signature (exit codes, durations, tar 141), because the CI logs carry no temp paths.

Eval Results

No prompt-related files changed, so evals were skipped.

Design Review

No frontend files changed, so design review was skipped.

Issues

Refs #3413

TODOS

No TODO items completed in this PR.

Documentation

Status: current. No documentation changes needed; the reviewed docs match the branch.

Scope: re-audit s4-docs-2 of flaky-2026-10-28-free (base 4b8c03b, head b826dd4): 6 committed files (CHANGELOG.md, docs/TESTING_INTERNALS.md, test/helpers/ship-queue-fixture.ts, test/qa-fix-loop-fixture.test.ts, test/ship-queue-train.test.ts, test/ubicloud-runner.test.ts). Nothing staged, unstaged or untracked. Docs were found by searching every tracked Markdown file and template for the changed names.

Documentation health:

  • docs/TESTING_INTERNALS.md: Current. The free-suite flake rule from s4-docs-1 is accurate. A live git 2.55.0 push confirmed that local-transport receive-pack drops GIT_CONFIG_*, and coordinatorClone uses --no-local.
  • docs/TEST_PORTFOLIO.md: Current. qa-fix-loop-fixture still owns the repair contract.
  • CHANGELOG.md: Current (read only). The Gate integrity bullet matches the 400 ms setup and the exact 600 ms capture assertion in slow-setup.
  • TODOS.md: Current (read only). The maintenance.lock item is already done.
  • CLAUDE.md: Current. It links to docs/TESTING_INTERNALS.md.

Coverage: all shipped changes are test fixtures with adequate documentation; no public surface changed. Diagram drift: none.

Test plan

  • Remote free tier, gstack-pregate --stage tests --remote --spend-cap 2 at b826dd4: 10,053 tests across 305 files, 0 failed ($0.10)
  • Dispatched at b826dd4 (CHANGELOG-only conflict leaves no merge ref): Free Tests and Windows Free Tests green; both requires-remote lanes cleared with those runs
  • bun test test/ubicloud-runner.test.ts 31/31; bun test test/ship-queue-train.test.ts test/gstack-ship-queue.test.ts 65/65; bun test test/qa-fix-loop-fixture.test.ts 11/11
  • 24 busy loops on 4 cores: qa-fix-loop late-timeout 0/10 → 20/20; slow-setup + late-timeout 10/10; gbrain-mcp-verify sources_add 10/10 (not reproduced; left Open)
  • bun run typecheck clean; bun run typecheck:test no new diagnostics; spawnsync tripwire, module-size ratchet, bun-entry isolation, git-ref tripwire green

🤖 Generated with Claude Code

garrytan and others added 3 commits October 28, 2026 13:48
…on a '-O' in the checkout name (GSTA-139)

The fake ssh's *-O* arm, meant for 'ssh -O exit' (whose last argument is the
host, so it never matched), caught any remote command whose path held '-O':
dest work/ubi-run-O????? whenever mkdtemp's first random character was O.
Sync then failed with exit 2 (tar child 141) or every phase exited 0, in about
1.3 s, before UBI_RUN_TIMEOUT could fire: Free Tests 40534568946 (:417, 2) and
40638411165 (:454, 0).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…it, not by copying its objects directory (GSTA-139)

Every push into the fixture origin leaves a detached 'git maintenance run
--auto' there (local transport drops the shard's GIT_CONFIG_COUNT backstop),
which creates and deletes objects/maintenance.lock after the push returns. A
local clone copies origin.git/objects entry by entry, so an entry appearing or
vanishing mid-copy fails it; --no-local never lists that directory.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… and registers the outer budget (GSTA-139)

The #3413 qa-only template: the late scenario's real setup ran inside its
1000 ms mocked work budget and a 1160 ms native deadline, so under load the
attempt never started (0/10 under 24 busy loops, attempts length 0). The clock
now stands still from the body's start until the capture begins, the capture
gets exactly 1000 ms, and the native deadline is the registered 6080 ms outer
budget. Gate edit: captureBudgetMs is asserted equal to 1000.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 4 commits October 28, 2026 13:50
…ment (GSTA-139)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…-subtraction check the late clock freeze dropped)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…in TESTING_INTERNALS

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title test(flaky): 2026-10-28 free Linux timing pass: ubicloud fake ssh, coordinator clone, qa-fix-loop late clock (GSTA-139 S4) v1.91.300.0 test(flaky): 2026-10-28 free Linux timing pass: ubicloud fake ssh, coordinator clone, qa-fix-loop late clock (GSTA-139 S4) Oct 28, 2026
@capy-ai
capy-ai Bot merged commit 766093d into main Oct 29, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant