Skip to content

Route each PR revision listing to its own task queue - #798

Merged
emmanuelmaduwuba-lyft merged 3 commits into
mainfrom
fix-pr-revision-slow-queue-routing
Nov 9, 2026
Merged

emmanuelmaduwuba-lyft merged 3 commits into
mainfrom
fix-pr-revision-slow-queue-routing

Conversation

@emmanuelmaduwuba-lyft

@emmanuelmaduwuba-lyft emmanuelmaduwuba-lyft commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In listModifiedFilesAsync (server/neptune/lyft/workflows/prrevision/workflow.go) the loop reassigns ctx with TaskQueue = pr_revision_slow for the first stale or automated PR and never resets it. PRs are listed newest-updated first, so every PR after the first automated PR also runs on the slow queue (0.15 activities/s instead of 2/s), even if it was updated yesterday. Separately, setRevision waits on each PR's listing in list order, so a recent PR's SetPRRevision waits behind every PR ahead of it, including PRs stuck on the slow queue.

The sticky ctx dates from the original multi-queue setup (6a3e746, 2023-04-06). It was safe then: the only condition was "not updated within N days", and because PRs are sorted newest-updated first, once one PR was stale every later PR was stale too. #753 added || pr.IsAutomatedPR, which can be true anywhere in the list, and that broke the assumption. #753 added no workflow test. The comment in setRevision about resolving default-queue futures before slow-queue ones relies on the same ordering assumption.

Change

  • Derive each PR's activity options from the shared ctx, so only stale and automated PRs use the slow queue.
  • Schedule all listings up front, then set each PR's revision as soon as its own listing completes, using one selector, instead of in list order. Without this, the queue fix alone would not help a recent PR listed after a slow one. A coroutine per PR was tried first and was about 8x slower at 600 PRs, so it is not used.
  • Gate both changes behind workflow.GetVersion("ConcurrentPRRevision"). The original code is kept as setRevisionLegacy / listModifiedFilesAsyncLegacy so in-flight workflows replay as they started.
  • Behavior difference: if a SetPRRevision call fails, the workflow still fails, but it now waits for every PR before returning the first error in list order. It used to return at the first error and abandon the rest.
  • Tests: routing cases, and a differential test of the original and new behavior over 200 randomized inputs.

Impact

Recent PRs listed after an automated or stale PR move from the slow queue back to the default queue, so their revisions are set sooner. The default queue gets more of the work, so pr_revision wait time may rise. GitHub API usage cannot exceed the existing per-queue caps (2/s and 0.15/s), so the API budget is unchanged. Rollback: revert this PR. Workflows that start after this deploys record the version marker, so reverting while they are still running could fail their replay; wait for in-flight PRRevision workflows to finish first.

Expected effect for one lyft/dsterraform PRRevision workflow, using its open PRs on 2026-11-06 (39 open, none stale, 5 automated, the first automated PR second in the list). Counts are GithubListModifiedFiles activities. Times use the queue caps (default 2/s, slow 0.15/s) and assume the queues are otherwise empty. Modelled from the PR list and the caps, not measured in production.

Per workflow Before After
File listings on the slow queue 37 5
File listings on the default queue 2 34
Time to finish all file listings about 4.1 min about 33 s
Time for the 34 recent PRs up to about 4.1 min about 17 s
A recent PR's revision waits on an earlier slow PR yes no

Testing

go test ./server/neptune/... passes (43 packages). -race -count=10 on the workflow packages is clean. go vet and golangci-lint are clean.

  • Differential test: the original and new behavior agree on which PRs are checked, which are revised and whether the workflow fails, over 200 randomized inputs. Routing matches the per-PR rule. Seven deliberate breakages of the new code were each caught. Removing the version gate fails the differential test and the legacy-version test.
  • Replay, run once locally and not committed: the new code replayed 29 production histories (24 completed, 5 in-flight; recorded by the deployed original code, exported read-only), 7 histories recorded by the original code and 7 recorded by the new code on a local Temporal server, all without a non-determinism error. With the version gate removed, the original-code histories failed. There is no replay test in the repo, so a later edit inside setRevisionLegacy that changes command order would not be caught by the tests here.
  • Benchmark (test environment, not production): 50 PRs 4.5 ms original vs 3.8 ms new; 600 PRs 62.8 ms vs 56.9 ms; memory at 600 PRs 31.5 MB vs 53.3 MB.

Not verified: drain time and queue behavior in production, and how the revision-setter service handles calls that arrive sooner.

The loop in listModifiedFilesAsync reassigned ctx with the slow task queue for
the first stale or automated PR and never reset it, so every PR after it also
ran on the slow queue. Choose the queue per PR from the shared ctx, and set each
PR's revision as its own listing completes so a PR stuck on the slow queue does
not hold up the PRs listed after it. Gated with workflow.GetVersion so workflows
started before this change replay unchanged.
…ests

Wait on one selector instead of one coroutine per PR, which grew quickly with
the number of open PRs. Add replay tests with histories recorded by the original
and the new workflow code, and a differential test that runs both behaviors over
randomized inputs.
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft marked this pull request as ready for review November 9, 2026 17:19
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft merged commit b6291a8 into main Nov 9, 2026
3 checks passed
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft deleted the fix-pr-revision-slow-queue-routing branch November 9, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants