Repository navigation
Route each PR revision listing to its own task queue - #798
Merged
Merged
Conversation
The loop in listModifiedFilesAsync reassigned ctx with the slow task queue for the first stale or automated PR and never reset it, so every PR after it also ran on the slow queue. Choose the queue per PR from the shared ctx, and set each PR's revision as its own listing completes so a PR stuck on the slow queue does not hold up the PRs listed after it. Gated with workflow.GetVersion so workflows started before this change replay unchanged.
…ests Wait on one selector instead of one coroutine per PR, which grew quickly with the number of open PRs. Add replay tests with histories recorded by the original and the new workflow code, and a differential test that runs both behaviors over randomized inputs.
emmanuelmaduwuba-lyft
marked this pull request as ready for review
November 9, 2026 17:19
jfernandoz
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
listModifiedFilesAsync(server/neptune/lyft/workflows/prrevision/workflow.go) the loop reassignsctxwithTaskQueue = pr_revision_slowfor the first stale or automated PR and never resets it. PRs are listed newest-updated first, so every PR after the first automated PR also runs on the slow queue (0.15 activities/s instead of 2/s), even if it was updated yesterday. Separately,setRevisionwaits on each PR's listing in list order, so a recent PR'sSetPRRevisionwaits behind every PR ahead of it, including PRs stuck on the slow queue.The sticky
ctxdates from the original multi-queue setup (6a3e746, 2023-04-06). It was safe then: the only condition was "not updated within N days", and because PRs are sorted newest-updated first, once one PR was stale every later PR was stale too. #753 added|| pr.IsAutomatedPR, which can be true anywhere in the list, and that broke the assumption. #753 added no workflow test. The comment insetRevisionabout resolving default-queue futures before slow-queue ones relies on the same ordering assumption.Change
ctx, so only stale and automated PRs use the slow queue.workflow.GetVersion("ConcurrentPRRevision"). The original code is kept assetRevisionLegacy/listModifiedFilesAsyncLegacyso in-flight workflows replay as they started.SetPRRevisioncall fails, the workflow still fails, but it now waits for every PR before returning the first error in list order. It used to return at the first error and abandon the rest.Impact
Recent PRs listed after an automated or stale PR move from the slow queue back to the default queue, so their revisions are set sooner. The default queue gets more of the work, so
pr_revisionwait time may rise. GitHub API usage cannot exceed the existing per-queue caps (2/s and 0.15/s), so the API budget is unchanged. Rollback: revert this PR. Workflows that start after this deploys record the version marker, so reverting while they are still running could fail their replay; wait for in-flightPRRevisionworkflows to finish first.Expected effect for one
lyft/dsterraformPRRevisionworkflow, using its open PRs on 2026-11-06 (39 open, none stale, 5 automated, the first automated PR second in the list). Counts areGithubListModifiedFilesactivities. Times use the queue caps (default 2/s, slow 0.15/s) and assume the queues are otherwise empty. Modelled from the PR list and the caps, not measured in production.Testing
go test ./server/neptune/...passes (43 packages).-race -count=10on the workflow packages is clean.go vetandgolangci-lintare clean.setRevisionLegacythat changes command order would not be caught by the tests here.Not verified: drain time and queue behavior in production, and how the revision-setter service handles calls that arrive sooner.