Repository navigation
Start the deploy apply job log with the plan output - #800
Merged
Merged
Conversation
In deploy mode the plan and apply run as separate jobs with separate logs, and only the apply job's log is persisted under the id that is audited and shown downstream. Before the apply starts, copy the plan job's output into the apply job so its log shows plan then apply. The copy is a new activity gated behind workflow.GetVersion so workflows already past this point replay unchanged. It is best effort and never retried (a retry would write the plan twice): a failure is logged and the apply still runs.
emmanuelmaduwuba-lyft
marked this pull request as ready for review
November 13, 2026 17:13
So readers of the combined apply log can see where the plan ends and the apply begins. Nothing is written when there is no plan output.
mariomsan
previously approved these changes
Nov 13, 2026
Broadcast drops a receiver that isn't ready to receive, so the test's unbuffered channel made it wait forever whenever Broadcast ran before the read. Buffer the channel so the message is not dropped.
jfernandoz
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In deploy mode the plan and apply run as separate jobs with separate logs. Only the apply job's log is persisted under the apply job id, which is the id that is audited and copied downstream, so anything reading the apply log (the apply link, and the deploy tooling that copies it) shows the apply without the plan that was applied.
Change
CopyJobOutputactivity andStreamHandler.CopyJob: before the apply starts, the plan job's output is written into the apply job, so the apply log reads plan then apply. The plan is read from memory or from the storage backend if it was already persisted.workflow.GetVersion("CopyPlanOutputToApplyJob")because it adds an activity: workflows already past this step replay without it, workflows still waiting on approval run it once approved.---- terraform apply ----line is written between the copied plan and the apply's own output, so the combined log shows where one ends and the other begins. Nothing is written when there is no plan output.Example of a deploy-mode apply log (illustrative output). Before, only the apply lines; after, the plan lines come first, then the divider:
Impact
After deploy, new deploy-mode applies have a log that starts with the plan. The apply link is unchanged (
/jobs/<apply-id>), the audited job id is unchanged, and nothing downstream needs to change. Plan output is stored a second time inside the apply log, so apply logs grow by the size of the plan, and the apply waits for one extra activity on the same worker. Applies that already ran keep their apply-only logs. Rollback is reverting this PR, but not while an apply is running: a workflow that already ran the copy has the version marker and the activity in its history, so replaying it on the old code fails with a non-determinism error. Roll back (or deploy a fix) only when no deploy-mode apply is in flight. During a rolling deploy, an apply whose pod is still on the old code has noCopyJobOutputto run, so the copy fails and is logged, and that apply keeps an apply-only log.Testing
TestStreamHandler_CopyJob(order, divider, broadcast, later apply lines follow the plan, missing source, read and write errors); workflow tests that deploy mode copies once with distinct plan and apply ids, that PR mode never copies, and that a failed copy does not fail the deploy.go vet,golangci-linton the touched packages, andgo testoverserver/neptune/workflows/...andserver/neptune/temporalworker/...pass.go test -race ./server/neptune/workflows/fails with a data race in the existingtestStreamCloser.RegisterJobtest double (deploy_test.go), the same way on main without this change.