Skip to content

Start the deploy apply job log with the plan output - #800

Merged
emmanuelmaduwuba-lyft merged 3 commits into
mainfrom
combine-plan-apply-job-logs
Nov 13, 2026
Merged

emmanuelmaduwuba-lyft merged 3 commits into
mainfrom
combine-plan-apply-job-logs

Conversation

@emmanuelmaduwuba-lyft

@emmanuelmaduwuba-lyft emmanuelmaduwuba-lyft commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In deploy mode the plan and apply run as separate jobs with separate logs. Only the apply job's log is persisted under the apply job id, which is the id that is audited and copied downstream, so anything reading the apply log (the apply link, and the deploy tooling that copies it) shows the apply without the plan that was applied.

Change

  • New CopyJobOutput activity and StreamHandler.CopyJob: before the apply starts, the plan job's output is written into the apply job, so the apply log reads plan then apply. The plan is read from memory or from the storage backend if it was already persisted.
  • It runs in the terraform workflow after the plan is approved, right before the apply job runs. Only deploy mode reaches that point, PR and adhoc workflows return earlier.
  • Gated behind workflow.GetVersion("CopyPlanOutputToApplyJob") because it adds an activity: workflows already past this step replay without it, workflows still waiting on approval run it once approved.
  • Best effort and never retried: a retry would write the plan twice. A failure is logged and the apply still runs.
  • A ---- terraform apply ---- line is written between the copied plan and the apply's own output, so the combined log shows where one ends and the other begins. Nothing is written when there is no plan output.
  • Not in this PR: a size cap on the copy.

Example of a deploy-mode apply log (illustrative output). Before, only the apply lines; after, the plan lines come first, then the divider:

# before
aws_s3_bucket.logs: Creating...
aws_s3_bucket.logs: Creation complete after 2s
aws_iam_role.worker: Modifying...
aws_iam_role.worker: Modifications complete after 1s

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

# after
$ terraform plan
  + aws_s3_bucket.logs will be created
  ~ aws_iam_role.worker will be updated in-place

Plan: 1 to add, 1 to change, 0 to destroy.
---- terraform apply ----
aws_s3_bucket.logs: Creating...
aws_s3_bucket.logs: Creation complete after 2s
aws_iam_role.worker: Modifying...
aws_iam_role.worker: Modifications complete after 1s

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

Impact

After deploy, new deploy-mode applies have a log that starts with the plan. The apply link is unchanged (/jobs/<apply-id>), the audited job id is unchanged, and nothing downstream needs to change. Plan output is stored a second time inside the apply log, so apply logs grow by the size of the plan, and the apply waits for one extra activity on the same worker. Applies that already ran keep their apply-only logs. Rollback is reverting this PR, but not while an apply is running: a workflow that already ran the copy has the version marker and the activity in its history, so replaying it on the old code fails with a non-determinism error. Roll back (or deploy a fix) only when no deploy-mode apply is in flight. During a rolling deploy, an apply whose pod is still on the old code has no CopyJobOutput to run, so the copy fails and is logged, and that apply keeps an apply-only log.

Testing

  • New tests: TestStreamHandler_CopyJob (order, divider, broadcast, later apply lines follow the plan, missing source, read and write errors); workflow tests that deploy mode copies once with distinct plan and apply ids, that PR mode never copies, and that a failed copy does not fail the deploy.
  • Broke the copy on purpose (skipping the write, and dropping the divider): the stream handler tests fail each time.
  • go vet, golangci-lint on the touched packages, and go test over server/neptune/workflows/... and server/neptune/temporalworker/... pass.
  • go test -race ./server/neptune/workflows/ fails with a data race in the existing testStreamCloser.RegisterJob test double (deploy_test.go), the same way on main without this change.
  • Not verified: a run on a real worker, how long the extra activity takes, and behavior for very large plan logs (memory on the worker, and the downstream copy of the stored log, which has a short timeout).

In deploy mode the plan and apply run as separate jobs with separate
logs, and only the apply job's log is persisted under the id that is
audited and shown downstream. Before the apply starts, copy the plan
job's output into the apply job so its log shows plan then apply.

The copy is a new activity gated behind workflow.GetVersion so
workflows already past this point replay unchanged. It is best effort
and never retried (a retry would write the plan twice): a failure is
logged and the apply still runs.
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft marked this pull request as ready for review November 13, 2026 17:13
So readers of the combined apply log can see where the plan ends and
the apply begins. Nothing is written when there is no plan output.
mariomsan
mariomsan previously approved these changes Nov 13, 2026
Broadcast drops a receiver that isn't ready to receive, so the test's
unbuffered channel made it wait forever whenever Broadcast ran before
the read. Buffer the channel so the message is not dropped.
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft merged commit 530f38f into main Nov 13, 2026
3 checks passed
@emmanuelmaduwuba-lyft
emmanuelmaduwuba-lyft deleted the combine-plan-apply-job-logs branch November 13, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants