Repository navigation
Sweep orphaned deployment directories from the worker volume - #799
Draft
emmanuelmaduwuba-lyft wants to merge 1 commit into
Draft
emmanuelmaduwuba-lyft wants to merge 1 commit into
emmanuelmaduwuba-lyft wants to merge 1 commit into
Conversation
Each Terraform run extracts the repo into /atlantis-data/deployments/<id> and deletes it when the run finishes. Runs that never reach that cleanup (a failed or interrupted fetch, a killed pod, adhoc runs) leave the directory behind and nothing removes it, so the worker volumes fill up. Add a cron on each temporal worker that removes directories under <data dir>/deployments older than a max age (15 days by default). It logs only unless deployment_sweeper.delete is set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Each Terraform run extracts the repo into
<data dir>/deployments/<id>and deletes it when the run finishes. A run that never reaches that cleanup (failed or interrupted fetch, killed pod, adhoc run) leaves the directory behind, and nothing removes it. Over time these directories accumulate on the worker volumes.Change
DeploymentSweepercron (server/neptune/sync/crons/deployment_sweeper.go), registered on each temporal worker. It deletes immediate child directories of<data dir>/deploymentswhose mtime is older thanmax_age(default 15 days, minimum 24h). It skips files and symlinks, never touches thedeploymentsdirectory itself or anything else under the data dir (plugin cache etc.), and keeps going if one directory fails to delete.deployment_sweeperblock in the global config (max_age,delete). Leaving it out, ordelete: false, makes it log only. Deletion has to be turned on explicitly.deployment_sweeper.{scanned,stale,deleted,errors}metrics.Impact
Dry run by default, so merging and deploying changes nothing on disk. Deletion needs a follow-up config change (
deployment_sweeper.delete: true) after the dry-run logs show only old directories. Runs in flight are protected by the age threshold. It only removes working copies, not state or deployment history. Rollback is removing the config or reverting the bump.Testing
/data dir, max age under 24h). Config tests for the YAML block.go vet,golangci-lintandgo test ./server/neptune/... ./server/config/... ./cmd/...pass;-raceon the two new test packages passes.