Skip to content

Sweep orphaned deployment directories from the worker volume - #799

Draft
emmanuelmaduwuba-lyft wants to merge 1 commit into
mainfrom
add-deployment-sweeper
Draft

emmanuelmaduwuba-lyft wants to merge 1 commit into
mainfrom
add-deployment-sweeper

Conversation

@emmanuelmaduwuba-lyft

Copy link
Copy Markdown
Contributor

Problem

Each Terraform run extracts the repo into <data dir>/deployments/<id> and deletes it when the run finishes. A run that never reaches that cleanup (failed or interrupted fetch, killed pod, adhoc run) leaves the directory behind, and nothing removes it. Over time these directories accumulate on the worker volumes.

Change

  • New DeploymentSweeper cron (server/neptune/sync/crons/deployment_sweeper.go), registered on each temporal worker. It deletes immediate child directories of <data dir>/deployments whose mtime is older than max_age (default 15 days, minimum 24h). It skips files and symlinks, never touches the deployments directory itself or anything else under the data dir (plugin cache etc.), and keeps going if one directory fails to delete.
  • Runs every 6 hours, not daily: cron ticks only fire after a full interval, and pods restart on every deploy.
  • Config: optional deployment_sweeper block in the global config (max_age, delete). Leaving it out, or delete: false, makes it log only. Deletion has to be turned on explicitly.
  • Logs a summary per sweep and emits deployment_sweeper.{scanned,stale,deleted,errors} metrics.
  • Not in this PR: the fetch step still doesn't remove its own partial directory on failure.

Impact

Dry run by default, so merging and deploying changes nothing on disk. Deletion needs a follow-up config change (deployment_sweeper.delete: true) after the dry-run logs show only old directories. Runs in flight are protected by the age threshold. It only removes working copies, not state or deployment history. Rollback is removing the config or reverting the bump.

Testing

  • New tests for the sweeper: deletes only stale directories, default max age, dry run, leaves files/symlinks/siblings/the deployments dir alone, missing directory, cancelled context, continues after a delete failure, rejects unsafe config (empty or / data dir, max age under 24h). Config tests for the YAML block.
  • Broke the sweeper on purpose four ways (dry run deleting, age check removed, non-directories not skipped, stopping on first failure); each made a test fail.
  • go vet, golangci-lint and go test ./server/neptune/... ./server/config/... ./cmd/... pass; -race on the two new test packages passes.
  • Not verified: a run on a real worker. The first signal will be the dry-run log lines.

Each Terraform run extracts the repo into /atlantis-data/deployments/<id> and
deletes it when the run finishes. Runs that never reach that cleanup (a failed
or interrupted fetch, a killed pod, adhoc runs) leave the directory behind and
nothing removes it, so the worker volumes fill up.

Add a cron on each temporal worker that removes directories under
<data dir>/deployments older than a max age (15 days by default). It logs only
unless deployment_sweeper.delete is set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant