Skip to content

feat(job): run pool handover jobs on a dedicated queue and worker - #657

Open
regdocs wants to merge 1 commit into
developfrom
feat/signup-worker
Open

regdocs wants to merge 1 commit into
developfrom
feat/signup-worker

Conversation

@regdocs

@regdocs regdocs commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Problem

A trial signup that gets a standby (pool) site waits for four short agent jobs:

  • Add Domain and Update Site Configuration on the app server
  • Add Domain to Upstream and Setup Redirects on Hosts on the proxy

Each runs in 1–2 seconds. But they shared the high/default queues with New Site, Update Site Migrate and Install App, which take minutes. Pool refills and pool site updates often start several of those on one server at the same time. When they hold every worker, the signup job waits for one to free up.

Over 7 days of pool signups, half of these jobs waited 2–4 s and 90% waited under 8 s. But 1% waited more than 80–90 s, and the longest waited about 5.5 minutes. In the cases checked, the signup job started within seconds of a long job ending on that server.

Change

  • The four handover jobs run on a new signup queue.
  • A new worker_signup supervisor program listens only on signup. It never picks up a long job while idle, so a signup job never waits behind one. This is unlike worker_backup, which also takes the shared queues.
  • The shared workers now list signup high default low. They also take signup jobs when free, so signups still run if the signup worker is down.
  • The worker count is signup_workers in the agent config, defaulting to 1. update_agent_web and update_agent_cli stop, start and restart the new program like worker_backup.

The same four jobs also run outside signups (custom domains, config changes). They are all short: in the last day the busiest server got 13 Update Site Configuration jobs in one minute, and each finishes in under a second. So one worker clears a burst in seconds.

These jobs take locks only for short config writes (the bench's nginx.config.lock, the proxy's config lock), so a long job can't block them through a lock.

Tests

TestServerSupervisorConfig renders the supervisor config and checks:

  • worker_signup listens only on signup.
  • The shared workers include signup.
  • worker_signup is in the agent group.

🤖 Generated with Claude Code

@mergify

mergify Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

A trial signup that gets a standby site waits for four short jobs: Add
Domain and Update Site Configuration on the app server, and Add Domain
to Upstream and Setup Redirects on Hosts on the proxy. Each runs in a
second or two, but they shared the queues with New Site, Update Site
Migrate and Install App, which take minutes. When long jobs held every
worker, the signup waited minutes for a free one.

Move these jobs to a new `signup` queue with a dedicated `worker_signup`
supervisor program that listens only on that queue, so a long job can
never hold it. The shared workers list `signup` first, so they also take
signup jobs when free. The worker count comes from `signup_workers` in
the agent config, defaulting to 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@regdocs
regdocs force-pushed the feat/signup-worker branch from 5b0302b to c74c403 Compare October 30, 2026 11:32
@regdocs regdocs changed the title feat(job): run signup domain jobs on a dedicated queue and worker feat(job): run pool handover jobs on a dedicated queue and worker Oct 30, 2026
@regdocs

regdocs commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

Press side of the same fix: frappe/press#7926. After these jobs finish, Press now reads them straight from the agent while the signup page is waiting, instead of waiting for the next poll.

Ordering check. With this PR, a short job can start before a long job queued earlier for the same site. Before, a short job could already run alongside that long job, but it couldn't start first. I checked prod for 26–29 Oct: 35,376 of these short jobs, joined with long site jobs on the same site.

  • About 60 short jobs were queued while a long job on that site was unfinished.
  • Migrate/Pull (low): 49 cases. The short jobs were high/default, so they already ran first. No change.
  • Restore/Archive (default): 12 cases. Update Site Configuration was high and already went first. The proxy jobs run on a different agent, so they were never ordered against app-server jobs. No change.
  • New Site / Rename Site / Recover… (high, the only case this PR changes): 2 cases, both queued after the long job had started. None were queued before it started.

On the Press side, dashboard config changes require the site to be Active or Broken (site_action), and site migration steps are queued one after another. The paths with no status check (marketplace sk_ key on install, team change, saas secret) didn't show up as queued before a long job started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant