Payload 4.0 stable timeline, and backport of the atomic job claim (#17441) to 3.x?

[tiendungnguyen24]

We run Payload 3.x jobs with more than one runner on the same queue and hit the non-atomic job claim: updateJobs in @payloadcms/db-mongodb claims a job in three steps (find → updateMany → find), so two runners can claim and run the same job. For now we limit ourselves to a single runner.

#17441 (feat!: support multiple workers on the same queue) fixes this, but it is only in 4.0, which has canary builds only.

1. Is there a rough timeline for a stable 4.0?
2. Since v3 is in maintenance mode, is a backport of the atomic claim to 3.x possible, at least findOneAndUpdate for limit: 1? It fixes a correctness bug (duplicate job runs), not a new feature.
3. Until then, is a single runner per queue the recommended setup on 3.x?

Thanks!

[tisankanj]

For the interim v3 setup, I would keep one active runner per queue and prevent overlapping run calls. That is a mitigation for this claim race, not an exactly-once guarantee or an official release recommendation. A single process can still overlap two cron/manual/API-triggered runs, and a rolling deployment can temporarily have both old and new runner replicas alive.

Your source-level diagnosis matches the MongoDB adapter in v3.90.2. In the limited, non-ID path, it finds candidate IDs, replaces the query with an _id filter, performs updateMany, and then reads those IDs back. That last read does not establish which runner won the claim. Setting limit: 1 alone still follows that path; it does not switch to findOneAndUpdate.

PR #17441 is useful here because its MongoDB fix still uses find/update/find. The difference is ownership: a conditional claim writes a unique processingToken, and the final read returns only jobs claimed with that runner's token. So three operations by themselves are not the issue; preserving eligibility during the update and identifying the winner are the important parts. The PR also explicitly separates claim exclusivity from leases/crash recovery.

Until you have a released fix validated for your version, I would check:

• Only one consumer is active for this queue across all replicas, including deployment overlap.
• autoRun, an external cron, a CLI runner and manual/API calls are not independently polling the same queue. Wait for one run to finish before starting the next.
• Job side effects tolerate retries, using a stable business-operation ID. For an external API that supports idempotency keys, reuse that key when retrying. A local “done” flag alone cannot protect the window where the external action succeeds and the worker crashes before recording completion.

A findOneAndUpdate branch for limit: 1 could address the MongoDB claim operation, but it needs to preserve the eligibility filter, sort, session and adapter transformations, and return only the successfully claimed document. I would treat it as a reviewed adapter change with a two-runner regression test, rather than patching just the final read or reducing the limit.

I can't give a 4.0 stable date or confirm a v3 backport commitment; those need a maintainer response. I reviewed the released source and the merged PR, but have not run your deployment or a MongoDB concurrency reproduction.