fix: Atomic job reservation to prevent race condition

[dimitri-yatsenko]

Summary

• Fixes Bug: multiple workers reserving the same key in Job.reserve() in 2.1 #1398 — multiple workers could reserve the same job key simultaneously
• Replace the non-atomic SELECT → UPDATE in Job.reserve() with a single atomic UPDATE ... WHERE status='pending', checking cursor.rowcount for success
• Reduces three database round-trips to one
• Works on both MySQL and PostgreSQL backends

Root cause

The previous reserve() performed a SELECT to check status='pending', then a separate UPDATE matching only on primary key. When concurrent workers (e.g., SLURM array jobs) hit this simultaneously, both SELECTs see status='pending', both UPDATEs succeed (since the WHERE matched only on PK), and both workers proceed to call make() on the same key.

Fix

A single atomic UPDATE includes AND status='pending' AND scheduled_time <= CURRENT_TIMESTAMP(3) in the WHERE clause. The database guarantees only one concurrent UPDATE can match; all others get rowcount=0 and return False.

Test plan

• Verify existing job reservation tests pass
• Test with concurrent workers (SLURM or multiprocessing) to confirm only one worker reserves each key

🤖 Generated with Claude Code