-
Notifications
You must be signed in to change notification settings - Fork 215
Open
Labels
Description
Related to #3551
process_runs currently locks terminating jobs only to override termination reason in case of abort:
dstack/src/dstack/_internal/server/services/runs/__init__.py
Lines 991 to 997 in ad9fd69
| unfinished_jobs_count += 1 | |
| if job_model.status == JobStatus.TERMINATING: | |
| if job_termination_reason == JobTerminationReason.ABORTED_BY_USER: | |
| # Override termination reason so that | |
| # abort actions such as volume force detach are triggered | |
| job_model.termination_reason = job_termination_reason | |
| continue |
In principle, this can be moved to process_terminating_jobs, so that process_runs does not need to lock terminating jobs and then process_runs and process_terminating_jobs can work simultaneously, reducing job lock contention.
Reactions are currently unavailable