Skip to content

Do not lock terminating jobs in process_runs #3580

@r4victor

Description

@r4victor

Related to #3551

process_runs currently locks terminating jobs only to override termination reason in case of abort:

unfinished_jobs_count += 1
if job_model.status == JobStatus.TERMINATING:
if job_termination_reason == JobTerminationReason.ABORTED_BY_USER:
# Override termination reason so that
# abort actions such as volume force detach are triggered
job_model.termination_reason = job_termination_reason
continue

In principle, this can be moved to process_terminating_jobs, so that process_runs does not need to lock terminating jobs and then process_runs and process_terminating_jobs can work simultaneously, reducing job lock contention.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions