Crawler can get softlocked when consuming the RequestQueue

[Pijukatel]

Example scenario:

There is only 1 request in RQ
The request is processed for a long time - it fails twice, and it will pass only on the third attempt.
After each failure, the request is reclaimed.

The problem is that AutoscalePool can go completely rouge and can start burning resources up to the point where it blocks itself due to the system being overloaded.

The root cause is that __is_task_ready_function (which decides whether a new task should be scheduled) is checking if the not await request_manager.is_empty(). Since there is one request in progress and RQ is not empty, this will return True and the AutoscalePool will schedule a new task. The problem is that RQ has no available requests as the only one is already in progress. This will create a situation where new tasks are spawned as fast as possible, only to get no request from RQ.

Preferably, in such scenario __is_task_ready_function should return False, because the RQ should be aware of the fact that the only request available is already being handled. So __is_task_ready_function should return True if there are some available requests and return False if the only requests available are already being processed - but there is no such interface on RQ for this.

Bottom lime not RequestManager.is_empty does not imply __is_task_ready, but our code has this expectation.

[Pijukatel]

Exposed when trying to add MacOS executors to CI: #1334

[janbuchar]

I see two problems here:

1. the apify.storages.ApifyRequestQueueClient.is_empty method returns True even when there are requests being handled
   • this is probably because is_empty and is_finished got merged into a single method at some point, which is a bit of a pickle
   • crawlee.storages.RequestQueue provides a general implementation of is_finished, but this does not take in-progress requests into account
2. the fact that not is_empty is expected to imply __is_task_ready is neither documented nor enforced, as you mentioned

[Pijukatel]

There might be something OS specific that makes this situation bad up to the softlock point, as it is fairly easy to reproduce only macOS.

[Pijukatel]

It is probably mainly a theoretical problem. The real issue when reproducing it on the MacOS executor in GitHub was the executor itself - which consumed 100% CPU in certain cases and was blocking the AutoscaledPool regardless of the scenario apify/workflows#236

[janbuchar]

It is probably mainly a theoretical problem.

I don't think so, the fact that we don't distinguish is_empty from is_finished in the request queue client sounds like something that could bite us later.