Always retry failed feedstocks if they have received commits since the last bot attempt

[h-vetinari]

In most bigger migrations, we come to a point where some package with lots of children failed spuriously and then takes ages to be retried. For the 3.14 migration, the bot stumbled over numpy (discussion) and despite the fact that conda-forge/numpy-feedstock#363 had long been merged, the blocker still exists since the restart of the 3.14 migration 36h ago.

.

In this particular case, the reason looks to be that the solver checks still fail as before even when the bot does retry things. Additionally, #4650 added some logic to boost feedstock with lots of children (i.e. broadly trying to tackle the same issue).

However, I think there's a bigger win to be had here, i.e. avoid wasteful retries (where nothing has changed), but prioritise retries where they're most impactful. The logic for that would be pretty simple (I'm assuming we have all the timestamps available in the graph and the bot-metadata):

whenever a feedstock has been updated since the last failed bot attempt, always retry that feedstock

Obviously, if there are several feedstocks like this, we can sort them in descending order by number of children and then cap it at some number (e.g. the top 5 previously-failed-but-updated-in-the-meantime feedstocks get retried unconditionally, outside of the bot share accounting, while the rest runs as usual).

The reason should be obvious: if the feedstock has been updated, the reason for the previous bot failure might very well be gone (and in the case of manual fixes to feedstocks because the migration was bottlenecked on them, that might even have been the whole point)

[beckermr]

I am down for this, but tracking the state of the feedstock is going to be real work. We'll need to embed a commit sha of the feedstock into the pr attempts data.

[h-vetinari]

I don't see why we'd need to save the sha -- can't we just take the timestamp of the last commit on the main branch, and compare that with the timestamp of the last attempt?

[beckermr]

Sure but that difference is not significant. We'll still need to add a timestamp to the attempts data. The file modification time won't work since that data structure holds more than one migrations info.

[h-vetinari]

Sure, we need to save the timestamp of the failed bot attempt. I assumed we had that already. What I meant is that I don't see why we'd need to save the sha/timestamp of the feedstock commit that failed to render

[h-vetinari]

I was looking at the annotated diff in #4774 (thanks for that @beckermr, very helpful!), and wondering if we could now simply add a check in

cf-scripts/conda_forge_tick/migrators/core.py

Lines 757 to 759 in 563b461

	def _attempt_pr(node):
	last_bot_attempt_ts, retries_so_far = _get_last_attempt_ts_and_try(node)
	return now > last_bot_attempt_ts + (base * (2 ** min(retries_so_far, 6)))

i.e. whether the feedstock has been updated after last_bot_attempt_ts. The question if that is part of the node metadata already (then it's super cheap), or whether we'd have to query that from the feedstock (which would be infeasibly expensive, and we'd have to add extend the node metadata that's produced by the graph scraper?).

Assuming we can get the timestamp of the last commit, then we could do

 def _attempt_pr(node): 
     last_bot_attempt_ts, retries_so_far = _get_last_attempt_ts_and_try(node)
     last_feedstock_commit_on_main_ts = is_this_available_somewhere_in(node)  # ?
     retry_due_to_new_commit = last_feedstock_commit_on_main_ts > last_bot_attempt_ts
     exponential_backoff_passed = now > last_bot_attempt_ts + (base * (2 ** min(retries_so_far, 6)))
     return retry_due_to_new_commit or exponential_backoff_passed

We could (and probably should) also reset the exponential backoff after a new commit on the feedstock.

[beckermr]

That info is not stored. We can have the bot start scarping it though.

[h-vetinari]

Cool! This would have to ultimately go into the bot-make-graph task, right? Or is it bot-update-nodes, perhaps?

[beckermr]

It goes in the update nodes job. That code is spread across the repo. I can send a pr this morning.

[beckermr]

These hashes+timestamps are appearing: regro/cf-graph-countyfair@5f3c0ff

Not every node will have them at first and so you'll need to only use them if you find the timestamp.