fix: add User-Agent header to download requests

[dakomura]

Problem

Dataset downloads fail with 403 Forbidden error when downloading from Zenodo.

HTTPError: 403 Client Error: Forbidden for url:
https://zenodo.org/api/records/1214456/files/NCT-CRC-HE-100K.zip/content

Cause

Zenodo (and some other servers) reject HTTP requests that don't include a proper User-Agent header. The default Python requests library sends python-requests/x.x.x as User-Agent, which is being blocked.

Solution

Add a User-Agent header identifying thunder-bench to all download requests in download_from_url().

Testing

Verified that the CRC dataset download works correctly after applying this fix.

[PierreMarza]

Hello! Thanks for this valuable fix! Just a quick question: to the best of my knowledge, we haven't experienced such an error when downloading datasets from Zenodo without specifying a header. I tried to download the crc dataset again with the current version of the code (without headers), and it seems to work well for me. Do you know whether the error you have could come from a different version of the requests library? I used version 2.32.3 in the test I just did. Anyway, I think adding a header as you did might be safer, so I'll be happy to merge your PR :)

[dakomura]

Thank you for the review and for your helpful question!

I did some investigation on my side, and I think I may have found why we're seeing different results.

What I found

When I tested different User-Agent strings, I noticed an interesting pattern:

User-Agent	Response
python-requests/2.32.5	403 Forbidden
python-requests/2.32.4	200 OK
python-requests/2.32.3	200 OK
No User-Agent (empty headers)	200 OK
Custom User-Agent	200 OK

It seems like Zenodo might be blocking requests specifically from python-requests/2.32.5 (my version), while 2.32.3 (your version) works fine.

Possible explanation

This could explain why you weren't able to reproduce the issue - the User-Agent string from your requests version isn't affected. I'm not entirely sure why Zenodo would block a specific version, but it might be related to some anti-bot measures on their end.

Simple test (if you're curious)

import requests
URL = 'https://zenodo.org/api/records/1214456'
for ua in ['python-requests/2.32.5', 'python-requests/2.32.3']:
   r = requests.head(URL, headers={'User-Agent': ua}, timeout=30)
   print(f'{ua}: {r.status_code}')

Conclusion

Regardless of the exact cause, I think adding a custom User-Agent is a reasonable safeguard, especially as more users may upgrade to newer versions of requests. Thank you again for considering this PR! 😊