Skip to content

Conversation

@coroa
Copy link
Member

@coroa coroa commented Jan 22, 2026

Add support for data.pypsa.org

This PR extends the cached-http storage plugin to support data.pypsa.org alongside the existing Zenodo support.

Changes

New Features:

  • Support for data.pypsa.org URLs with manifest-based metadata
  • Checksums are read from manifest.yaml files (searched up the directory tree)
  • Redirect support for data.pypsa.org files that point to other locations

Tests:

  • Refactored tests to use parametrized fixtures (["zenodo", "pypsa"])
  • All tests now run against both backends
  • Added test_is_valid_query_pypsa for URL validation

Documentation:

  • Updated README.md with data.pypsa.org examples and supported sources
  • Updated CLAUDE.md with architecture details for both backends

Example Usage

rule download_pypsa:
    input:
        storage("https://data.pypsa.org/workflows/eur/eez/v12_20231025/World_EEZ_v12_20231025_LR.zip"),
    output:
        "resources/eez.zip"
    shell:
        "cp {input} {output}"

@coroa coroa requested a review from lkstrp January 22, 2026 14:53
# The file is a small JSON file, should be less than 1MB
assert size < 1_000_000
else:
# data.pypsa.org manifests don't include size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As said above, worth to add?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's important, but it would not hurt.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, coming with the next sync (not too soon, might wanna add/ organize the metadata before that)

@coroa coroa marked this pull request as ready for review January 22, 2026 15:33
Copy link
Member

@lkstrp lkstrp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tested this but CI will 🤞

@coroa coroa merged commit 27cffbf into main Jan 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants