Skip to content

Feedback wanted - Design outline for the temporal window pipeline step #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tennlee merged 7 commits into from
Oct 29, 2025
Merged

Feedback wanted - Design outline for the temporal window pipeline step #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
42997ce
Update zenodo.json to reflect contribution rates
tennlee Oct 29, 2025
9c9892f
Design outline for the temporal window pipeline step
tennlee Aug 13, 2025
da37311
Ran black.
tennlee Aug 13, 2025
e2a076b
Add the TemporalWindow class to the available index modifiers
tennlee Oct 29, 2025
aee378d
Add temporal window to docs
tennlee Oct 29, 2025
8ed4b7e
Remove doc entries that were causing errors (classes have moved)
tennlee Oct 29, 2025
e2fc35d
Code reformat
tennlee Oct 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 85 additions & 87 deletions .zenodo.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,89 +1,87 @@
{
"creators": [
{
"orcid": "https://orcid.org/0009-0009-3207-4876",
"affiliation": "Work undertaken while at the Bureau of Meteorology, Australia",
"name": "Cook, Harrison"
},
{
"orcid": "https://orcid.org/0009-0008-2024-1967",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Leeuwenburg, Tennessee"
},
{
"orcid": "https://orcid.org/0000-0003-2553-0721",
"affiliation": "NIWA - The National Institute of Water and Atmospheric Research, New Zealand",
"name": "Rio, Maxime"
},
{
"orcid": "https://orcid.org/0000-0002-3148-4496",
"affiliation": "Met Office, United Kingdom",
"name": "Miller, Joel"
},
{
"orcid": "https://orcid.org/0000-0002-6569-1178",
"affiliation": "NIWA - The National Institute of Water and Atmospheric Research, New Zealand",
"name": "Mason, Gemma"
},
{
"orcid": "https://orcid.org/0009-0002-7406-7438",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Ramanathan, Nikeeth"
},
{
"orcid": "https://orcid.org/0009-0001-6250-3987",
"affiliation": "Met Office, United Kingdom",
"name": "Pill, John"
},
{
"orcid": "https://orcid.org/0000-0003-3430-2456",
"affiliation": "Met Office, United Kingdom",
"name": "Haddad, Stephen"
},
{
"orcid": "https://orcid.org/0000-0002-1975-0042",
"affiliation": "Bureau of Meteorology, Australia",
"name": "de Burgh-Day, Catherine"
},
{
"orcid": "https://orcid.org/0009-0000-5048-4240",
"affiliation": "Met Office, United Kingdom",
"name": "Sullivan, Ben"
},
{
"orcid": "https://orcid.org/0000-0001-6825-3854",
"affiliation": "University of New South Wales Sydney, Australia",
"name": "Hobeichi, Sanaa"

},
{
"orcid": "https://orcid.org/0000-0002-6799-9109",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Holmes, Ryan"
},
{
"orcid": "https://orcid.org/0009-0009-8268-8358",
"affiliation": "Independent researcher",
"name": "Potokina, Margarita"
},
{
"orcid": "https://orcid.org/0009-0001-4486-0120",
"affiliation": "Independent researcher, Vietnam",
"name": "Bogacheva, Jenya"
},
{
"orcid": "https://orcid.org/0009-0004-9553-8387",
"affiliation": "Independent researcher, Australia",
"name": "James, Matthew"
},
{
"orcid": "https://orcid.org/0000-0002-5407-4297",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Stassen, Christian"
}
],
"license": "Apache-2.0",
"title": "PyEarthTools: Machine learning for Earth system science",
"keywords": ["modelling", "geoscience", "earth system science"]

"creators": [
{
"orcid": "https://orcid.org/0009-0008-2024-1967",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Leeuwenburg, Tennessee"
},
{
"orcid": "https://orcid.org/0009-0009-3207-4876",
"affiliation": "Work undertaken while at the Bureau of Meteorology, Australia",
"name": "Cook, Harrison"
},
{
"orcid": "https://orcid.org/0000-0003-2553-0721",
"affiliation": "NIWA - The National Institute of Water and Atmospheric Research, New Zealand",
"name": "Rio, Maxime"
},
{
"orcid": "https://orcid.org/0000-0001-6825-3854",
"affiliation": "University of New South Wales Sydney, Australia",
"name": "Hobeichi, Sanaa"
},
{
"orcid": "https://orcid.org/0000-0002-3148-4496",
"affiliation": "Met Office, United Kingdom",
"name": "Miller, Joel"
},
{
"orcid": "https://orcid.org/0000-0002-6569-1178",
"affiliation": "NIWA - The National Institute of Water and Atmospheric Research, New Zealand",
"name": "Mason, Gemma"
},
{
"orcid": "https://orcid.org/0009-0002-7406-7438",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Ramanathan, Nikeeth"
},
{
"orcid": "https://orcid.org/0009-0001-6250-3987",
"affiliation": "Met Office, United Kingdom",
"name": "Pill, John"
},
{
"orcid": "https://orcid.org/0000-0003-3430-2456",
"affiliation": "Met Office, United Kingdom",
"name": "Haddad, Stephen"
},
{
"orcid": "https://orcid.org/0000-0002-5407-4297",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Stassen, Christian"
},
{
"orcid": "https://orcid.org/0000-0002-1975-0042",
"affiliation": "Bureau of Meteorology, Australia",
"name": "de Burgh-Day, Catherine"
},
{
"orcid": "https://orcid.org/0000-0002-6799-9109",
"affiliation": "Bureau of Meteorology, Australia",
"name": "Holmes, Ryan"
},
{
"orcid": "https://orcid.org/0009-0009-8268-8358",
"affiliation": "Independent researcher",
"name": "Potokina, Margarita"
},
{
"orcid": "https://orcid.org/0009-0001-4486-0120",
"affiliation": "Independent researcher, Vietnam",
"name": "Bogacheva, Jenya"
},
{
"orcid": "https://orcid.org/0009-0004-9553-8387",
"affiliation": "Independent researcher, Australia",
"name": "James, Matthew"
},
{
"orcid": "https://orcid.org/0009-0000-5048-4240",
"affiliation": "Met Office, United Kingdom",
"name": "Sullivan, Ben"
}
],
"license": "Apache-2.0",
"title": "PyEarthTools: Machine learning for Earth system science",
"keywords": ["modelling", "geoscience", "earth system science"]
}
37 changes: 24 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@
|![](https://pyearthtools.readthedocs.io/en/latest/_images/notebooks_demo_FourCastNeXt_Inference_9_1.png)<br>A weather prediction from a model trained with PyEarthTools.|![](https://pyearthtools.readthedocs.io/en/latest/_images/notebooks_tutorial_Working_with_Climate_Data_14_2.svg)<br>A data processing flow composed for working with climate data.|
|:-:|:-:|

Source Code: [github.com/ACCESS-Community-Hub/PyEarthTools](https://github.com/ACCESS-Community-Hub/PyEarthTools)
Documentation: [pyearthtools.readthedocs.io](https://pyearthtools.readthedocs.io)
Tutorial Gallery: [available here](https://pyearthtools.readthedocs.io/en/latest/notebooks/Gallery.html)
New Users Guide: [available here](https://pyearthtools.readthedocs.io/en/latest/newuser.html)
Source Code: [github.com/ACCESS-Community-Hub/PyEarthTools](https://github.com/ACCESS-Community-Hub/PyEarthTools)
Documentation: [pyearthtools.readthedocs.io](https://pyearthtools.readthedocs.io)
Tutorial Gallery: [available here](https://pyearthtools.readthedocs.io/en/latest/notebooks/Gallery.html)
New Users Guide: [available here](https://pyearthtools.readthedocs.io/en/latest/newuser.html)

**If you use `PyEarthTools` for your work or a publication, [please city our work](https://pyearthtools.readthedocs.io/en/latest/#acknowleging-or-citing-pyearthtools).**

## Installation

Expand Down Expand Up @@ -53,7 +55,7 @@ PyEarthTools is a Python framework containing modules for:
- defining machine learning (ML) models;
- training ML models and managing experiments;
- performing inference with ML models;
- and evaluating ML models.
- and evaluating ML models (coming soon).

## Overview of the Packages within PyEarthTools

Expand All @@ -74,11 +76,11 @@ PyEarthTools comprises multiple sub-packages which can be used individually or t

If you use PyEarthTools for your work, we would appreciate you citing our software as below:

Cook, H., Leeuwenburg, T., Rio, M., Miller, J., Mason, G., Ramanathan, N., Pill, J., Haddad, S., & de Burgh-Day, C. (2025). PyEarthTools: Machine learning for Earth system science (0.1.1). Zenodo. https://doi.org/10.5281/zenodo.15760769
Cook, H., Leeuwenburg, T., Rio, M., Miller, J., Mason, G., Ramanathan, N., Pill, J., Haddad, S., de Burgh-Day, C., Sullivan, B., Hobeichi, S., Holmes, R., Potokina, M., Bogacheva, J., James, M., & Stassen, C. (2025). PyEarthTools: Machine learning for Earth system science (0.4.0). Zenodo. https://doi.org/10.5281/zenodo.17429589

BibTeX:
```
@software{cook_2025_15760769,
@software{cook_2025_17429589,
author = {Cook, Harrison and
Leeuwenburg, Tennessee and
Rio, Maxime and
Expand All @@ -87,13 +89,22 @@ BibTeX:
Ramanathan, Nikeeth and
Pill, John and
Haddad, Stephen and
de Burgh-Day, Catherine},
title = {{PyEarthTools: Machine learning for Earth system science}},
month = jun,
de Burgh-Day, Catherine and
Sullivan, Ben and
Hobeichi, Sanaa and
Holmes, Ryan and
Potokina, Margarita and
Bogacheva, Jenya and
James, Matthew and
Stassen, Christian},
title = {PyEarthTools: Machine learning for Earth system
science
},
month = oct,
year = 2025,
publisher = {Zenodo},
version = {0.1.1},
doi = {10.5281/zenodo.15760769},
url = {https://doi.org/10.5281/zenodo.15760769}
version = {0.4.0},
doi = {10.5281/zenodo.17429589},
url = {https://doi.org/10.5281/zenodo.17429589},
}
```
5 changes: 0 additions & 5 deletions docs/api/data/data_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,6 @@
:members:

.. autofunction:: pyearthtools.data.transforms.coordinates.get_longitude
.. autofunction:: pyearthtools.data.transforms.coordinates.weak_cast_to_int

.. autoclass:: pyearthtools.data.transforms.coordinates.StandardLongitude
:members:
Expand All @@ -341,12 +340,8 @@
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.Drop
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.Flatten
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.Expand
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.SelectFlatten
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.Assign
:members:
.. autoclass:: pyearthtools.data.transforms.coordinates.Pad
Expand Down
1 change: 0 additions & 1 deletion docs/api/data/data_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,6 @@ The rest of this page contains reference information for the components of the D
| | | - [attributes.SetType](data_api.md#pyearthtools.data.transforms.attributes.SetType) |
| | | - [attributes.Rename](data_api.md#pyearthtools.data.transforms.attributes.Rename) |
| | | - [coordinates.get_longitude](data_api.md#pyearthtools.data.transforms.coordinates.get_longitude) |
| | | - [coordinates.weak_cast_to_int](data_api.md#pyearthtools.data.transforms.coordinates.weak_cast_to_int) |
| | | - [coordinates.StandardLongitude](data_api.md#pyearthtools.data.transforms.coordinates.StandardLongitude) |
| | | - [coordinates.ReIndex](data_api.md#pyearthtools.data.transforms.coordinates.ReIndex) |
| | | - [coordinates.StandardCoordinateNames](data_api.md#pyearthtools.data.transforms.coordinates.StandardCoordinateNames) |
Expand Down
2 changes: 2 additions & 0 deletions docs/api/pipeline/pipeline_api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@
:members:
.. autoclass:: pyearthtools.pipeline.modifications.TemporalRetrieval
:members:
.. autoclass:: pyearthtools.pipeline.modifications.TemporalWindow
:members:
.. autoclass:: pyearthtools.pipeline.modifications.idx_modification
:members:
```
Expand Down
1 change: 1 addition & 0 deletions docs/api/pipeline/pipeline_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ The rest of this page contains reference information for the components of the P
| | | - [TimeIdxModifier](pipeline_api.md#pyearthtools.pipeline.modifications.TimeIdxModifier) |
| | | - [SequenceRetrieval](pipeline_api.md#pyearthtools.pipeline.modifications.SequenceRetrieval) |
| | | - [TemporalRetrieval](pipeline_api.md#pyearthtools.pipeline.modifications.TemporalRetrieval) |
| | | - [TemporalWindow](pipeline_api.md#pyearthtools.pipeline.modifications.TemporalWindow) |
| | | - [idx_modification](pipeline_api.md#pyearthtools.pipeline.modifications.idx_modification) |
| `pipeline.operations` | | - [Transforms](pipeline_api.md#pyearthtools.pipeline.operations.Transforms) |
| `pipeline.operations.xarray` | | - [Compute](pipeline_api.md#pyearthtools.pipeline.operations.xarray.Compute) |
Expand Down
35 changes: 23 additions & 12 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@
<figcaption>A data processing flow composed for working with climate data.</figcaption>
</figure>

Source Code: [github.com/ACCESS-Community-Hub/PyEarthTools](https://github.com/ACCESS-Community-Hub/PyEarthTools)
Documentation: [pyearthtools.readthedocs.io](https://pyearthtools.readthedocs.io)
Tutorial Gallery: [available here](./notebooks/Gallery)
Source Code: [github.com/ACCESS-Community-Hub/PyEarthTools](https://github.com/ACCESS-Community-Hub/PyEarthTools)
Documentation: [pyearthtools.readthedocs.io](https://pyearthtools.readthedocs.io)
Tutorial Gallery: [available here](./notebooks/Gallery)

**If you use `PyEarthTools` for your work or a publication, [please city our work](https://pyearthtools.readthedocs.io/en/latest/#acknowleging-or-citing-pyearthtools).**

## Installation

Expand Down Expand Up @@ -71,7 +73,7 @@ PyEarthTools is a Python framework containing modules for:
- defining machine learning (ML) models;
- training ML models and managing experiments;
- performing inference with ML models;
- and evaluating ML models.
- and evaluating ML models (coming soon).

## Overview of the Packages within PyEarthTools

Expand All @@ -95,11 +97,11 @@ If you use PyEarthTools for your work, we would appreciate you citing our softwa

:::::{tab-set}
::::{tab-item} APA
Cook, H., Leeuwenburg, T., Rio, M., Miller, J., Mason, G., Ramanathan, N., Pill, J., Haddad, S., & de Burgh-Day, C. (2025). PyEarthTools: Machine learning for Earth system science (0.1.1). Zenodo. https://doi.org/10.5281/zenodo.15760769
Cook, H., Leeuwenburg, T., Rio, M., Miller, J., Mason, G., Ramanathan, N., Pill, J., Haddad, S., de Burgh-Day, C., Sullivan, B., Hobeichi, S., Holmes, R., Potokina, M., Bogacheva, J., James, M., & Stassen, C. (2025). PyEarthTools: Machine learning for Earth system science (0.4.0). Zenodo. https://doi.org/10.5281/zenodo.17429589
::::
::::{tab-item} BibTeX
```
@software{cook_2025_15760769,
@software{cook_2025_17429589,
author = {Cook, Harrison and
Leeuwenburg, Tennessee and
Rio, Maxime and
Expand All @@ -108,14 +110,23 @@ Cook, H., Leeuwenburg, T., Rio, M., Miller, J., Mason, G., Ramanathan, N., Pill,
Ramanathan, Nikeeth and
Pill, John and
Haddad, Stephen and
de Burgh-Day, Catherine},
title = {{PyEarthTools: Machine learning for Earth system science}},
month = jun,
de Burgh-Day, Catherine and
Sullivan, Ben and
Hobeichi, Sanaa and
Holmes, Ryan and
Potokina, Margarita and
Bogacheva, Jenya and
James, Matthew and
Stassen, Christian},
title = {PyEarthTools: Machine learning for Earth system
science
},
month = oct,
year = 2025,
publisher = {Zenodo},
version = {0.1.1},
doi = {10.5281/zenodo.15760769},
url = {https://doi.org/10.5281/zenodo.15760769}
version = {0.4.0},
doi = {10.5281/zenodo.17429589},
url = {https://doi.org/10.5281/zenodo.17429589},
}
```
::::
Expand Down
4 changes: 2 additions & 2 deletions docs/newuser.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# New Users Guide

Welcome new user! This document will continue to be updated based on user feedback. This table quickly explains how to get things done in `PyEarthTools`
Welcome new user! This document will continue to be updated based on user feedback. This table quickly explains how to get things done in `PyEarthTools`
| Step | Without PyEarthTools | With PyEarthTools |
|----------------|-----------------------------|----------------------|
| Obtaining and loading data | Manual download or open from disk + data cleaning | Use or adapt in-built fetchers and openers (no cleaning) |
| Process, subset, augment, tranform and normalise data | Custom code | Use pre-defined validated Pipelines |
| Present your data to PyTorch or another framework as a Python iterator | Custom code to iterate over data | Pipelines are iterators |
| Define a machine learning model | Clone someone's repo or define your own | Use a 'bundled model' or write your own |
| Denormalise model outputs | Manual code | Pipelines are reversible |
| Model evaluation | Use a separate framework | Use pre-defined evaluation to generate standard scorecards |
| Model evaluation | Use a separate framework | (coming soon) Use pre-defined evaluation to generate standard scorecards |

This approach also allows your research to be more targetted into varying only the part of the end-to-end process which you are investigating. By starting with a baseline implementation to provide a strong basis for comparison and modifying only the relevant step, you can undertake a more controlled investigative process, confidently generating results from experimentation along the way.

Expand Down
Loading