[not an issue] Tutorial notes

[nikeethr]

I'm just using this as a placeholder to track my notes while going through tutorials, if you know the solutions to some of the questions/issues I'm facing, kindly help me along...

probably @tennlee is best in the first instance since I'm doing most of this in gadi.

Setup

• gadi @ NCI
• pixi for environment installation
• ARE jupyterlab
• singularity + nix-shell + justfiles for reproducible environment setup
  • no need to dwell on this too much.
  • it keeps my environments neat and tidy and removes things/squashes them appropriately for re-use.
  • a thing to be aware of is that: not everything in GADI software-wise will be available in this setup.
  • I don't see that as a bad thing (actually the opposite).

I will log any observations in the comments as I go through the tutorials.

[nikeethr]

cannot find site_archive_nci (Interfacing_to_Data_at_NCI.ipynb)

I know a similar thing needed to be loaded in the old EDIT. The way I found that is to grep the codebase. In this instance I couldn't find this import, but I did find a similar pyearthtools_archives_NCI - unfortunately gitlab is down and I can't access it.

The rest of the tutorial does seem to work.

[nikeethr]

These seem fine:

• Download ERA5 data
• Accessing ERA5 data

[tennlee]

This will only work using the NCI dk92 environment, where the nci data archives are also installed. The data archive accessors are not a part of PyEarthTools because they are specific to the NCI facility and we are working with them to determine if it is suitable for bundling into the main repository. It's best to set up a time to meet to discuss this one I think.

[tennlee]

In leiu of the NCI site archive, many of the tutorials will continue to function using a cloud-based retrieval of the data, however this is much slower and less reliable.

[nikeethr]

Ah okay no worries. let’s talk about how to approach loading those archives the way it’s currently implemented on NCI. I’m still going through the tutorials - they mostly seem fine otherwise.

Most of my thoughts/questions are actually to do with the pipeline design (internal code - wrapper code around external libraries etc) - these aren’t really the purpose of the tutorials so it’s not a priority.

In any case, since I’ve observed them anyway I will show a couple of examples and pose some design questions. With the caveat that these don’t need to be addressed right now; more that if addresses will simplify code a lot.

[nikeethr]

cannot find site_archive_nci (Interfacing_to_Data_at_NCI.ipynb)

This issue is manually resolved by downloading an data archive "adapter" that is internal to NCI - note that it seems to have legacy naming conventions - but otherwise it works fine.

I believe it is in discussion of whether or not this adapter can just be included publicly

• makes it easier to install for people working in gadi.
• it also means it'll be more easy to test if any updates break compatibility with adapter

For now would it be worth including the location of this adapter in the tuts or alternatively pointing people to the dp9 option?

[nikeethr]

pipelines

I'll preface by saying that I think the strength of PET so far is its ability to access data seamlessly in a consolidated way.

However, the design and purpose of pipelines is less obvious to me.

Again, having a way to load and save a pipeline in a semi-readable document is desirable, but this is in my opinion independent to the implementation of the pipeline itself.

My assertion is not that any of these need to be fixed immediately, they are just opinions and my personal preferences.

Basically the questions that come up are:

1. What is the advantage of having our own pipelining tool? DAGs are not a unique concept and saving their graphs and information is also quite trivial. There are open source alternatives:

   • dask (this is already a core part of PET, so we aren't actually adding anything new)
   • airflow (highly popular apache project that will probably exist for the forseeable future)
   • celery/any job scheduler (again a backbone for a lot of python backends)
   • in fact since the code is doing this in a single threaded way for the most part - a lot of this can just be done using reduce, map, filter, itertools and other inbuilt functions, that essentially do "pipelining".

   Having our own pipeline implementation adds risk and additional maintenance burden, using something existing means we can rely on a larger open source community to maintain the pipeline and only need to worry about how we should save and load the pipeline.

2. I'm not fully convinced that an interface is needed for computational libraries like xarray. e.g. pet.transformer.Xarray.do_something(...) (paraphrasing). I think xarray.do_something directly is preferable. By wrapping it in a stage, the stage is a structural layer that compatibility of input-output and I think this is sufficient.

   Again having to maintain API compatibility between xarray, numpy, dask etc. and PET is additional burden. With things like typeguards and runtime guards, its easier now to directly apply operations. The benefit to having a explicitly implemented interface is if it serves a specific use case - e.g. if we want to lump multiple stages together and ship it as a pattern - because its common to do so. IMO this is more useful than having interfaces to atomic operations which can just be called directly instead.

3. Referential transparency. Right now its hard to tell if the PET pipeline actually keeps state static between stages. Ideally if we replace any one stage with its computed result - the pipeline should not change - roughly what I mean by referential transparency. You don't need immutable state for this, but immutable state does help in making things more predictable.

   This actually goes back to 1. and 2. airflow for example propagates "task information" and "dag information" between stages which I believe is the main immutable (? to the extent that it is possible in python) representation of state. While it may seem like this is a trivial thing to have - it is actually quite complicated to get right.

The common theme in the above is a focus on maintainability and extensibility. Right now there is a heavy requirement in getting tests running and extensibility would require increased effort to satisfy these tests. I would strongly suggest considering other options for implementing pipelining and transformer operations, while we can still retain the ability and focus on saving and load pipelines appropriately and compatibility between stages in a more lightweight structure.

[tennlee]

Thanks for your feedback, that's very much appreciated. In terms of the general theme, I think they're all good points. Some of them have partial answers and of course the original decisions were made at a point in time and things might have moved on since then also. I'll try to organise some software design meetings/meetups where we can work through the points and prioritise some of the software maintenance work. I also wonder if it would be worth creating some software design documentation which discusses what design decisions have been made and why, so that it's more clear what was done as a goal vs what is just a means to an end in the framework.

[nikeethr]

Absolutely - my hope is that these are points that can help start conversations rather than closed recommendations. An exercise in documenting software design decisions would expose any holes, conversely justify uncertainties.

There are just a couple of other things (e.g. disabling dask/crashes) I noticed in the tutorial that I will comment on shortly. Again this doesn't directly impact the tutorial, more to do with the underlying software and system restrictions.

But I'll finish up the rest of the tutorials before commenting - since most of it seems fine.

[nikeethr]

Issue with StandardLongtitude that we briefly discussed is already being addressed here: #73