Integrate openpmd-scipp into the main API

[franzpoeschel]

This merges the code from https://github.com/pordyna/openpmd_scipp into the main API such that it can be used by series.to_scipp().

TODO:

• Documentation (fields)
• Documentation (particles) COMMENT: The TODO there refers to particles not being even implemented yet, I think?
• Clean up the example
• Have a look at the rest of what is in the repository, see what needs to / can be transferred
• CI integration
• Licensing headers
• remove Jupyter Notebook
• check image sizes

[pordyna]

I just took a quick look. This looks quite nice and wouldn't change much of the interface. The other stuff in the repo is mostly just for creating the README and testing, formatting etc. so I guess the only thing you would need to take over are the requirements, but this is only numpy and scipp. Also, it would make sense to make the example an integration test. Running the Jupyter notebook without verifying any outputs was also the only test I had.

[franzpoeschel]

so I guess the only thing you would need to take over are the requirements, but this is only numpy and scipp

Also the Readme; I will try to integrate that into the Readthedocs page.
For the requirements, note that the code in this PR loads scipp lazily, so the main API can still be used without having that installed. numpy is already a hard requirement, but scipp should be at most an optional requirement (if there is a way to tell that to the Python packaging).

Also, it would make sense to make the example an integration test. Running the Jupyter notebook without verifying any outputs was also the only test I had.

I'd suggest that we add a test separately as part of APITest.py. The "target audience" for the examples are users who can use them as guidelines for their own code, so I would not clutter the example with testing code.

[ax3l]

Is the notebook important to add or does the .py script + README have the same details?

[pordyna]

@ax3l the README from my repo is identical to the notebook since I was generating my README.md from it with:

rm -rf README_files
python -m nbconvert --to notebook --execute README.ipynb --output README.executed.ipynb
python -m nbconvert --to markdown README.executed.ipynb --output README.md - TagRemovePreprocessor.enabled=True --TagRemovePreprocessor.remove_cell_tags remove_cell
rm -f README.executed.ipynb

so you don't need both

[franzpoeschel]

I added both since having this documentation page alternatively as an executable and interactive Notebook would be a useful extra for learning how to use this. It's of course not ideal for maintenance..

[ax3l]

Can you potentially upload the four .svg images to a public GH gist and use them by URL? That way, we do not add larger binary artifacts to our git repo here, which keeps it small on git clone for CMake superbuilds.

[franzpoeschel]

SVG files are not binary, these are text files with each ~800 lines, less than many of our source files.
I tried putting them on a Gist, but it seems that Github does not hand out usable raw links.

[ax3l]

@pordyna for openPMD-api, we use the LGPL license. Please confirm we can change the license for files in this PR and then please update the headers accordingly @franzpoeschel .

[pordyna]

yes sure go for it!

[pordyna]

Btw. when it comes to handling particles, I have an idea on what would make sense there, but haven't implemented anything yet. Since with particle data people are mostly just creating histograms, I think reading a species as a DataArray of weighting with all other attributes as coordinates would work well with the workflows described here: https://scipp.github.io/user-guide/binned-data/binned-data.html. This should be probably a separate PR though.

[pordyna]

And the particle part could be, probably, quite easily implemented with https://scipp.github.io/user-guide/reading-and-writing-files.html#Using-pandas. Just load to a data frame with to_df() first and use this to convert it.

[ax3l]

Suggested change

	Series.to_scipp = series_to_scipp # noqa
	Series.to_scipp = series_to_scipp # noqa

[pordyna]

@franzpoeschel I discovered some problems with the implementation. I already fixed them, but we need to bring the changes here. For that, I just want to update my docstrings first.

What was wrong:

• the relay functionality was not working
  • the full mesh array was always allocated -> I rewrote the DataRelay class
• there was an extra copy when loading data, since sc.array always copies values -> switched to sc.empty + load_chunk with preallocated buffer
• the isclose contiguity check was not good enough, using sc.islinspace instead.

[franzpoeschel]

@franzpoeschel I discovered some problems with the implementation. I already fixed them, but we need to bring the changes here. For that, I just want to update my docstrings first.

What was wrong:

* the relay functionality was not working
  
  * the full mesh array was always allocated -> I rewrote the `DataRelay` class

* there was an extra copy when loading data, since `sc.array` always copies values -> switched to `sc.empty` + `load_chunk` with preallocated buffer

* the `isclose` contiguity check was not good enough, using `sc.islinspace` instead.

Hey @pordyna, can you implement those changes on a branch so I can pull them into this PR?

[pordyna]

Hey @pordyna, can you implement those changes on a branch so I can pull them into this PR?

Yes, I can branch from your branch and add the changes.