Move README into documentation

[jjokella]

Adding all README sections to the corresponding documentation pages. Leaving the old documentation intact, but renaming section header with (old). Another PR could then work on actually merging the two documentations.

Immediate goal of this PR: Have a single source of truth for eCLM static file generator documentation.

Modelled after solution for https://github.com/HPSCTerrSys/eCLM_atm-forcing-generator

[mvhulten]

Sorry, I've seen this PR only after our interchange with Yohanes.

I still find having a doc/ and stuff worse from a simple README in the root. So I don't know what to do with this.

[jjokella]

No worries, yeah this PR is also a reaction to the mail by Yohanes.

I can see advantages of the README (very compact, easy to search, immediately visible on GitHub).

However, I favor having all documentations in our repos in the same style, thus a docs that generates a website. This does have also other advantages apart from the similarity to other repos, I would say (nice rendering and linking, packaging of information). Thus, it can hold possible future additions (like input-documentation, howtos, separate developer-documentation)

Let's discuss it in our meeting tomorrow.

[jjokella]

One more change toward the "single source of truth": I collapsed the "old"/"deprecated" documentation sections into clickable <details> sections.

This way a reader has to actively open these old parts.

This renders the contents of the visible docs on the website of this PR identical to the README in main. (except the all-in-one-file aspect...)

[mvhulten]

Moving this to docs does not help. It's still the same repo with the same content.
When I go into docs/ I find a README.md and I'd open this and find documentation building instructions that I do now want to do. first and I am happy, but I suppose this file cannot be maintained, so it gets out of sync with docs/user_guide/.
Some might open INDEX.md who are either not wiser if they want local documentation (or they are happy if they don't object using an internet connection and a web browser and rely on the availability of some generated documentation at a different domain to be available and up-to-date) or they have to click on another link when they are using a web browser (where they might have to click another link before being able to see the actual documentation; right now the situation is so bad that it links to very old documentation).
So one might see the subdirectory users_guide/ where there is a README.md as well. Opening this results in a nice Markdown file that one can open locally in a viewer or editor (next to another terminal in another tmux pane, e.g.) and go ahead. Oh, there are some HTML tags, but that's okay. Then at the end there is some syntax for a ToC that is non-standard Markdown (it is besides the point that this does not exactly exists), rather some extension. (On the github.com repo webpage there is an empty box.) Going back to my local view, typically on a terminal, e.g. because I'm on an HPC, I quit this file in some frustrataion. Just as I was about to inform the devs that the documentation is incomplete, I see the files ?_*_file.md and I can continue with the workflow by typing vim -R ?_*_file.md.

User is happy enough, but was more so if she could just open the README from the root.

Maintenance/development is another issue. You have to edit several files, apparently include some additional syntax and hope that the documentation website is automatically updated (points of failure: developer, GitHub runner, other domain, webserver &c. and internet connection).

[jjokella]

Moving this to docs does not help. It's still the same repo with the same content.

Idea: Have all documentation of a certain point together in one file. For example remapping in 2_create_mapping_file.md.

Problem before: There are two similar but divergent texts in README and one in docs/user_guide.

When I go into docs/ I find a README.md and I'd open this first and I am happy, but I suppose this file cannot be maintained, so it gets out of sync with docs/user_guide/.

I don't get this fully. The docs/REDAME.md tells the user how to build the webpage locally.

Local usage

If one cannot use the online webpage, there are two solutions:

• make the webpage locally using docs/README.md
• browse the local MarkDown files (this is what I do usually)

User is happy enough, but was more so if she could just open the README from the root.

At least the README from the root should contain directly the link to the docs.

Maintenance/development is another issue. You have to edit several files, apparently include some additional syntax and hope that the documentation website is automatically updated (points of failure: developer, GitHub runner, other domain, webserver &c. and internet connection).

So far this is working ok for the other repositories, I would say. It is still important to follow two guidelines in my opinion (that I think you will also like)

1. single source of truth (meaning the same thing will not be written in two different MarkDown files)
2. everything has to be in-repo and available as text files (MarkDown)

[mvhulten]

I erred about docs/README.md: I thought it was a copy of the root README, but this contains indeed instructions to build it.

When you want to build it, you need to use pip, which is ill adviced on at least some platforms. Having to use something beyond what is in the system including Lmod modules is something we try to avoid for all our workflows. That you'd need external tools just for documentation is worse. I can look at this, but see the ghp-import dependency and don't see how such package relates to documentation that should be buildable locally. Even if it were simpler, I don't think anyone should need to run even a single command just to see documentation.

That said, within the context of wanting a nicely integrated, nicely rendered documentation of all the workflows, I see your points and maybe some compromises need to be made. If on top of this some minor tweaking is done, it could be usable also in a repo-contained/local way.

[jjokella]

I agree with everything you write here^^

Two points:

Regarding pip, we could think about more robust solutions in the future. However, not a top-priority I would say.

Regarding locally checking out the docs: I myself usually don't build the doc (only when I want to check the style), but rather use the MarkDown-sources, which should be nicely organised to be usable without any build-commands (for example searching for a variable-name or similar in the repo, you will get the source code and the MarkDown files that contain the searched keyword).

[kvrigor]

When you want to build it, you need to use pip, which is ill adviced on at least some platforms. Having to use something beyond what is in the system including Lmod modules is something we try to avoid for all our workflows.

Docs build is typically done on your local machine, not on an HPC cluster. The latter is possible but I don't think it makes sense to use HPC resources for mere document generation. Also users aren't expected to build the docs; that task is reserved for maintainers and users should only refer the final rendered form which is online. Case in point: one usually accesses help pages for some tool X via a man page or even online, but it's highly unlikely that one would download tool X's source codes just to read its embedded documentation.

[kvrigor]

My 2¢: generally I also tend to write monolithic READMEs, but such style isn't always suitable for public-facing docs. Public documentation should cater to a wide range of users; thus the most important consideration is how well our docs read to outside and/or novice users. Basically, "public readability preferences" trumps "personal readability preferences". To me the current README is long and dense: steps+substeps go across multiple, medium-sized paragraphs, and made more complicated with "if condition X holds, then do Y (otherwise Z)" sentence constructions. I wouldn't be surprised if most can't maintain patience with our current style of writing READMEs. We in DETECT-Z are outliers in the sense that we take RTFM seriously. Unfortunately this quality is not shared by the larger public, and it isn't really their fault.

I think a dedicated docs/ folder following a clear documentation structure is the way. This enables doc build tools (e.g. jupyter-book, sphinx-build, pandoc etc.) is to render raw docs into a final form suitable for consumption. The same set of docs/ could be rendered into different forms: online docs for novice users, single-page TXT file for power users, and a PDF for me 😜.

We can still keep the current README along with the docs/ if it satisfies both camps. The only issue is duplication. I don't have any preferences regarding this.

[jjokella]

Thanks @kvrigor , you make the point clearer that is also implicit in my goals - the documentation for the "public user". But also the expert, like Marco, should of course have a nice interface to the doc!

For me, I do have a strong opinion against duplication, though. In particular, confusion by an outside user due to the current duplication of docs, was the main motivation to start this PR^^

[mvhulten]

I agree with most of this, but not the catering to "public users". People who cannot view and read a longish text file and rather need a fancy interface are not our users. The coupled model can only be run on HPC and there users who don't understand computers are a problem.

What I don't understand is why there would not be a way to parse all README.md from (a subset of) repos in our HPSCTerrSys group and automatically build a nice documentation of this. Markdown is not perfect or standardised, but we do use sections and paragraphs pretty correctly/consistently throughout the repos, so it should be able to build pretty documentation of this. You don't even need runners connected to the different repos. Maybe there is already a nice tool for this. I have not searched for it.

Apropos, using pandoc on the original READMEs works perfectly; I don't even know how to use pandoc in the new multi-file structure.

[jjokella]

@kvrigor @mvhulten
I think it is good that we now discussed here on READMEs and there are some points that can be turned into general issues / discussion points.

However, now I would like to wrap up this PR.

So I am re-asking you for review. Please then either add technical review points or (in the case of Marco I guess^^) indicate that you would like to block this PR. In this case we move the decision about the PR to higher authorities, I guess.

[mvhulten]

Discuss in meeting first.

[kvrigor]

People who cannot view and read a longish text file and rather need a fancy interface are not our users. The coupled model can only be run on HPC and there users who don't understand computers are a problem.

Controversial view—I hold a completely opposite opinion

What I don't understand is why there would not be a way to parse all README.md from (a subset of) repos in our HPSCTerrSys group and automatically build a nice documentation of this.

It's definitely possible. One only has to set time aside to build a doc generator tool tailored to our needs. This could be actually a good task for a HiWi/intern.

[mvhulten]

Earlier this afternoon we discussed and agreed that this merge is okay (with my additional patches).

My last commit is not ideal as the two sort of additional commands would not be in the github.io documentation.

As a fourth option, one could generate a PDF as well with

cat ?_*.md | pandoc --pdf-engine=xelatex -o manual.pdf

but even with the xelatex or lualatex there are lines that fall off the page, so I left that out. Later one might take care of that with \ for unwrappable code and URLs.