eCLM branch with full history from CLM5, CIME, FATES, MOSART

[jjokella]

Triggered by discussions with @DCaviedesV and @AGonzalezNicolas (additionally interesting for @kvrigor and @spoll ), I try to construct an eCLM-branch with the full history of all source files included: master-with-history-clean.

Goal

Have all the history of the source files of eCLM, all merged into the initial commit of eCLM, such that the eCLM-history since the initial commit is completely linear and identical to current master.

Workflow

Only to be tried out locally in a new clone/copy of eCLM!

Could possibly be made more stream-lined (e.g. two filter-repo commands turned into one).

Major points

• adding upstream history with replace command
• removing LFS files from the upstream history

    # 0. Add remotes and fetch branches/tags
    git remote add clm5 git@github.com:ESCOMP/CTSM.git
    git remote add cime git@github.com:ESMCI/cime.git
    git remote add fates https://github.com/NGEET/fates
    git remote add mosart https://github.com/ESCOMP/MOSART
    git fetch --all

    # 1. Create new clean branch from master-with-history
    git checkout -b master-with-history-clean master

    # 2. Apply graft to connect upstream history
    git replace --graft f6b981fe6 release-clm5.0 cime5.6.47 release-cesm2.0.04 sci.1.30.0_api.8.0.0

    # 3. Filter out LFS files - REMOVE the --refs parameter and add more file types
    rm -rf .git/filter-repo  # Clear any previous filter-repo state
    git filter-repo --force --invert-paths \
      --path-glob '*.nc' \
      --path-glob '*.pdf' \
      --path-glob '*.rda' \
      --path-glob '*.bin' \
      --path-glob '*.dat' \
      --refs master-with-history-clean

    # 4. Remove the graft replacement (now permanent in history)
    git replace -d f6b981fe6

    # 5. Optimize repository size
    git gc --aggressive --prune=now

    # 6. Uninstall LFS hooks and remove LFS config
    git filter-repo --force --blob-callback 'if len(blob.data) < 200 and blob.data.startswith(b"version https://git-lfs.github.com/spec/v1"): blob.skip()'

    #7. Check LFS
    git lfs migrate info --everything info

    # 8. Optimize repository size II
    git gc --aggressive --prune=now

    # 8. Checking that the blames contain the history
    git blame src/clm5/biogeochem/CNFUNMod.F90 | head -1
    git blame src/clm5/fates/biogeophys/EDSurfaceAlbedoMod.F90 | head -1
    git blame src/mosart/src/riverroute/RtmIO.F90 | head -1
    git blame src/eclm/prep_ice_mod.F90 | head -1
    git blame src/stub_comps/wav_comp_mct.F90 | head -1

[jjokella]

Relates to (and comes from discussions in) #95

[kvrigor]

As always I will play the devil's advocate and question the motives of this exercise 😈

• Who benefits from shared history? If the goal is to automate syncing changes from CLM5, then this could be done instead on a dedicated branch without tampering the history of eCLM master branch.
• If one wishes to investigate the history of some eCLM source file from its very origin, one could simply look it up directly from the source repo. I see the appeal of appending CLM5 history to eCLM, but IMO the gains are marginal that drowning eCLM master history with noisy commits is not worth it. It just makes filtering relevant commits painful.
• Adding CLM5 history seem to be a precursor for developing an automated CLM5-eCLM merge tool. Why create such a tool in the first place? Isn't manual sync every 1-N years more efficient and safe? Even on-demand integration of new features done manually would be safer and faster because:
  • An automated tool would still require manual updates to fix edge cases. Dev resources allocated in maintaining such tool could have been better used in understanding which new features are most relevant to us and our users. I think actually doing the merge is more productive than tinkering with an automated tool, since in fixing the tool one would have already understood the context necessary to do a manual merge. I would rather cherry-pick commits that has direct impact (e.g. updates to soil ice parameterizations) rather than mindlessly ingesting every new feature. Syncing with high-velocity CLM5 master branch and dealing with edge cases is, I think, devolves into busywork with nobody benefitting from it.
  • Manual merge might be unfashionable, but that's how one learns about the codebase. And understanding the context around a commit is hardly likely to be automated!
  • To be clear I'm not proposing to strictly do a 100% manual merge. Some level of automation is necessary (e.g. git merge has several options to control conflict resolution behavior). I'm just opposed to counterproductive automation (combined with questionable goals to make it worse).
• Will CLM5 developers really be receptive in upstream changes from us? A PR from an eCLM branch with prepended CLM5 history will always look odd w.r.t. CLM main branch, due to eCLM commits that are never in CLM5. If I were a CLM5 developer, I would be wary of a PR from some weird patched-up CLM5-like codebase whose author/s apparently didn't even take time to read and follow CLM5 contribution rules.

Please don't take the the sharp language personally 😜; hard questions need to be asked and efforts like these affect long-term stability of eCLM, so I'm quite critical to pouring huge, irreversible efforts that could potentially end up in vain in the long run.

[kvrigor]

Apologies for the previous wall of text; this is a culmination of the long, ongoing debate about CLM5-eCLM history and I just wanted to document my main points of contention as clearly as I could. To be clear I don't wish to hinder efforts if one is really willing to pursue this. My comments might sound discouraging, but I'd still encourage anybody who wishes to pursue this idea despite the potential pitfalls. Just consider yourselves warned 😜. I actually like how the master-with-history-clean was generated, especially the documentation of the exact commands used. Feel free to experiments on this branch. If I see the process less worse than I expected, I might change my stubborn mind 😅

[jjokella]

Please don't take the the sharp language personally

No offense taken - would not even have thought of the language as sharp and think it is important to make these points clearly as you did ;-)

• Who benefits from shared history?

My goal with this is actually not an automated merge-tool.

• Merging into CLM5: I think I am aligned with your view on merging into CLM5 that it could be smoothest to leave a manual component.

• Merging from CLM5/others: For merging changes from CLM5 (or others), I think the common history could actually help in the sense that the power of git merge and in particular of merge conflicts is immediately available. One just has to add CLM5 (MOSART, FATES, CIME) as remote to eCLM and merge a new version.

• Blame/contribution clarity: This is my main goal. I think it is a question of clarity that (if we can) source code lines in our repo should contain the information of who actually changed them last. By license and README, I think we are "legally" safe about this and anyone with the skill and time can look all the information up in the corresponding repos. However, having the history readily available in eCLM makes looking it up possible with almost no effort. In the end I think this is a trade-off for eCLM: Very concise history vs better lookup.

• eCLM-history: Keeping eCLM history clean and linear is important and I think this is where the proposed master-with history-clean is better than previous solutions. All history is merged in one merge commit and then, on top of that, we have the completely linear history of eCLM just as before! To illustrate: Running git log -n <#of eclm commits> reproduces the eCLM-history as before! To me this is kind of a "best of both worlds".

• Disadvantage of this proposal: To me the main disadvantage master-with-history-clean is that all PRs of eCLM would have to be re-generated based on this branch. It is not too much work though as it helps that we have a very clean, linear history and usually squash PRs.

All in all, I think that considering including the history is worth it. Still I agree with your post in the essential points (as I understand them): (1) eCLM history should be clean and linear and (2) merges to CLM5 are hard (and maybe not worth) to be automated.