Write an additional log file in PROCESS working directory

[timothy-nunn]

Writes an additional logfile in the PROCESS working directory (where the input file is) that follows the name of the input file:

• my.IN.DAT produces a my.process.log
• my_IN.DAT produces a my_process.log

This does not remove the old process.log file which is created in the terminal working directory.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 57.89474% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.93%. Comparing base (8a4d67c) to head (2c33138).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
process/main.py	57.89%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4035      +/-   ##
==========================================
- Coverage   46.30%   43.93%   -2.37%     
==========================================
  Files         123      123              
  Lines       28961    30881    +1920     
==========================================
+ Hits        13411    13569     +158     
- Misses      15550    17312    +1762     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[je-cook]

After thinking about what I said yesterday I agree with this change.

I think it would also be good to change the default file handler to append mode so you don't overwrite the log file.
change L727 to:
logging_file_handler = logging.FileHandler("process.log", mode="a")

alternatively you could rotate the logs somehow but that is more complex and possibly overkill (You could do a symlink process.log to a timed and dated filename)

[je-cook]

Will moving setup_loggers here not lead to logging not being properly initialised if someone runs process not using the CLI or this class but by direct call of SingleRun or VaryRun?

[timothy-nunn]

I think this has always been the case but you are correct

[timothy-nunn]

I have added a fix for this to this PR

[je-cook]

With this new layout do you get multiple single file file handlers eg if you run two single runs on the same prompt? There are other permutations of that we might need to guard against

[je-cook]

possibly this is not the case if basicConfig overwrites previous state?

[timothy-nunn]

eg if you run two single runs on the same prompt

As in you have a script that does something like this?:

run1 = SingleRun(...)
run1.run()

run2 = SingleRun(...)
run2.run()

[timothy-nunn]

I have fixed this problem by simplifying the setup_logger method. Instead of adding our loggers to the root logger, we only modify the process package logger. Each time we setup_logger we remove all existing handlers on that logger and add them back.

I have added in a way (not very nice) of disabling these loggers--expect for the model logger because this is required for PROCESS' error handing system--during tests so that running pytest does not write to process.log