PytestAssertRewriteWarning due to runpytest and pytest_plugins

[McSinyx]

Originally posted by @McSinyx in #1275 (comment)

On Python 3.11 on Guix (full test log), PytestAssertRewriteWarning resurfaces for the following tests due to 18afc9d replacing runpytest with runpytest_subprocess:

tests/markers/test_class_scope.py::test_standalone_test_does_not_trigger_warning_about_no_current_event_loop_being_set
tests/markers/test_function_scope.py::test_warns_when_scope_argument_is_present
tests/markers/test_module_scope.py::test_standalone_test_does_not_trigger_warning_about_no_current_event_loop_being_set
tests/markers/test_session_scope.py::test_standalone_test_does_not_trigger_warning_about_no_current_event_loop_being_set
tests/test_event_loop_fixture.py::test_event_loop_fixture_handles_unclosed_async_gen
tests/test_event_loop_fixture.py::test_event_loop_fixture_asyncgen_error

IIUC this commit tries to workaround a case in GH-1227 where a system pytest-asyncio package takes precedence over a local one. Our fix for Guix also involves removing pytest_plugins = 'pytest_asyncio' in the following tests because it also marks the package for assertion rewriting while it's already imported before executing the scripts as part of plugin autodiscovery.

tests/test_event_loop_fixture.py::test_event_loop_fixture_handles_unclosed_async_gen
tests/test_event_loop_fixture.py::test_event_loop_fixture_asyncgen_error

[seifertm]

So this means you have to revert 18afc9d for pytest-asyncio tests to run successfully on Guix and additionally get rid of pytest_plugins = 'pytest_asyncio'?

That's regrettable. I think downstream packagers should be able to run the tests without major issues. I'm not entirely sure how to get there, though.

Do you have a suggestion how this can be addressed so it works on Guix, FreeBSD (as in the linked issue) and the other systems?

[McSinyx]

So this means you have to revert 18afc9d for pytest-asyncio tests to run successfully on Guix and additionally get rid of pytest_plugins = 'pytest_asyncio'?

FWIW we don't have to revert the entirety of 18afc9d, only for the 6 mentioned tests. What effect does removing pytest_plugins = 'pytest_asyncio' might have though?

Do you have a suggestion how this can be addressed so it works on Guix, FreeBSD (as in the linked issue) and the other systems?

I still don't have a clear insight into the environment to reproduce GH-1227, i.e. why the package is not built in an isolated environment (e.g. a chroot) and even when it's not, how PYTHONPATH is constructed in a way that the system package is found before the local one. Cc: @yurivict for FreeBSD details.

I think if the goal is only compatibility with Guix's and FreeBSD's known issues, assert_outcomes could add the count of PytestAssertRewriteWarning to the number of warnings, but (1) that's really ugly and (2) we don't know why such warning only pops up on these specific tests and whether there will be more.

Where do I find the order Pytest import things to maybe find the root cause to the assertion rewrite issue, i.e. why does 81c5032 work?