[cppyy] Review xfail'ed tests #20906

guitargeek · 2026-01-15T20:36:38Z

All xfail'ed cppyy tests have been reviewed, and the ones that are
actually failing are now marked as xfail(strict=True), so the test
fails if it unexpectedly passes.

This gives us a very useful baseline for what works now in ROOT and what
doesn't, and we'll learn when our developments like the cppyy upgrade on
the CppInterOp-based version will fix some tests.

There are some tests remaining that can't be run because they crash, and
therefore the strict mode doesn't make sense there. But the number of
these tests is not high, and now we always give a reason for why they
crash:

git grep xfail | grep -v "strict"

test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because TClingCallFunc generates wrong code")
test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because the interpreter emits too many warnings")
test_concurrent.py:    @mark.xfail(run=False, reason="segmentation violation")
test_cpp11features.py:    @mark.xfail(run=False, reason = "Crashes")
test_datatypes.py:    @mark.xfail(run=False, reason="segmentation violation")
test_datatypes.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")
test_doc_features.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit")
test_fragile.py:    @mark.xfail(run=False, condition=has_asserts(),
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, condition=is_modules_off(), reason="Crashes on build with modules off: Fatal Python error: Segmentation fault")
test_lowlevel.py:    @mark.xfail(run=False, condition=IS_WINDOWS, reason="Windows fatal exception: access violation")
test_regression.py:    @mark.xfail(run=WINDOWS_BITS != 64, condition=IS_MAC_ARM | WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit and fails macOS ARM with" \
test_stltypes.py:    @mark.xfail(run=False, reason="Fatal Python error: Segmentation fault")
test_stltypes.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason="Crashes on Windows 64 bit")
test_templates.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")

A future development could be to ensure in Cling or cppyy that these
tests at least don't crash but fail gracefully, but that's better to be
done after the cppyy upgrade because some tests might be fixed anyway.

Some tests remain skipped for good reasons:

git grep "mark\.skip"

test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:    @mark.skipif(noboost, reason="boost not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_fragile.py:    @mark.skip(reason="This test is very verbose since it sets gDebug to True")
test_fragile.py:    @mark.skip(reason="Not actually a cppyy test")
test_fragile.py:    @mark.skipif(not has_cpp_20(), reason="std::span requires C++20")
test_leakcheck.py:@mark.skipif(nopsutil == True, reason="module psutil not installed")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_eigen.py:    @mark.skipif(eigen_path is None, reason="Eigen not found")
test_pythonify.py:    @mark.skip(reason="Garbage collection tests are fragile")
test_regression.py:    @mark.skip(reason="For ROOT, we don't enable AVX by default ('-mavx' is not passed to Cling)")
test_stltypes.py:@mark.skipif(not has_cpp_20(), reason="std::span requires C++20")

Sorry that these git grep commands don't show the names of the tests!
They are to illustrate how many tests are still marked as skip/xfail
without strict=True.

Closes #20085.

github-actions · 2026-01-16T05:42:25Z

Test Results

22 files 22 suites 3d 11h 21m 54s ⏱️
3 812 tests 3 812 ✅ 0 💤 0 ❌
76 684 runs 76 684 ✅ 0 💤 0 ❌

Results for commit ae95b68.

♻️ This comment has been updated with latest results.

bindings/pyroot/cppyy/CPyCppyy/src/Pythonize.cxx

vepadulano · 2026-01-16T13:57:57Z

bindings/pyroot/cppyy/cppyy/test/test_fragile.py

+def has_cpp_20():
+    import cppyy
+
+    return cppyy.gbl.gInterpreter.ProcessLine("__cplusplus;") >= 202002


How does this work? ProcessLine is returning an integer corresponding to the value of __cplusplus?

That's correct. I got this pattern from the upstream cppyy test suite:
https://github.com/wlav/cppyy/blob/92967c195e47361dc04a6f1b0a77cfc745f4d914/test/test_stltypes.py#L1673

All xfail'ed cppyy tests have been reviewed, and the ones that are actually failing are now marked as `xfail(strict=True)`, so the test fails if it unexpectedly passes. This gives us a very useful baseline for what works now in ROOT and what doesn't, and we'll learn when our developments like the cppyy upgrade on the CppInterOp-based version will fix some tests. There are some tests remaining that can't be run because they crash, and therefore the `strict` mode doesn't make sense there. But the number of these tests is not high, and now we always give a reason for why they crash: ```txt git grep xfail | grep -v "strict" ``` ```txt test_concurrent.py: @mark.xfail(run=False, reason="Crashes because TClingCallFunc generates wrong code") test_concurrent.py: @mark.xfail(run=False, reason="Crashes because the interpreter emits too many warnings") test_concurrent.py: @mark.xfail(run=False, reason="segmentation violation") test_cpp11features.py: @mark.xfail(run=False, reason = "Crashes") test_datatypes.py: @mark.xfail(run=False, reason="segmentation violation") test_datatypes.py: @mark.xfail(run=False, reason="error code: Subprocess aborted") test_doc_features.py: @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit") test_fragile.py: @mark.xfail(run=False, condition=has_asserts(), test_fragile.py: @mark.xfail(run=False, reason="Fatal Python error: Aborted") test_fragile.py: @mark.xfail(run=False, reason="Fatal Python error: Aborted") test_fragile.py: @mark.xfail(run=False, condition=is_modules_off(), reason="Crashes on build with modules off: Fatal Python error: Segmentation fault") test_lowlevel.py: @mark.xfail(run=False, condition=IS_WINDOWS, reason="Windows fatal exception: access violation") test_regression.py: @mark.xfail(run=WINDOWS_BITS != 64, condition=IS_MAC_ARM | WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit and fails macOS ARM with" \ test_stltypes.py: @mark.xfail(run=False, reason="Fatal Python error: Segmentation fault") test_stltypes.py: @mark.xfail() test_stltypes.py: @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason="Crashes on Windows 64 bit") test_templates.py: @mark.xfail(run=False, reason="error code: Subprocess aborted") ``` A future development could be to ensure in Cling or cppyy that these tests at least don't crash but fail gracefully, but that's better to be done after the cppyy upgrade because some tests might be fixed anyway. Some tests remain skipped for good reasons: ```txt git grep "mark\.skip" ``` ```txt test_boost.py:@mark.skipif(noboost == True, reason="boost not found") test_boost.py:@mark.skipif(noboost == True, reason="boost not found") test_boost.py:@mark.skipif(noboost == True, reason="boost not found") test_boost.py:@mark.skipif(noboost == True, reason="boost not found") test_boost.py: @mark.skipif(noboost, reason="boost not found") test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found") test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found") test_fragile.py: @mark.skip(reason="This test is very verbose since it sets gDebug to True") test_fragile.py: @mark.skip(reason="Not actually a cppyy test") test_fragile.py: @mark.skipif(not has_cpp_20(), reason="std::span requires C++20") test_leakcheck.py:@mark.skipif(nopsutil == True, reason="module psutil not installed") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_leakcheck.py: @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs") test_numba.py:@mark.skipif(has_numba == False, reason="numba not found") test_numba.py: @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically") test_numba.py: @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.") test_numba.py: @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.") test_numba.py: @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.") test_numba.py:@mark.skipif(has_numba == False, reason="numba not found") test_eigen.py: @mark.skipif(eigen_path is None, reason="Eigen not found") test_pythonify.py: @mark.skip(reason="Garbage collection tests are fragile") test_regression.py: @mark.skip(reason="For ROOT, we don't enable AVX by default ('-mavx' is not passed to Cling)") test_stltypes.py:@mark.skipif(not has_cpp_20(), reason="std::span requires C++20") ``` Sorry that these `git grep` commands don't show the names of the tests! They are to illustrate how many tests are still marked as `skip`/`xfail` without `strict=True`. Closes root-project#20085.

guitargeek self-assigned this Jan 15, 2026

guitargeek requested a review from vepadulano as a code owner January 15, 2026 20:36

guitargeek added the in:Python Interface label Jan 15, 2026

guitargeek force-pushed the issue-20085 branch from 519d892 to 5f370c9 Compare January 15, 2026 22:06

guitargeek force-pushed the issue-20085 branch 2 times, most recently from e36515c to 16ab12d Compare January 16, 2026 13:01

vepadulano reviewed Jan 16, 2026

View reviewed changes

guitargeek force-pushed the issue-20085 branch 15 times, most recently from 400ec11 to e25b499 Compare January 18, 2026 12:51

guitargeek mentioned this pull request Jan 18, 2026

[ci] Build and test mac14 with the minimal option activated #20930

Open

guitargeek changed the title ~~[cppyy] Mark most of the xfail-ed tests as "strict"~~ [cppyy] Review xfail'ed tests Jan 18, 2026

guitargeek force-pushed the issue-20085 branch 3 times, most recently from 00e1820 to e81987f Compare January 18, 2026 18:24

guitargeek force-pushed the issue-20085 branch from e81987f to ae95b68 Compare January 18, 2026 19:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[cppyy] Review xfail'ed tests #20906

[cppyy] Review xfail'ed tests #20906

guitargeek commented Jan 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

vepadulano Jan 16, 2026

Uh oh!

guitargeek Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[cppyy] Review xfail'ed tests #20906

Are you sure you want to change the base?

[cppyy] Review xfail'ed tests #20906

Conversation

guitargeek commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

vepadulano Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

guitargeek Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

guitargeek commented Jan 15, 2026 •

edited

Loading

github-actions bot commented Jan 16, 2026 •

edited

Loading