[browser] test robot! #123753
[browser] test robot! #123753
Conversation
pavelsavara
added
NO-REVIEW
github-actions
bot
added
the
needs-area-label
|
Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara |
There was a problem hiding this comment.
Pull request overview
This PR introduces a small “browser test robot” workflow for Browser/WASM + CoreCLR library tests, including helper scripts, documentation, and checked-in sample results for a few suites, plus a minor docs tweak around disabling tests.
Changes:
- Add a Browser/WASM CoreCLR test workflow: setup instructions, execution guide, and a plan document under
browser-tests/. - Introduce scripts to run browser tests locally, download Mono baselines from Helix, and compare CoreCLR vs Mono test sets, along with checked-in example result artifacts (logs, XML, summaries, and failure docs).
- Adjust CI documentation for disabling tests to reference skip attributes, including Mono-specific skips (with one malformed example that should be corrected).
Reviewed changes
Copilot reviewed 19 out of 34 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/workflow/ci/disabling-tests.md | Documents how to disable runtime tests using xUnit attributes; one example for skipping on Mono is currently malformed (OnMono] instead of [SkipOnMono]). |
| browser-tests/before-testing.md | Provides one-time setup instructions for running Browser/WASM + CoreCLR tests (HTTPS dev cert, initial browser build, environment variables). |
| browser-tests/test-suite.md | Defines the end-to-end process for running individual browser test suites, collecting outputs, documenting failures, and maintaining per-suite summaries and comparisons. |
| browser-tests/run-browser-test.sh | Bash helper to run a single Browser/WASM CoreCLR test suite via ./dotnet.sh build /t:Test, capturing console output and copying xharness testResults.xml and wasm-console.log into browser-tests/results/<TestProject>/. |
| browser-tests/download-mono-baseline.sh | Bash script to pull the Mono Helix baseline for a given test project by querying Mono-chrome-workitems.json with jq and curl, storing mono-console.log and mono-testResults.xml under browser-tests/results/<TestProject>/. |
| browser-tests/compare-test-results.sh | Bash script that extracts test names from Mono vs CoreCLR testResults.xml, computes differences with comm, prints a human-readable summary, and writes test-comparison.txt into browser-tests/results/<TestProject>/. |
| browser-tests/plan.md | High-level plan and progress tracker for Browser/WASM + CoreCLR library tests, including platform characteristics, goals, and a status table for completed suites. |
| browser-tests/failures/System.Net.WebSockets.Tests/GetImplMethodDesc.Assert.md | Records a non-fatal CoreCLR interpreter assert (GetImplMethodDesc) observed after WebSockets tests complete, including stack trace and categorization. |
| browser-tests/results/.gitignore | Overrides the repository-wide ignore rules to ensure .log and .xml result files under browser-tests/results/ are kept in version control. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/wasm-console_20260129_160809.log | Captured Browser/WASM + CoreCLR xharness console output for System.Runtime.InteropServices.JavaScript.Tests as a reference run. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/test-comparison.txt | Machine-generated comparison of Mono vs CoreCLR test names for System.Runtime.InteropServices.JavaScript.Tests (showing extra/missing tests). |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/mono-console.log | Helix Mono baseline console log for System.Runtime.InteropServices.JavaScript.Tests, used by the comparison tooling. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/Summary.md | Hand-authored summary of the JavaScript interop tests’ CoreCLR vs Mono results, including benign differences and new tests list. |
| browser-tests/results/System.Net.WebSockets.Tests/test-comparison.txt | Machine-generated comparison of Mono vs CoreCLR for System.Net.WebSockets.Tests confirming identical test sets. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-testResults.xml | Helix Mono baseline testResults.xml for the WebSockets tests, feeding the comparison and summary docs. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-console.log | Helix Mono baseline console log for WebSockets tests. |
| browser-tests/results/System.Net.WebSockets.Tests/Summary.md | Summary of WebSockets CoreCLR vs Mono results and a link to the recorded interpreter assert. |
| browser-tests/results/System.Net.Http.Functional.Tests/wasm-console_20260129_162929.log | Captured console for a CoreCLR Browser/WASM run of System.Net.Http.Functional.Tests that discovered zero tests (Debug configuration / assembly-level skip). |
| browser-tests/results/System.Net.Http.Functional.Tests/testResults_20260129_162929.xml | Corresponding zero-test CoreCLR testResults.xml for System.Net.Http.Functional.Tests (Debug run). |
| browser-tests/results/System.Net.Http.Functional.Tests/mono-console.log | Helix Mono baseline console log for System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/mono-testResults.xml | Helix Mono baseline testResults.xml for System.Net.Http.Functional.Tests used in comparisons. |
| browser-tests/results/System.Net.Http.Functional.Tests/console_20260129_162929.log | Local build+test console for the zero-test CoreCLR Debug run of System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/testResults_20260129_163140.xml | CoreCLR testResults.xml for a subsequent run of System.Net.Http.Functional.Tests (non-zero test count; used by comparison and summary). |
| browser-tests/results/System.Net.Http.Functional.Tests/wasm-console_20260129_163140.log | Browser/WASM + CoreCLR console output for the non-zero-test run of System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/test-comparison.txt | Comparison of Mono vs CoreCLR test names for System.Net.Http.Functional.Tests (differences are only in server port values). |
| browser-tests/results/System.Net.Http.Functional.Tests/Summary.md | Summary of Http functional tests on Browser/WASM + CoreCLR, explaining configuration requirements and interpreting comparison results. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-testResults.xml | (Same as above row for this path) Baseline WebSockets test result data used to power comparisons and summaries. |
| - **Configuration:** Release (Debug skipped by `[SkipOnCoreClr]`) | ||
| - **CoreCLR:** Tests run: 901, Passed: 781, Failed: 0, Skipped: 120 | ||
| - **Mono Baseline:** Tests run: 901, Passed: 781, Failed: 0, Skipped: 120 |
There was a problem hiding this comment.
The CoreCLR test summary here claims Tests run: 901, Passed: 781, Failed: 0, Skipped: 120, but the recorded CoreCLR artifacts in this directory (testResults_20260129_162929.xml and wasm-console_20260129_162929.log) show Tests run: 0. Please either re-run the suite in the required Release configuration and update the artifacts, or adjust the CoreCLR summary to reflect the actual recorded results so the documentation stays accurate.
| int32_t SystemNative_FTruncate (void *, int64_t); | ||
| int32_t SystemNative_FUTimens (void *, void *); | ||
| int32_t SystemNative_FcntlSetFD (void *, int32_t); | ||
| int32_t SystemNative_FileSystemSupportsLocking (void *, int32_t, int32_t); |
There was a problem hiding this comment.
SystemNative_FileSystemSupportsLocking is declared here with an int32_t return type, but the native implementation and header (src/native/libs/System.Native/pal_io.h:792 / pal_io.c:1595) declare it as returning uint32_t. The mismatched signature can lead to undefined behavior at the C level and diverges from the existing declarations; please change this declaration back to uint32_t to match the native API.
| nohup ./dotnet.sh build -bl \ | ||
| /p:TargetOS=browser \ | ||
| /p:TargetArchitecture=wasm \ | ||
| /p:Configuration=${CONFIGURATION} \ | ||
| /t:Test \ | ||
| "$CSPROJ_PATH" 2>&1 | tee "${RESULTS_DIR}/console_${TIMESTAMP}.log" |
There was a problem hiding this comment.
The CONFIGURATION value from the -c option is interpolated into the nohup ./dotnet.sh build command without quoting, which allows shell metacharacters in CONFIGURATION to break out of the intended argument and inject additional commands. An attacker (or untrusted automation) controlling the -c value could pass something like Debug; rm -rf /tmp/data to execute arbitrary shell commands with the current user's privileges. Quote the CONFIGURATION expansion (or strictly validate it against an allowlist such as Debug/Release) before using it in the build command to prevent command injection.
| grep -oP 'name="[^"]*"' "$xml_file" | \ | ||
| grep -v 'assembly name=' | \ | ||
| grep -v 'collection.*name=' | \ | ||
| sed 's/name="//;s/"$//' | \ | ||
| sort -u | ||
| } | ||
|
|
There was a problem hiding this comment.
The filtering of assembly and collection names happens after grep -oP 'name="[^"]*"', so those lines no longer contain the assembly or collection text and are not actually excluded; this causes assembly and collection names to be included as "tests" and skews the Mono/CoreCLR counts (e.g., the MetricOuterLoop1 comparison reports 1 test even though total="0" in the XML). Restrict the extraction to <test ...> elements before pulling the name attribute (for example by grepping for <test first or by adjusting the regex to only match name="..." on <test> lines) so only real test cases are counted.
| grep -oP 'name="[^"]*"' "$xml_file" | \ | |
| grep -v 'assembly name=' | \ | |
| grep -v 'collection.*name=' | \ | |
| sed 's/name="//;s/"$//' | \ | |
| sort -u | |
| } | |
| grep -oP '<test\b[^>]*\Kname="[^"]*"' "$xml_file" | \ | |
| sed 's/name="//;s/"$//' | \ | |
| sort -u | |
| } |
| @@ -0,0 +1,16 @@ | |||
| ## Rebuild After Changes | |||
There was a problem hiding this comment.
The comment says "Rebuild just libraries" but the command uses -subset clr+libs+host, which also rebuilds the host; either update the text to reflect that host is included or change the subset to clr+libs if the intention is truly to rebuild only libraries. Aligning the wording and the command will help avoid confusion for people following these instructions.
| Read the `browser-tests/plan.md` | ||
| Run the steps in `browser-tests/before-testing.md`. | ||
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | ||
|
|
||
|
|
||
| Read the `browser-tests/plan.md` | ||
| Run the steps in `browser-tests/before-testing.md` except building the runtime, that already done. | ||
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests` | ||
|
|
||
| Read the `browser-tests/plan.md` | ||
| The before testing steps are already done. | ||
| Now we want to run `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests` No newline at end of file |
There was a problem hiding this comment.
This file repeats similar instructions three times with slightly different preconditions and target test suites, which makes it unclear whether these are sequential steps, alternative flows, or just notes; consolidating into a single, clearly structured set of steps (or explicitly labeling them as separate scenarios) would make it easier for others to understand and follow.
| Read the `browser-tests/plan.md` | |
| Run the steps in `browser-tests/before-testing.md`. | |
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| Read the `browser-tests/plan.md` | |
| Run the steps in `browser-tests/before-testing.md` except building the runtime, that already done. | |
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests` | |
| Read the `browser-tests/plan.md` | |
| The before testing steps are already done. | |
| Now we want to run `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests` | |
| This document describes three alternative scenarios for running browser tests. | |
| ## Scenario 1: First-time run of System.Runtime.Tests | |
| 1. Read `browser-tests/plan.md`. | |
| 2. Run all steps in `browser-tests/before-testing.md`. | |
| 3. Run the `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| ## Scenario 2: Runtime already built, rerun System.Runtime.Tests | |
| Use this scenario if the runtime has already been built, but the other "before testing" steps still need to run. | |
| 1. Read `browser-tests/plan.md`. | |
| 2. Run the steps in `browser-tests/before-testing.md` except for building the runtime. | |
| 3. Run the `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| ## Scenario 3: Before-testing steps already completed, run System.Collections.Immutable.Tests | |
| Use this scenario if all steps in `browser-tests/before-testing.md` have already been completed. | |
| 1. (Optional) Re-read `browser-tests/plan.md` if you need a refresher. | |
| 2. Run the `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests`. |
1 participant
No description provided.