Skip to content

Commit 4994ac1

Browse files
Add Windows debugging scripts for flaky stdio test
Add PowerShell scripts to help debug test_stdio_context_manager_exiting flakiness on Windows with Python 3.11/3.12: - check-tee-command.ps1: Verify tee command availability - test-stdio-flakiness-200-runs.ps1: Run test 200 times for statistics - test-stdio-flakiness-until-failure.ps1: Run until failure occurs - test-stdio-verbose-debug.ps1: Single run with maximum debug output - README.md: Documentation for using the scripts
1 parent 0b4ce00 commit 4994ac1

File tree

5 files changed

+282
-0
lines changed

5 files changed

+282
-0
lines changed

scripts/windows-debug/README.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Windows Test Debugging Scripts
2+
3+
This folder contains PowerShell scripts to help debug the flaky `test_stdio_context_manager_exiting` test on Windows with Python 3.11/3.12.
4+
5+
## Prerequisites
6+
7+
- Windows PowerShell or PowerShell Core
8+
- Python 3.11 or 3.12
9+
- `uv` installed and configured
10+
- Virtual environment activated
11+
12+
## Scripts
13+
14+
### 1. `check-tee-command.ps1`
15+
Checks if the `tee` command is available on your Windows system. The test requires `tee` to run.
16+
17+
```powershell
18+
.\check-tee-command.ps1
19+
```
20+
21+
### 2. `test-stdio-flakiness-200-runs.ps1`
22+
Runs the test 200 times and reports the failure rate.
23+
24+
```powershell
25+
.\test-stdio-flakiness-200-runs.ps1
26+
```
27+
28+
### 3. `test-stdio-flakiness-until-failure.ps1`
29+
Runs the test in a loop until it fails, useful for catching intermittent failures.
30+
31+
```powershell
32+
.\test-stdio-flakiness-until-failure.ps1
33+
```
34+
35+
### 4. `test-stdio-verbose-debug.ps1`
36+
Runs the test once with maximum debugging output enabled.
37+
38+
```powershell
39+
.\test-stdio-verbose-debug.ps1
40+
```
41+
42+
## Usage Notes
43+
44+
1. Make sure your virtual environment is activated before running these scripts
45+
2. Run scripts from the repository root directory
46+
3. If scripts fail with execution policy errors, run:
47+
```powershell
48+
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
49+
```
50+
51+
## Interpreting Results
52+
53+
- If `tee` is not available, the test will be skipped
54+
- Flaky failures typically show up as timeout errors or process cleanup issues
55+
- Look for patterns in failure output, especially around process termination timing
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
#!/usr/bin/env pwsh
2+
# Script to check if tee command is available on Windows
3+
# Usage: .\check-tee-command.ps1
4+
5+
Write-Host "Checking for 'tee' command availability on Windows..." -ForegroundColor Cyan
6+
Write-Host ""
7+
8+
# Method 1: Using where.exe
9+
Write-Host "Method 1: Using where.exe" -ForegroundColor Yellow
10+
try {
11+
$whereResult = where.exe tee 2>&1
12+
if ($LASTEXITCODE -eq 0) {
13+
Write-Host " Found tee at: $whereResult" -ForegroundColor Green
14+
} else {
15+
Write-Host " tee not found via where.exe" -ForegroundColor Red
16+
}
17+
} catch {
18+
Write-Host " Error checking with where.exe: $_" -ForegroundColor Red
19+
}
20+
21+
Write-Host ""
22+
23+
# Method 2: Using Get-Command
24+
Write-Host "Method 2: Using Get-Command" -ForegroundColor Yellow
25+
try {
26+
$getCommandResult = Get-Command tee -ErrorAction Stop
27+
Write-Host " Found tee:" -ForegroundColor Green
28+
Write-Host " Name: $($getCommandResult.Name)"
29+
Write-Host " CommandType: $($getCommandResult.CommandType)"
30+
Write-Host " Source: $($getCommandResult.Source)"
31+
Write-Host " Version: $($getCommandResult.Version)"
32+
} catch {
33+
Write-Host " tee not found via Get-Command" -ForegroundColor Red
34+
}
35+
36+
Write-Host ""
37+
38+
# Method 3: Check common locations
39+
Write-Host "Method 3: Checking common locations" -ForegroundColor Yellow
40+
$commonPaths = @(
41+
"C:\Program Files\Git\usr\bin\tee.exe",
42+
"C:\Program Files (x86)\Git\usr\bin\tee.exe",
43+
"C:\tools\msys64\usr\bin\tee.exe",
44+
"C:\msys64\usr\bin\tee.exe",
45+
"C:\cygwin64\bin\tee.exe",
46+
"C:\cygwin\bin\tee.exe"
47+
)
48+
49+
$found = $false
50+
foreach ($path in $commonPaths) {
51+
if (Test-Path $path) {
52+
Write-Host " Found at: $path" -ForegroundColor Green
53+
$found = $true
54+
}
55+
}
56+
57+
if (-not $found) {
58+
Write-Host " tee not found in common locations" -ForegroundColor Red
59+
}
60+
61+
Write-Host ""
62+
63+
# Method 4: Check if it's a PowerShell alias
64+
Write-Host "Method 4: Checking PowerShell aliases" -ForegroundColor Yellow
65+
$alias = Get-Alias tee -ErrorAction SilentlyContinue
66+
if ($alias) {
67+
Write-Host " Found PowerShell alias:" -ForegroundColor Green
68+
Write-Host " Name: $($alias.Name)"
69+
Write-Host " Definition: $($alias.Definition)"
70+
} else {
71+
Write-Host " No PowerShell alias for tee" -ForegroundColor Yellow
72+
}
73+
74+
Write-Host ""
75+
76+
# Method 5: Python check (what the test uses)
77+
Write-Host "Method 5: Python shutil.which() check" -ForegroundColor Yellow
78+
$pythonCheck = python -c "import shutil; print(shutil.which('tee'))"
79+
if ($pythonCheck -and $pythonCheck -ne "None") {
80+
Write-Host " Python found tee at: $pythonCheck" -ForegroundColor Green
81+
} else {
82+
Write-Host " Python shutil.which() did not find tee" -ForegroundColor Red
83+
}
84+
85+
Write-Host ""
86+
Write-Host "========== SUMMARY ==========" -ForegroundColor Cyan
87+
if ($whereResult -or $getCommandResult -or $found -or ($pythonCheck -and $pythonCheck -ne "None")) {
88+
Write-Host "tee command is available" -ForegroundColor Green
89+
Write-Host ""
90+
Write-Host "The test_stdio_context_manager_exiting test should run." -ForegroundColor Green
91+
} else {
92+
Write-Host "tee command is NOT available" -ForegroundColor Red
93+
Write-Host ""
94+
Write-Host "The test_stdio_context_manager_exiting test will be SKIPPED." -ForegroundColor Yellow
95+
Write-Host ""
96+
Write-Host "To install tee on Windows, you can:" -ForegroundColor Cyan
97+
Write-Host " 1. Install Git for Windows (includes tee in Git Bash)"
98+
Write-Host " 2. Install WSL (Windows Subsystem for Linux)"
99+
Write-Host " 3. Install MSYS2 or Cygwin"
100+
Write-Host " 4. Use PowerShell's Tee-Object cmdlet (different syntax)"
101+
}
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/usr/bin/env pwsh
2+
# Script to run test_stdio_context_manager_exiting 200 times to detect flakiness
3+
# Usage: .\test-stdio-flakiness-200-runs.ps1
4+
5+
Write-Host "Running test_stdio_context_manager_exiting 200 times to detect flakiness..." -ForegroundColor Cyan
6+
Write-Host "Test: tests/client/test_stdio.py::test_stdio_context_manager_exiting" -ForegroundColor Yellow
7+
Write-Host ""
8+
9+
$startTime = Get-Date
10+
$count = 0
11+
$failures = 0
12+
$failedRuns = @()
13+
14+
for ($i = 1; $i -le 200; $i++) {
15+
Write-Host "Run $i of 200..." -NoNewline
16+
17+
$output = uv run --frozen pytest tests/client/test_stdio.py::test_stdio_context_manager_exiting -xvs 2>&1
18+
$exitCode = $LASTEXITCODE
19+
20+
if ($exitCode -ne 0) {
21+
$failures++
22+
$failedRuns += $i
23+
Write-Host " FAILED" -ForegroundColor Red
24+
Write-Host "Failure output:" -ForegroundColor Red
25+
Write-Host $output
26+
Write-Host ""
27+
} else {
28+
Write-Host " PASSED" -ForegroundColor Green
29+
}
30+
}
31+
32+
$endTime = Get-Date
33+
$duration = $endTime - $startTime
34+
35+
Write-Host ""
36+
Write-Host "========== SUMMARY ==========" -ForegroundColor Cyan
37+
Write-Host "Total runs: 200"
38+
Write-Host "Successful runs: $($count - $failures)" -ForegroundColor Green
39+
Write-Host "Failed runs: $failures" -ForegroundColor Red
40+
if ($failures -gt 0) {
41+
Write-Host "Failed on runs: $($failedRuns -join ', ')" -ForegroundColor Red
42+
}
43+
Write-Host "Duration: $($duration.ToString())"
44+
Write-Host "Failure rate: $([math]::Round(($failures / 200) * 100, 2))%"
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env pwsh
2+
# Script to run test_stdio_context_manager_exiting until it fails
3+
# Usage: .\test-stdio-flakiness-until-failure.ps1
4+
5+
Write-Host "Running test_stdio_context_manager_exiting until failure..." -ForegroundColor Cyan
6+
Write-Host "Test: tests/client/test_stdio.py::test_stdio_context_manager_exiting" -ForegroundColor Yellow
7+
Write-Host "Press Ctrl+C to stop" -ForegroundColor Gray
8+
Write-Host ""
9+
10+
$startTime = Get-Date
11+
$i = 0
12+
13+
while ($true) {
14+
$i++
15+
Write-Host "Run $i..." -NoNewline
16+
17+
$output = uv run --frozen pytest tests/client/test_stdio.py::test_stdio_context_manager_exiting -xvs 2>&1
18+
$exitCode = $LASTEXITCODE
19+
20+
if ($exitCode -ne 0) {
21+
$endTime = Get-Date
22+
$duration = $endTime - $startTime
23+
24+
Write-Host " FAILED!" -ForegroundColor Red
25+
Write-Host ""
26+
Write-Host "========== FAILURE DETECTED ==========" -ForegroundColor Red
27+
Write-Host "Failed on run: $i" -ForegroundColor Red
28+
Write-Host "Time until failure: $($duration.ToString())" -ForegroundColor Yellow
29+
Write-Host ""
30+
Write-Host "Failure output:" -ForegroundColor Red
31+
Write-Host $output
32+
break
33+
} else {
34+
Write-Host " PASSED" -ForegroundColor Green
35+
}
36+
37+
# Small delay to prevent overwhelming the system
38+
Start-Sleep -Milliseconds 100
39+
}
40+
41+
Write-Host ""
42+
Write-Host "Exiting after failure detection." -ForegroundColor Cyan
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/usr/bin/env pwsh
2+
# Script to run test_stdio_context_manager_exiting with maximum debugging output
3+
# Usage: .\test-stdio-verbose-debug.ps1
4+
5+
Write-Host "Running test_stdio_context_manager_exiting with verbose debug output..." -ForegroundColor Cyan
6+
Write-Host ""
7+
8+
# Set environment variables for debugging
9+
$env:PYTHONFAULTHANDLER = "1"
10+
$env:PYTEST_CURRENT_TEST = "1"
11+
12+
Write-Host "Environment variables set:" -ForegroundColor Yellow
13+
Write-Host " PYTHONFAULTHANDLER = 1 (enables Python fault handler)"
14+
Write-Host ""
15+
16+
Write-Host "Running test with maximum verbosity..." -ForegroundColor Cyan
17+
Write-Host ""
18+
19+
# Run the test with all debugging options
20+
uv run --frozen pytest `
21+
tests/client/test_stdio.py::test_stdio_context_manager_exiting `
22+
-xvs `
23+
--log-cli-level=DEBUG `
24+
--log-cli-format="%(asctime)s [%(levelname)s] %(name)s: %(message)s" `
25+
--capture=no `
26+
--tb=long `
27+
--full-trace
28+
29+
$exitCode = $LASTEXITCODE
30+
31+
Write-Host ""
32+
if ($exitCode -eq 0) {
33+
Write-Host "Test PASSED" -ForegroundColor Green
34+
} else {
35+
Write-Host "Test FAILED with exit code: $exitCode" -ForegroundColor Red
36+
}
37+
38+
# Clean up environment variables
39+
Remove-Item Env:PYTHONFAULTHANDLER -ErrorAction SilentlyContinue
40+
Remove-Item Env:PYTEST_CURRENT_TEST -ErrorAction SilentlyContinue

0 commit comments

Comments
 (0)