Commit ee62b00
authored
🤖 Improve integration test diagnostics for flaky tool policy tests (#46)
## Problem
The tool policy integration tests are flaky in CI, timing out waiting
for `stream-end` events with no useful diagnostic information. When they
fail, we only see:
```
Expected: true
Received: false
at assertStreamSuccess (tests/ipcMain/helpers.ts:191:36)
```
This tells us nothing about what actually went wrong.
## Solution
This PR makes **general improvements to test helpers** that benefit all
integration tests:
### 1. Enhanced `waitForEvent` helper
- Automatically logs stream-error details when timing out
- Shows error message and error type for debugging
- Makes all integration tests more debuggable
### 2. Enhanced `assertStreamSuccess` helper
- Replaced generic expect() calls with descriptive Error messages
- Shows all collected events when any assertion fails
- Distinguishes between different failure modes:
- Stream didn't complete (no stream-end)
- Stream errored (has stream-error)
- Stream completed but missing final message
- Includes the actual error message in the failure output
### 3. Fixed tool policy tests to wait for completion
- Now wait for either `stream-end` OR `stream-error` (prevents timeout)
- Rely on improved helpers for diagnostics
- Much simpler and more maintainable
## Impact
These changes benefit **all integration tests**, not just tool policy
tests. Any test using `waitForEvent` or `assertStreamSuccess` will now
provide much better error messages when failing.
## Next Steps
Once this PR is merged and we see the tests fail in CI, we'll get
detailed error logs that will help us:
- Identify if the AI is trying to use disabled tools
- See the actual API errors if any
- Understand the sequence of events leading to failure
- Fix the root cause instead of masking it with timeouts
_Generated with `cmux`_1 parent 4006e3f commit ee62b00
2 files changed
+61
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
145 | 154 | | |
146 | 155 | | |
147 | 156 | | |
| |||
186 | 195 | | |
187 | 196 | | |
188 | 197 | | |
| 198 | + | |
189 | 199 | | |
190 | 200 | | |
191 | | - | |
192 | | - | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
193 | 229 | | |
194 | | - | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
195 | 235 | | |
196 | 236 | | |
197 | 237 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
833 | 833 | | |
834 | 834 | | |
835 | 835 | | |
836 | | - | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
837 | 845 | | |
838 | 846 | | |
839 | 847 | | |
| |||
884 | 892 | | |
885 | 893 | | |
886 | 894 | | |
887 | | - | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
888 | 904 | | |
889 | 905 | | |
890 | 906 | | |
| |||
0 commit comments