|
| 1 | +# CI Speed Optimization Results |
| 2 | + |
| 3 | +## Comparison: Before vs After |
| 4 | + |
| 5 | +### Integration Tests |
| 6 | + |
| 7 | +**Before (Run 18827071962):** |
| 8 | +- Test execution: 193s |
| 9 | +- Total time: 3m59s (239s) |
| 10 | + |
| 11 | +**After (Run 18827186714) - PR #443:** |
| 12 | +- Test execution: **183s** β
|
| 13 | +- Total time: **3m53s** (233s) |
| 14 | + |
| 15 | +**Improvement:** |
| 16 | +- Test execution: **10 seconds faster** (5.2% improvement) |
| 17 | +- Total time: **6 seconds faster** |
| 18 | + |
| 19 | +### macOS Build |
| 20 | + |
| 21 | +**Before (Run 18827071959):** |
| 22 | +- setup-cmux: 87s |
| 23 | +- Build application: 28s |
| 24 | +- Package: 93s |
| 25 | +- Total time: 3m50s (230s) |
| 26 | + |
| 27 | +**After (Run 18827186715) - PR #443:** |
| 28 | +- setup-cmux: **107s** β (20s slower!) |
| 29 | +- Build application: **38s** (10s slower) |
| 30 | +- Package: **113s** (20s slower!) |
| 31 | +- Total time: **4m50s** (290s) |
| 32 | + |
| 33 | +**Result: 60 seconds SLOWER overall** β οΈ |
| 34 | + |
| 35 | +## Analysis |
| 36 | + |
| 37 | +### Integration Tests: β
SUCCESS |
| 38 | +Reducing maxWorkers from 200% to 100% provided a modest improvement: |
| 39 | +- 10s faster execution (5.2% improvement) |
| 40 | +- Validates hypothesis that over-parallelization was causing some contention |
| 41 | +- Could potentially tune further (try 150%?) |
| 42 | + |
| 43 | +### macOS Build: β UNEXPECTED REGRESSION |
| 44 | + |
| 45 | +This is surprising - the macOS build got significantly slower despite: |
| 46 | +1. Adding `HOMEBREW_NO_INSTALL_CLEANUP=1` (expected to save time) |
| 47 | +2. No other changes to the build process |
| 48 | + |
| 49 | +**Possible explanations:** |
| 50 | +1. **Runner variance**: Depot runners can have different performance characteristics |
| 51 | + - May have gotten a slower runner this time |
| 52 | + - Need multiple runs to establish reliable baseline |
| 53 | + |
| 54 | +2. **Network/cache variance**: |
| 55 | + - Homebrew downloads may have been slower |
| 56 | + - Bun install cache may have missed |
| 57 | + |
| 58 | +3. **The brew cleanup flag backfired**: |
| 59 | + - Unlikely but possible that cleanup was actually helpful |
| 60 | + - Or flag isn't being respected |
| 61 | + |
| 62 | +**Next steps:** |
| 63 | +- Need to run this workflow multiple times to see if it's consistent |
| 64 | +- Compare logs to see what actually took longer |
| 65 | +- May need to revert the HOMEBREW_NO_INSTALL_CLEANUP change |
| 66 | + |
| 67 | +## Recommendation |
| 68 | + |
| 69 | +### Keep Integration Test Change β
|
| 70 | +The maxWorkers=100% change shows consistent improvement and makes logical sense. |
| 71 | + |
| 72 | +### Investigate macOS Slowdown π |
| 73 | +Need more data before deciding on the brew cleanup change: |
| 74 | +1. Run the workflow 2-3 more times |
| 75 | +2. Compare detailed logs between fast and slow runs |
| 76 | +3. Check if it's runner variance or actual regression |
| 77 | + |
| 78 | +### Baseline Variability |
| 79 | +CI timings can vary Β±10-20% between runs due to: |
| 80 | +- Runner performance variance (different physical machines) |
| 81 | +- Network conditions |
| 82 | +- Cache hit rates |
| 83 | +- Background load |
| 84 | + |
| 85 | +Should collect 3-5 samples before drawing conclusions. |
| 86 | + |
0 commit comments