Skip to content

Commit f36c984

Browse files
ammar-agentammar-agent
committed
🤖 Optimize CI speed: reduce integration test workers and skip brew cleanup
Changes: 1. Integration tests: maxWorkers 200% -> 100% - 64 workers on 32-core runner may cause contention - Each test spawns child processes, does I/O, git ops - Expected: 20-30% speedup (38-58s savings) 2. macOS brew: Skip post-install cleanup - Add HOMEBREW_NO_INSTALL_CLEANUP=1 - Expected: 5-10s savings on ImageMagick install 3. Integration tests: Remove --silent flag - Allows profiling of individual test timing - Can identify which tests are slowest Analysis document: CI_SPEED_ANALYSIS.md Baseline (from run 18827071962): - Integration tests: 193s execution - macOS setup-cmux: 87s (ImageMagick ~45-70s) Generated with `cmux`
1 parent 36d4af2 commit f36c984

File tree

3 files changed

+207
-2
lines changed

3 files changed

+207
-2
lines changed

‎.github/actions/setup-cmux/action.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ runs:
4646
echo "✅ ImageMagick already available"
4747
else
4848
echo "📦 Installing ImageMagick..."
49-
HOMEBREW_NO_AUTO_UPDATE=1 brew install imagemagick
49+
HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1 brew install imagemagick
5050
fi
5151
magick --version | head -1
5252

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ jobs:
9797
- uses: ./.github/actions/setup-cmux
9898

9999
- name: Run integration tests with coverage
100-
run: TEST_INTEGRATION=1 bun x jest --coverage --maxWorkers=200% --silent ${{ github.event.inputs.test_filter || 'tests' }}
100+
run: TEST_INTEGRATION=1 bun x jest --coverage --maxWorkers=100% ${{ github.event.inputs.test_filter || 'tests' }}
101101
env:
102102
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
103103
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

‎CI_SPEED_ANALYSIS.md‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# CI Speed Investigation - October 2025
2+
3+
## Current State
4+
5+
### Integration Tests: 3m59s
6+
```
7+
Set up job: 1s
8+
Checkout: 5s
9+
setup-cmux: 10s
10+
Test execution: 193s ← PRIMARY TARGET
11+
Upload coverage: 4s
12+
Post actions: 5s
13+
---
14+
Total: 218s
15+
```
16+
17+
### macOS Build: 3m50s
18+
```
19+
Set up job: 2s
20+
Checkout: 1s
21+
setup-cmux: 87s ← MAJOR BOTTLENECK (ImageMagick ~45-70s)
22+
Build app: 28s
23+
Package (signing): 93s ← MAJOR BOTTLENECK
24+
Upload artifacts: 10s
25+
Post actions: 7s
26+
---
27+
Total: 228s
28+
```
29+
30+
## Optimization Opportunities
31+
32+
### 1. Integration Tests (193s execution)
33+
34+
#### Current Configuration
35+
- Runner: depot-ubuntu-24.04-32 (32 cores)
36+
- Jest config: `maxWorkers=200%` (64 parallel workers)
37+
- Test files: 17 files in `tests/ipcMain/` + `tests/runtime/`
38+
- Flags: `--coverage --maxWorkers=200% --silent`
39+
40+
#### Potential Issues
41+
1. **Over-parallelization**: 64 workers on 32 cores may cause contention
42+
- Each test spawns child processes, file I/O, git operations
43+
- May be better to use fewer workers (100% or 150%)
44+
45+
2. **Silent mode hides timing data**: Can't see which tests are slow
46+
- Remove `--silent` temporarily to profile
47+
- Add `--verbose` to see timing per test
48+
49+
3. **Coverage overhead**: Jest coverage adds significant overhead
50+
- Consider running coverage only on specific files
51+
- Or separate coverage runs from speed-critical PRs
52+
53+
4. **Test setup duplication**: Each test file creates full test environments
54+
- May benefit from shared setup/teardown
55+
- Check if any tests can be simplified
56+
57+
#### Action Items
58+
- [ ] Profile tests without `--silent` to identify slow tests
59+
- [ ] Experiment with `maxWorkers=100%` vs `200%` vs `150%`
60+
- [ ] Measure overhead of `--coverage` flag
61+
- [ ] Review slowest test files for optimization opportunities
62+
- [ ] Check if test files can share more setup code
63+
64+
### 2. macOS setup-cmux (87s)
65+
66+
#### Current Breakdown
67+
- Setup Bun: ~5s
68+
- Cache operations: ~5s
69+
- bun install: ~5-10s (usually cached)
70+
- **ImageMagick install: ~45-70s** ← Unavoidable on ephemeral runners
71+
72+
#### Homebrew ImageMagick Install
73+
Current command:
74+
```bash
75+
HOMEBREW_NO_AUTO_UPDATE=1 brew install imagemagick
76+
```
77+
78+
Takes 45-70s on depot-macos-15 runners.
79+
80+
#### Potential Optimizations
81+
1. **Skip brew cleanup**: Add `HOMEBREW_NO_INSTALL_CLEANUP=1`
82+
- May save 5-10s by skipping post-install cleanup
83+
84+
2. **Use pre-compiled bottles**: Brew should already use bottles, but verify
85+
- Check logs to ensure not building from source
86+
87+
3. **Investigate alternative ImageMagick sources**:
88+
- Could we use a pre-built binary from GitHub releases?
89+
- Would need to be compatible with electron-builder expectations
90+
91+
#### Action Items
92+
- [ ] Add `HOMEBREW_NO_INSTALL_CLEANUP=1` and measure impact
93+
- [ ] Verify ImageMagick is using bottles (not building from source)
94+
- [ ] Research if electron-builder can use a simpler ImageMagick setup
95+
- [ ] Consider if we can cache ImageMagick binary itself (risky with dependencies)
96+
97+
### 3. macOS Packaging (93s)
98+
99+
#### Current Process
100+
electron-builder creates:
101+
- x64 DMG (~45s)
102+
- arm64 DMG (~45s)
103+
- Both are signed and notarized
104+
105+
#### Current Makefile Target
106+
```make
107+
dist-mac:
108+
bun run dist:mac
109+
```
110+
111+
Which runs electron-builder with platform-specific config.
112+
113+
#### Potential Optimizations
114+
1. **Parallel architecture builds**: electron-builder may build sequentially
115+
- Check if `--arm64 --x64` can build in parallel
116+
- May need to split into separate CI jobs
117+
118+
2. **Skip notarization on PRs**: Notarization takes time
119+
- Already configured via `CSC_FOR_PULL_REQUEST`
120+
- Verify this is working correctly
121+
122+
3. **Optimize electron-builder config**:
123+
- Check compression settings (can trade size for speed)
124+
- Review if all build steps are necessary for CI
125+
126+
4. **Code signing optimization**:
127+
- Verify signing doesn't happen twice
128+
- Check if there's redundant identity verification
129+
130+
#### Action Items
131+
- [ ] Profile electron-builder to see x64 vs arm64 timing
132+
- [ ] Investigate parallel architecture builds
133+
- [ ] Verify notarization is actually skipped on PRs
134+
- [ ] Review electron-builder config for speed optimizations
135+
- [ ] Consider if we need both architectures on every PR (vs just merge/release)
136+
137+
### 4. Quick Wins (Already Implemented)
138+
139+
✅ **Removed ImageMagick from test workflows** (PR #434)
140+
- Saved ~30s per test workflow
141+
- Total savings: ~150s per CI run
142+
143+
## Testing Strategy
144+
145+
### Baseline Measurements
146+
Need to run controlled tests to measure current performance:
147+
148+
1. Integration tests with different worker counts
149+
2. macOS build with brew optimizations
150+
3. Packaging with different electron-builder settings
151+
152+
### Experimental Branches
153+
For each optimization, create a test branch and measure:
154+
- Run the same workflow 3 times
155+
- Take average timing
156+
- Compare against baseline
157+
158+
### Measurement Commands
159+
```bash
160+
# Check recent run timings
161+
gh run list --workflow=ci.yml --limit 5 --json number,conclusion,displayTitle,createdAt
162+
163+
# Get detailed step timings for a run
164+
gh api repos/coder/cmux/actions/runs/<RUN_ID>/jobs | \
165+
jq -r '.jobs[] | select(.name == "Integration Tests") | .steps[] | "\(.name): \(((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)))s"'
166+
167+
# Run integration tests locally with timing
168+
time TEST_INTEGRATION=1 bun x jest --maxWorkers=100% tests
169+
```
170+
171+
## Priority Order
172+
173+
1. **HIGH PRIORITY - Integration Tests Parallelization**
174+
- Quick to test locally
175+
- Could yield 20-30% improvement (38-58s savings)
176+
- No infrastructure risk
177+
178+
2. **MEDIUM PRIORITY - Homebrew Cleanup Optimization**
179+
- Small change, low risk
180+
- Potential 5-10s savings on macOS
181+
- Easy to test and revert
182+
183+
3. **LOW PRIORITY - Packaging Optimization**
184+
- Complex changes required
185+
- Need to understand electron-builder deeply
186+
- Risk of breaking packaging
187+
- But largest potential savings (20-40s possible)
188+
189+
4. **RESEARCH - Coverage Overhead**
190+
- May be significant (10-20% of test time)
191+
- But losing coverage visibility has cost
192+
- Consider selective coverage or separate runs
193+
194+
## Next Steps
195+
196+
1. Profile integration tests without --silent
197+
2. Test different maxWorkers settings locally
198+
3. Implement brew cleanup optimization for macOS
199+
4. Measure and iterate
200+
201+
---
202+
203+
**Branch**: `investigate-speed-2`
204+
**Created**: October 2025
205+
**Status**: Investigation phase

0 commit comments

Comments
 (0)