Skip to content

Commit 86deac2

Browse files
ammar-agentammar-agent
committed
🤖 feat: add selective integration test execution system
Implements a coverage-based test selection system to reduce CI time by running only the integration tests affected by code changes. Components: - generate-coverage-map.ts: Runs tests individually with coverage to build a reverse index (source file → tests that cover it) - select-affected-tests.ts: Selects tests based on changed files, with safe fallbacks for infrastructure changes, new tests, and unmapped files - coverage-map.yml: Daily workflow to regenerate the map and cache it - Updated ci.yml: Restores map from cache and runs selective tests Safety features: - Falls back to all tests when map is missing/stale (>7 days) - Infrastructure files (jest config, package.json, etc.) trigger all tests - New/unmapped source files trigger all tests - Exit code 2 signals fallback, ensuring CI always runs correct tests _Generated with `mux`_
1 parent 4d1947a commit 86deac2

File tree

7 files changed

+1169
-2
lines changed

7 files changed

+1169
-2
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 54 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -100,14 +100,66 @@ jobs:
100100
- name: Checkout code
101101
uses: actions/checkout@v4
102102
with:
103-
fetch-depth: 0 # Required for git describe to find tags
103+
fetch-depth: 0 # Required for git describe to find tags and diff
104104

105105
- uses: ./.github/actions/setup-mux
106106

107107
- name: Build worker files
108108
run: make build-main
109109

110-
- name: Run all integration tests with coverage
110+
# Try to restore coverage map from cache for selective testing
111+
- name: Restore coverage map
112+
if: github.event.inputs.test_filter == ''
113+
id: coverage-map-cache
114+
uses: actions/cache/restore@v4
115+
with:
116+
path: coverage-map.json
117+
key: coverage-map-${{ github.sha }}
118+
restore-keys: |
119+
coverage-map-latest-
120+
coverage-map-
121+
122+
- name: Select affected tests
123+
if: github.event.inputs.test_filter == '' && steps.coverage-map-cache.outputs.cache-hit != ''
124+
id: select-tests
125+
run: |
126+
# Run selection script and capture output
127+
set +e
128+
SELECTED=$(bun scripts/selective-tests/select-affected-tests.ts \
129+
--map coverage-map.json \
130+
--base origin/${{ github.base_ref || 'main' }} \
131+
--head ${{ github.sha }} \
132+
--output jest \
133+
--verbose)
134+
EXIT_CODE=$?
135+
set -e
136+
137+
echo "exit_code=$EXIT_CODE" >> $GITHUB_OUTPUT
138+
139+
if [[ $EXIT_CODE -eq 0 ]]; then
140+
echo "selected_tests=$SELECTED" >> $GITHUB_OUTPUT
141+
echo "run_selective=true" >> $GITHUB_OUTPUT
142+
else
143+
echo "run_selective=false" >> $GITHUB_OUTPUT
144+
fi
145+
146+
- name: Run selective integration tests
147+
if: steps.select-tests.outputs.run_selective == 'true' && steps.select-tests.outputs.selected_tests != '--testPathPattern=^$'
148+
run: |
149+
echo "Running selective tests: ${{ steps.select-tests.outputs.selected_tests }}"
150+
TEST_INTEGRATION=1 bun x jest --coverage --maxWorkers=100% --silent ${{ steps.select-tests.outputs.selected_tests }}
151+
env:
152+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
153+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
154+
155+
- name: Skip tests (no affected tests)
156+
if: steps.select-tests.outputs.run_selective == 'true' && steps.select-tests.outputs.selected_tests == '--testPathPattern=^$'
157+
run: |
158+
echo "::notice::No integration tests affected by changes - skipping"
159+
echo "No integration tests were affected by the changes in this PR." >> $GITHUB_STEP_SUMMARY
160+
161+
- name: Run all integration tests (fallback or manual filter)
162+
if: steps.select-tests.outputs.run_selective != 'true'
111163
# --silent suppresses per-test output (17+ test files × workers = overwhelming logs)
112164
run: TEST_INTEGRATION=1 bun x jest --coverage --maxWorkers=100% --silent ${{ github.event.inputs.test_filter || 'tests' }}
113165
env:

‎.github/workflows/coverage-map.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: Coverage Map Generation
2+
3+
on:
4+
# Run daily at 2 AM UTC
5+
schedule:
6+
- cron: "0 2 * * *"
7+
# Allow manual trigger
8+
workflow_dispatch:
9+
# Also regenerate when the selective test system changes
10+
push:
11+
branches: [main]
12+
paths:
13+
- "scripts/selective-tests/**"
14+
- "tests/integration/**"
15+
- ".github/workflows/coverage-map.yml"
16+
17+
concurrency:
18+
group: coverage-map-${{ github.ref }}
19+
cancel-in-progress: true
20+
21+
jobs:
22+
generate-coverage-map:
23+
name: Generate Coverage Map
24+
runs-on: ${{ github.repository_owner == 'coder' && 'depot-ubuntu-22.04-16' || 'ubuntu-latest' }}
25+
timeout-minutes: 60
26+
steps:
27+
- name: Checkout code
28+
uses: actions/checkout@v4
29+
with:
30+
fetch-depth: 0
31+
32+
- uses: ./.github/actions/setup-mux
33+
34+
- name: Build worker files
35+
run: make build-main
36+
37+
- name: Generate coverage map
38+
run: bun scripts/selective-tests/generate-coverage-map.ts --output coverage-map.json
39+
env:
40+
TEST_INTEGRATION: "1"
41+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
42+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
43+
44+
- name: Upload coverage map artifact
45+
uses: actions/upload-artifact@v4
46+
with:
47+
name: coverage-map
48+
path: coverage-map.json
49+
retention-days: 30
50+
51+
- name: Cache coverage map for CI
52+
uses: actions/cache/save@v4
53+
with:
54+
path: coverage-map.json
55+
key: coverage-map-${{ github.sha }}
56+
57+
# Also save with a "latest" key that PRs can restore from
58+
- name: Cache coverage map (latest)
59+
uses: actions/cache/save@v4
60+
with:
61+
path: coverage-map.json
62+
key: coverage-map-latest-${{ github.run_id }}
63+
# This creates a new cache entry each run, but restore-keys in CI
64+
# will match the most recent one
65+
66+
- name: Summary
67+
run: |
68+
echo "## Coverage Map Generated" >> $GITHUB_STEP_SUMMARY
69+
echo "" >> $GITHUB_STEP_SUMMARY
70+
echo "- **Commit:** ${{ github.sha }}" >> $GITHUB_STEP_SUMMARY
71+
echo "- **Generated at:** $(date -u)" >> $GITHUB_STEP_SUMMARY
72+
echo "" >> $GITHUB_STEP_SUMMARY
73+
echo "### Map Statistics" >> $GITHUB_STEP_SUMMARY
74+
echo '```json' >> $GITHUB_STEP_SUMMARY
75+
jq '{
76+
version,
77+
generatedAt,
78+
commitSha,
79+
testCount: (.allTests | length),
80+
filesWithCoverage: ([.fileToTests | to_entries[] | select(.value | length > 0)] | length),
81+
totalMappings: ([.fileToTests | to_entries[] | .value | length] | add)
82+
}' coverage-map.json >> $GITHUB_STEP_SUMMARY
83+
echo '```' >> $GITHUB_STEP_SUMMARY

‎scripts/selective-tests/README.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Selective Test Execution System
2+
3+
This system reduces CI time by running only the integration tests affected by code changes, rather than the full test suite on every PR.
4+
5+
## How It Works
6+
7+
1. **Coverage Map Generation**: A daily workflow runs all integration tests individually with code coverage, building a reverse index from source files to tests.
8+
9+
2. **Affected Test Selection**: When a PR runs, the system:
10+
- Restores the coverage map from cache
11+
- Identifies changed files via git diff
12+
- Looks up which tests cover those files
13+
- Runs only the affected tests
14+
15+
3. **Safe Fallbacks**: The system falls back to running all tests when:
16+
- Coverage map is missing or stale (>7 days)
17+
- Infrastructure files change (jest.config, package.json, etc.)
18+
- New test files are added
19+
- Changed source files aren't in the coverage map
20+
- Any error occurs during selection
21+
22+
## Files
23+
24+
- `types.ts` - Shared types and constants
25+
- `generate-coverage-map.ts` - Generates the coverage map by running tests with coverage
26+
- `select-affected-tests.ts` - Selects tests based on changed files
27+
- `run-selective-tests.sh` - CI wrapper script with fallback handling
28+
29+
## Usage
30+
31+
### Generate Coverage Map (local testing)
32+
33+
```bash
34+
bun scripts/selective-tests/generate-coverage-map.ts --output coverage-map.json
35+
```
36+
37+
This takes ~30-60 minutes as it runs each test file individually.
38+
39+
### Select Affected Tests
40+
41+
```bash
42+
# Using git diff
43+
bun scripts/selective-tests/select-affected-tests.ts \
44+
--map coverage-map.json \
45+
--base origin/main \
46+
--head HEAD \
47+
--output jest
48+
49+
# Using explicit file list
50+
bun scripts/selective-tests/select-affected-tests.ts \
51+
--map coverage-map.json \
52+
--changed "src/node/services/workspaceService.ts,src/node/config.ts" \
53+
--output json
54+
```
55+
56+
### Output Formats
57+
58+
- `jest` - Space-separated test files for Jest CLI, or `tests` for all tests
59+
- `json` - Full result object with reasoning
60+
- `list` - Newline-separated test files, or `ALL` for all tests
61+
62+
### Exit Codes
63+
64+
- `0` - Selection successful (may be empty test list)
65+
- `2` - Fallback triggered (run all tests)
66+
- `1` - Error
67+
68+
## CI Integration
69+
70+
The system integrates with `.github/workflows/ci.yml`:
71+
72+
1. Restores coverage map from cache
73+
2. Runs selection script
74+
3. Either runs selected tests or falls back to all tests
75+
4. Skips tests entirely if no tests are affected
76+
77+
The coverage map is regenerated daily by `.github/workflows/coverage-map.yml` and cached for PR use.
78+
79+
## Infrastructure Files
80+
81+
These files trigger a full test run when changed (see `INFRASTRUCTURE_PATTERNS` in `types.ts`):
82+
83+
- Test configuration: `jest.config.cjs`, `babel.config.cjs`
84+
- Build config: `tsconfig.json`, `package.json`, `bun.lockb`
85+
- Test infrastructure: `tests/setup.ts`, `tests/integration/helpers.ts`
86+
- Service container: `src/node/services/serviceContainer.ts`
87+
- Shared types: `src/types/**`, `src/constants/**`
88+
89+
## Shadow Mode
90+
91+
For validation, run with `--shadow-mode` to log what would have been selected while still running all tests:
92+
93+
```bash
94+
./scripts/selective-tests/run-selective-tests.sh --shadow-mode
95+
```
96+
97+
## Debugging
98+
99+
Use `--verbose` for detailed logging:
100+
101+
```bash
102+
bun scripts/selective-tests/select-affected-tests.ts \
103+
--map coverage-map.json \
104+
--changed "src/node/services/aiService.ts" \
105+
--output json \
106+
--verbose
107+
```

0 commit comments

Comments
 (0)