backport: improve e2e testing approach to reduce flakiness #10309
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add retry mechanism for failed tests - Use environment variable isolation with WRANGLER_E2E_TEST_FILE - Implement glob-based test file discovery - Add better error handling and reporting - Include GitHub Actions console grouping This backports the improved e2e testing approach from main branch to reduce flakiness in e2e tests on v3-maintenance branch. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
The improved e2e runner imports glob directly, so it needs to be added as a dependency to the tools package.json. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
This resolves the CI failure: ERR_PNPM_OUTDATED_LOCKFILE Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with tools/package.json Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
petebacondarwin
added
the
skip-pr-description-validation
- Add globalPassThroughEnv to root turbo.json with essential environment variables - Update packages/wrangler/turbo.json with WRANGLER_E2E_TEST_FILE and other missing env vars - Add passThroughEnv array to build task with comprehensive environment variable list - Update build inputs to use $TURBO_DEFAULT$ pattern from main branch - Add create-cloudflare#build dependency to test:e2e task These changes ensure the backported e2e runner can properly utilize turbo cache by recognizing the WRANGLER_E2E_TEST_FILE environment variable and other essential configuration, preventing cache misses that cause test timeouts. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
…:e2e task The create-cloudflare#build dependency doesn't exist in v3-maintenance branch, causing E2E tests to fail with 'Could not find package create-cloudflare from task create-cloudflare#build'. This fix removes the problematic dependency while keeping the essential WRANGLER_E2E_TEST_FILE environment variable that enables proper turbo caching for the backported e2e runner. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
… handling This ensures the e2e test runner uses the same environment variable handling pattern as other turbo tasks, optimizing turbo cache behavior. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
…assing issues The dotenv prefix was causing the WRANGLER environment variable to not be passed through properly, causing E2E test validation failures. The core turbo cache optimizations (env var configuration, dependency fixes) remain in place and provide the intended performance improvements. Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
Backport the vitest configuration from main branch to properly handle the WRANGLER_E2E_TEST_FILE environment variable for individual test file execution. This completes the e2e testing improvements backport. - Add retry: 1 configuration for automatic test retries - Add dynamic include pattern using WRANGLER_E2E_TEST_FILE env var - Fixes E2E test failures by enabling proper test file filtering Co-Authored-By: pbacondarwin@cloudflare.com <pete@bacondarwin.com>
petebacondarwin
approved these changes
Aug 12, 2025
CarmenPopoviciu
approved these changes
Aug 13, 2025
workers-devprod
added
the
contribution
|
Congratulations @devin-ai-integration[bot], the maintainer of this repository has issued you a holobyte! Here it is: https://holopin.io/holobyte/cme9utunb033607l1n41cnb0g This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR backports the improved e2e testing approach from the
mainbranch to thev3-maintenancebranch to reduce flakiness in e2e tests.Related PRs:
Changes Summary
Replaces the simple e2e test runner in
tools/e2e/runIndividualE2EFiles.tswith the improved version from main branch that includes:Key Improvements
WRANGLER_E2E_TEST_FILEto pass specific test files while reusing cached buildse2e/**/*.test.ts) instead of simple directory reading for more robust file discovery::group::) for better CI output organization-ufor snapshot updates)Technical Details
The main changes include:
assert,statSync,path, andglobIterateSyncreaddirSyncwithglobIterateSyncfor more flexible file discoveryHuman Review Checklist
pnpm test:e2e:wranglerworks on v3-maintenanceWRANGLER_E2E_TEST_FILEis properly handled by v3-maintenance test infrastructureLink to Devin run: https://app.devin.ai/sessions/a6bcd0abb9074acea6c9d2b286bce88a
Requested by: @petebacondarwin