[pull] canary from vercel:canary #735
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Summary Updates the instruction in `agents-md` [CLAUDE.md](http://CLAUDE.md) output to force LLMs to actually read the docs instead of relying on stale pre-training knowledge. **Before:** ``` IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks. ``` **After:** ``` STOP. What you remember about Next.js is WRONG for this project. Always search docs and read before any task. ``` ## Why This Matters Through systematic testing with the [next-evals-oss](https://github.com/vercel/next-evals-oss) eval suite, we discovered that the original "prefer retrieval-led reasoning" instruction was ineffective. Agents would ignore it and use outdated pre-training knowledge (e.g., creating `middleware.ts` instead of the new Next.js 16 `proxy.ts` convention). ## Testing Methodology We created an "indirect proxy" eval that tests if agents know `proxy.ts` is needed for request interception in Next.js 16+, using only the prompt "Log every request to the console" (no mention of proxy/middleware). | Instruction | Pass Rate | | --- | --- | | "Prefer retrieval-led reasoning" | 0% | | "CRITICAL: MUST read docs" | ~20-40% | | "What you remember is WRONG" | **100% (6/6)** | ## Why It Works The phrase **"What you remember is WRONG"** creates psychological doubt in the model's confidence, forcing it to actually read the docs rather than trusting its pre-training. Key elements: - **"STOP"** - Attention grabber, interrupts default behavior - **"WRONG for this project"** - Creates doubt without claiming universal wrongness - **"Always search docs and read"** - Actionable instruction - **"before any task"** - Applies to everything, not just code writing ## Full Suite Results - The new indirect-proxy eval passed consistently
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )