Skip to content

[Do Not Merge] Optimizations on Qwen3-Next GatedDeltaNet w/ Kernel & XProf Agent #3077

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Rohan-Bierneni wants to merge 27 commits into
base: main
Choose a base branch
from
Open

[Do Not Merge] Optimizations on Qwen3-Next GatedDeltaNet w/ Kernel & XProf Agent #3077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
5872358
Script to test gdn changes
Rohan-Bierneni Feb 3, 2026
121678b
Update dummy configs to align with q3-next
Rohan-Bierneni Feb 4, 2026
09f85a0
Update tflops calc to align with WY-optimized GDN
Rohan-Bierneni Feb 4, 2026
c644681
remove mixed precision
Rohan-Bierneni Feb 10, 2026
0fb6d5b
Update config for chunk size
Rohan-Bierneni Feb 10, 2026
3b28ad7
update dtype
Rohan-Bierneni Feb 10, 2026
6c0fb21
Add NaN test in backward pass
Rohan-Bierneni Feb 10, 2026
812ffc7
Fix exploding gradient in gdn
Rohan-Bierneni Feb 10, 2026
3a362c6
Reintroduce mixed precision
Rohan-Bierneni Feb 10, 2026
a1d66fb
typo in bloat16
Rohan-Bierneni Feb 10, 2026
cb46652
typo fixed
Rohan-Bierneni Feb 10, 2026
719a3d8
convert to float
Rohan-Bierneni Feb 10, 2026
1593b0e
update WY matrices to be bf16
Rohan-Bierneni Feb 10, 2026
737c533
test pallas kernel for gdn
Rohan-Bierneni Feb 10, 2026
d475ed5
wrong api name
Rohan-Bierneni Feb 10, 2026
dcea418
fix function positional args
Rohan-Bierneni Feb 10, 2026
f94b5e0
fix pallas code
Rohan-Bierneni Feb 10, 2026
a9d7495
fix tensor indexing error
Rohan-Bierneni Feb 10, 2026
7d2e9af
only optimize forward pass
Rohan-Bierneni Feb 10, 2026
4f7ebf6
update pallas code
Rohan-Bierneni Feb 10, 2026
e167840
use float mask
Rohan-Bierneni Feb 10, 2026
bcabdc4
fix function returns
Rohan-Bierneni Feb 10, 2026
7461955
add shardmap to kernel
Rohan-Bierneni Feb 10, 2026
150ccda
update with kernel agent suggestions
Rohan-Bierneni Feb 10, 2026
eb3d7e6
fix matrix indexing
Rohan-Bierneni Feb 10, 2026
df27a4a
fix matrix indexing
Rohan-Bierneni Feb 10, 2026
db5b69e
mask before exp
Rohan-Bierneni Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading