Merge q,kv by WANDY666 · Pull Request #1199 · ModelTC/LightLLM

WANDY666 · 2026-02-02T04:35:25Z

No description provided.

…wen3

gemini-code-assist · 2026-02-02T04:35:46Z

Summary of Changes

Hello @WANDY666, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates a collection of newly generated autotuned kernel configurations for Triton 3.5.1, specifically targeting NVIDIA H200 GPUs. These additions aim to enhance the performance of critical deep learning operations within LightLLM, such as grouped matrix multiplications, MoE layer components, and activation functions, by leveraging optimal kernel parameters for various input dimensions and data types like bfloat16.

Highlights

New Triton Kernel Configurations: This pull request introduces a set of newly autotuned kernel configurations for Triton 3.5.1, specifically optimized for NVIDIA H200 GPUs.
Performance Optimization: The added configurations are designed to enhance the performance of various critical computational kernels within LightLLM.
Specific Kernels Covered: Optimizations are included for grouped_matmul, moe_align_fused, moe_sum_reduce, and silu_and_mul_fwd operations.
Bfloat16 Support: Many of the new configurations specify torch.bfloat16 as the output data type, indicating improved support and performance for mixed-precision operations.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds several autotuning kernel configuration files for NVIDIA H200 GPUs. The changes appear to be generated configurations for various Triton kernels related to Mixture-of-Experts (MoE) models. My main feedback is about the structure of these JSON files. The keys are not sorted, which impacts readability and maintainability. I've provided suggestions to sort the keys numerically in each file. This will make them easier to inspect and manage in the future.

gemini-code-assist · 2026-02-02T04:37:23Z

..._routed_weight=true,out_dtype=torch.bfloat16,topk_num=1,use_fp8_w8a8=false}_NVIDIA_H200.json

+{
+  "1024": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 64,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "128": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 16,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "2048": {
+    "BLOCK_SIZE_K": 32,
+    "BLOCK_SIZE_M": 32,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 16,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "256": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "512": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 4,
+    "num_warps": 4
+  },
+  "64": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "8": {
+    "BLOCK_SIZE_K": 32,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 64,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "800": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 32,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "8192": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 64,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 32,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  }
+}


The keys in this JSON configuration file are not sorted, which makes it difficult to read and maintain. For better readability, it's recommended to sort the keys numerically. This is especially helpful when manually inspecting or debugging the autotuning configurations.

{ "8": { "BLOCK_SIZE_K": 32, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 64, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "64": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "128": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 16, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 }, "256": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "512": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 4, "num_warps": 4 }, "800": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 32, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "1024": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 64, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "2048": { "BLOCK_SIZE_K": 32, "BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 16, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 }, "8192": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 32, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 } }

gemini-code-assist · 2026-02-02T04:37:23Z

...routed_weight=false,out_dtype=torch.bfloat16,topk_num=8,use_fp8_w8a8=false}_NVIDIA_H200.json

+{
+  "1": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 64,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 4,
+    "num_warps": 4
+  },
+  "100": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "1024": {
+    "BLOCK_SIZE_K": 32,
+    "BLOCK_SIZE_M": 64,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 64,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "128": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 32,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 8
+  },
+  "16": {
+    "BLOCK_SIZE_K": 64,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 1,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "256": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 32,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 16,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "32": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 64,
+    "GROUP_SIZE_M": 16,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 4
+  },
+  "64": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 32,
+    "NEED_TRANS": false,
+    "num_stages": 2,
+    "num_warps": 4
+  },
+  "8": {
+    "BLOCK_SIZE_K": 128,
+    "BLOCK_SIZE_M": 16,
+    "BLOCK_SIZE_N": 128,
+    "GROUP_SIZE_M": 32,
+    "NEED_TRANS": false,
+    "num_stages": 3,
+    "num_warps": 8
+  }
+}


The keys in this JSON configuration file are not sorted, which makes it difficult to read and maintain. For better readability, it's recommended to sort the keys numerically. This is especially helpful when manually inspecting or debugging the autotuning configurations.

{ "1": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 64, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 4, "num_warps": 4 }, "8": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 32, "NEED_TRANS": false, "num_stages": 3, "num_warps": 8 }, "16": { "BLOCK_SIZE_K": 64, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 }, "32": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 64, "GROUP_SIZE_M": 16, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 }, "64": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 32, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "100": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 1, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 }, "128": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 16, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 32, "NEED_TRANS": false, "num_stages": 2, "num_warps": 8 }, "256": { "BLOCK_SIZE_K": 128, "BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 16, "NEED_TRANS": false, "num_stages": 2, "num_warps": 4 }, "1024": { "BLOCK_SIZE_K": 32, "BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "GROUP_SIZE_M": 64, "NEED_TRANS": false, "num_stages": 3, "num_warps": 4 } }

gemini-code-assist · 2026-02-02T04:37:23Z

...une_kernel_configs/triton_3.5.1/NVIDIA_H200/moe_align_fused:v1/{topk_num=8}_NVIDIA_H200.json

+{
+  "1": {
+    "BLOCK_SIZE": 256,
+    "num_warps": 4
+  },
+  "100": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  },
+  "1024": {
+    "BLOCK_SIZE": 256,
+    "num_warps": 4
+  },
+  "128": {
+    "BLOCK_SIZE": 256,
+    "num_warps": 8
+  },
+  "16": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  },
+  "256": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  },
+  "32": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  },
+  "64": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  },
+  "8": {
+    "BLOCK_SIZE": 128,
+    "num_warps": 8
+  }
+}


The keys in this JSON configuration file are not sorted, which makes it difficult to read and maintain. For better readability, it's recommended to sort the keys numerically. This is especially helpful when manually inspecting or debugging the autotuning configurations.

{ "1": { "BLOCK_SIZE": 256, "num_warps": 4 }, "8": { "BLOCK_SIZE": 128, "num_warps": 8 }, "16": { "BLOCK_SIZE": 128, "num_warps": 8 }, "32": { "BLOCK_SIZE": 128, "num_warps": 8 }, "64": { "BLOCK_SIZE": 128, "num_warps": 8 }, "100": { "BLOCK_SIZE": 128, "num_warps": 8 }, "128": { "BLOCK_SIZE": 256, "num_warps": 8 }, "256": { "BLOCK_SIZE": 128, "num_warps": 8 }, "1024": { "BLOCK_SIZE": 256, "num_warps": 4 } }

gemini-code-assist · 2026-02-02T04:37:23Z

...200/moe_sum_reduce:v1/{hidden_dim=2048,out_dtype=torch.bfloat16,topk_num=8}_NVIDIA_H200.json

+{
+  "1": {
+    "BLOCK_DIM": 256,
+    "BLOCK_M": 2,
+    "NUM_STAGE": 2,
+    "num_warps": 8
+  },
+  "100": {
+    "BLOCK_DIM": 1024,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 1,
+    "num_warps": 8
+  },
+  "1024": {
+    "BLOCK_DIM": 1024,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 4,
+    "num_warps": 1
+  },
+  "128": {
+    "BLOCK_DIM": 1024,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 1,
+    "num_warps": 16
+  },
+  "16": {
+    "BLOCK_DIM": 128,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 1,
+    "num_warps": 2
+  },
+  "256": {
+    "BLOCK_DIM": 1024,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 4,
+    "num_warps": 2
+  },
+  "32": {
+    "BLOCK_DIM": 128,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 4,
+    "num_warps": 4
+  },
+  "64": {
+    "BLOCK_DIM": 128,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 4,
+    "num_warps": 4
+  },
+  "8": {
+    "BLOCK_DIM": 1024,
+    "BLOCK_M": 1,
+    "NUM_STAGE": 1,
+    "num_warps": 16
+  }
+}


The keys in this JSON configuration file are not sorted, which makes it difficult to read and maintain. For better readability, it's recommended to sort the keys numerically. This is especially helpful when manually inspecting or debugging the autotuning configurations.

{ "1": { "BLOCK_DIM": 256, "BLOCK_M": 2, "NUM_STAGE": 2, "num_warps": 8 }, "8": { "BLOCK_DIM": 1024, "BLOCK_M": 1, "NUM_STAGE": 1, "num_warps": 16 }, "16": { "BLOCK_DIM": 128, "BLOCK_M": 1, "NUM_STAGE": 1, "num_warps": 2 }, "32": { "BLOCK_DIM": 128, "BLOCK_M": 1, "NUM_STAGE": 4, "num_warps": 4 }, "64": { "BLOCK_DIM": 128, "BLOCK_M": 1, "NUM_STAGE": 4, "num_warps": 4 }, "100": { "BLOCK_DIM": 1024, "BLOCK_M": 1, "NUM_STAGE": 1, "num_warps": 8 }, "128": { "BLOCK_DIM": 1024, "BLOCK_M": 1, "NUM_STAGE": 1, "num_warps": 16 }, "256": { "BLOCK_DIM": 1024, "BLOCK_M": 1, "NUM_STAGE": 4, "num_warps": 2 }, "1024": { "BLOCK_DIM": 1024, "BLOCK_M": 1, "NUM_STAGE": 4, "num_warps": 1 } }

gemini-code-assist · 2026-02-02T04:37:23Z

...iton_3.5.1/NVIDIA_H200/silu_and_mul_fwd:v1/{N=192,out_dtype=torch.bfloat16}_NVIDIA_H200.json

+{
+  "1024": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 2,
+    "num_warps": 4
+  },
+  "128": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 1,
+    "num_warps": 8
+  },
+  "2048": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 1,
+    "num_warps": 1
+  },
+  "256": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 1,
+    "num_warps": 8
+  },
+  "512": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 128,
+    "NUM_STAGES": 2,
+    "num_warps": 4
+  },
+  "64": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 64,
+    "NUM_STAGES": 4,
+    "num_warps": 1
+  },
+  "8": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 64,
+    "NUM_STAGES": 4,
+    "num_warps": 1
+  },
+  "800": {
+    "BLOCK_M": 1,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 2,
+    "num_warps": 1
+  },
+  "8192": {
+    "BLOCK_M": 8,
+    "BLOCK_N": 256,
+    "NUM_STAGES": 4,
+    "num_warps": 1
+  }
+}


The keys in this JSON configuration file are not sorted, which makes it difficult to read and maintain. For better readability, it's recommended to sort the keys numerically. This is especially helpful when manually inspecting or debugging the autotuning configurations.

{ "8": { "BLOCK_M": 1, "BLOCK_N": 64, "NUM_STAGES": 4, "num_warps": 1 }, "64": { "BLOCK_M": 1, "BLOCK_N": 64, "NUM_STAGES": 4, "num_warps": 1 }, "128": { "BLOCK_M": 1, "BLOCK_N": 256, "NUM_STAGES": 1, "num_warps": 8 }, "256": { "BLOCK_M": 1, "BLOCK_N": 256, "NUM_STAGES": 1, "num_warps": 8 }, "512": { "BLOCK_M": 1, "BLOCK_N": 128, "NUM_STAGES": 2, "num_warps": 4 }, "800": { "BLOCK_M": 1, "BLOCK_N": 256, "NUM_STAGES": 2, "num_warps": 1 }, "1024": { "BLOCK_M": 1, "BLOCK_N": 256, "NUM_STAGES": 2, "num_warps": 4 }, "2048": { "BLOCK_M": 1, "BLOCK_N": 256, "NUM_STAGES": 1, "num_warps": 1 }, "8192": { "BLOCK_M": 8, "BLOCK_N": 256, "NUM_STAGES": 4, "num_warps": 1 } }

WANDY666 added 3 commits January 26, 2026 06:30

autotune

228b3b7

Merge branch 'main' of https://github.com/ModelTC/LightLLM into dev_q…

c52089c

…wen3

merge qkv

3c1313c

gemini-code-assist bot reviewed Feb 2, 2026

View reviewed changes

fix assert.

45ce36a

hiworldwzj merged commit bbdc7ba into main Feb 3, 2026
1 check passed

hiworldwzj deleted the merge_qkv branch February 3, 2026 05:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge q,kv#1199

Merge q,kv#1199
hiworldwzj merged 4 commits intomainfrom
merge_qkv

WANDY666 commented Feb 2, 2026

Uh oh!

gemini-code-assist bot commented Feb 2, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Feb 2, 2026

Uh oh!

gemini-code-assist bot Feb 2, 2026

Uh oh!

gemini-code-assist bot Feb 2, 2026

Uh oh!

gemini-code-assist bot Feb 2, 2026

Uh oh!

gemini-code-assist bot Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

WANDY666 commented Feb 2, 2026

Uh oh!

gemini-code-assist bot commented Feb 2, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants