How to Reproduce 1000 TPS Results from Paper & Ablation Study Details

[JamesYen220]

Environment

• Hardware: Single A100-40GB GPU
• Model: LLaDA 1.5 (demo in README) / LLaDA-MoE-7B-A1B-Instruct-fused (benchmark)

Current Results

We tested the following configurations:

• README demo: 40.15 TPS
• benchmarks/benchmark.py: 47.29 TPS
• benchmarks/benchmark.py with tp=2: 22 TPS

Questions

1. Reproduction Setup

What is the exact hardware and software setup used to achieve the 1000 TPS reported in the paper?

2. Ablation Studies

Could you provide details on the individual contribution of each design component to the TPS improvement? Specifically:

• Decoding strategy
• Model architecture/selection
• Distillation techniques
• Other optimizations mentioned in the paper

3. Reproduction Guide

Are there additional instructions or configurations needed to reproduce the paper's results?

~/dInfer$ python demo.py 
INFO 10-15 06:43:07 [__init__.py:244] Automatically detected platform cuda.
WARNING 10-15 06:43:12 [config.py:4703] Current vLLM config is not set.
WARNING 10-15 06:43:17 [config.py:4703] Current vLLM config is not set.
INFO 10-15 06:43:17 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 10-15 06:43:58 [fused_moe.py:683] Using default MoE config. Performance might be sub-optimal! Config file not found at /home/tonywei/miniconda3/envs/dinfer/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=64,N=1024,device_name=NVIDIA_A100-PCIE-40GB.json
Here are all the prime numbers between 1 and 100:

2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97

There are **25 prime numbers** in this range.<|role_end|>
Tokens per second (TPS): 40.15 tokens/sec
[rank0]:[W1015 06:44:01.228481383 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

~/dInfer$ python benchmarks/benchmark.py
INFO 10-15 06:49:19 [__init__.py:244] Automatically detected platform cuda.
Namespace(model_name='/nfsdata/sjtu/huggingface/LLaDA-1.5/', input_data=None, gpu='0', batch_size=1, num_test_iter=2, gen_len=1024, block_length=64, threshold=0.9, exp_name='exp', cache='', tp=False, sliding=False, prefix_look=0, after_look=0, warmup_steps=1)
started 1 0 0 Namespace(model_name='/nfsdata/sjtu/huggingface/LLaDA-1.5/', input_data=None, gpu='0', batch_size=1, num_test_iter=2, gen_len=1024, block_length=64, threshold=0.9, exp_name='exp', cache='', tp=False, sliding=False, prefix_look=0, after_look=0, warmup_steps=1)
rank=0, world size=1
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 119.78it/s]
prompt len: 50 , total len: 1074
[rank0]:[W1015 06:49:56.166279944 ProcessGroupNCCL.cpp:4715] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can pecify device_id in init_process_group() to force use of a particular device.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:15<00:00,  7.80s/it]
Iter: 1, Forward: 158, cache updates: 0, Time: 15.60623288154602, FPS: 10.124160083938708, TPS: 47.28879836675169
To determine how many kilometers Lily can run in 8 hours, we need to break down the problem into two parts: the distance she runs in the first 4 hours and the distance she runs in the next 4 hours.

**Step 1: Calculate the distance Lily runs in the first 4 hours.**

Lily runs at a speed of 12 kilometers per hour for 4 hours. The distance she runs in the first 4 hours is:
\[
\text{Distance} = \text{Speed} \times \text{Time} = 12 \, \text{km/h} \times 4 \, \text{h} = 48 \, \text{km}
\]

**Step 2: Calculate the distance Lily runs in the next 4 hours.**

After the first 4 hours, Lily runs at a speed of 6 kilometers per hour for the next 4 hours. The distance she runs in the next 4 hours is:
\[
\text{Distance} = \text{Speed} \times \text{Time} = 6 \, \text{km/h} \times 4 \, \text{h} = 24 \, \text{km}
\]

**Step 3: Calculate the total distance Lily runs in 8 hours.**

To find the total distance, we add the distance run in the first 4 hours to the distance run in the next 4 hours:
\[
\text{Total Distance} = 48 \, \text{km} + 24 \, \text{km} = 72 \, \text{km}
\]

Therefore, the total distance Lily can run in 8 hours is \boxed{72} kilometers.<|eot_id|>

~/dInfer$ python benchmarks/benchmark.py --gpu 0,1
INFO 10-15 07:11:21 [__init__.py:244] Automatically detected platform cuda.
Namespace(model_name='/nfsdata/sjtu/huggingface/LLaDA-1.5/', input_data=None, gpu='0,1', batch_size=1, num_test_iter=2, gen_len=1024, block_length=64, threshold=0.9, exp_name='exp', cache='', tp=False, sliding=False, prefix_look=0, after_look=0, warmup_steps=1)
WARNING 10-15 07:11:37 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: [Report of increased memory overhead during cudagraph capture with nccl >= 2.19 · Issue #1234 · NVIDI](https://github.com/NVIDIA/nccl/issues/1234)
INFO 10-15 07:11:38 [__init__.py:244] Automatically detected platform cuda.
WARNING 10-15 07:11:38 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: [Report of increased memory overhead during cudagraph capture with nccl >= 2.19 · Issue #1234 · NVIDI](https://github.com/NVIDIA/nccl/issues/1234)
INFO 10-15 07:11:38 [__init__.py:244] Automatically detected platform cuda.
started 2 1 1 Namespace(model_name='/nfsdata/sjtu/huggingface/LLaDA-1.5/', input_data=None, gpu='0,1', batch_size=1, num_test_iter=2, gen_len=1024, block_length=64, threshold=0.9, exp_name='exp', cache='', tp=False, sliding=False, prefix_look=0, after_look=0, warmup_steps=1)
started 2 0 0 Namespace(model_name='/nfsdata/sjtu/huggingface/LLaDA-1.5/', input_data=None, gpu='0,1', batch_size=1, num_test_iter=2, gen_len=1024, block_length=64, threshold=0.9, exp_name='exp', cache='', tp=False, sliding=False, prefix_look=0, after_look=0, warmup_steps=1)
rank=1, world size=2
rank=0, world size=2
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 65.70it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 36.60it/s]
prompt len: 50 , total len: 1074
prompt len: 50 , total len: 1074
/home/miniconda3/envs/dinfer/lib/python3.12/site-packages/torch/_inductor/lowering.py:7007: UserWarning: 
Online softmax is disabled on the fly since Inductor decides to
split the reduction. Cut an issue to PyTorch if this is an
important use case and you want to speed it up with online
softmax.

[zheng-da]

we benchmarked our framework on 8x h800 GPU. we'll release inference speed on smaller GPUs.

[lilijiangg]

Same issue here. LLaDA-MoE README demo achieves 58.11 TPS on single H20 GPU. The output is:

INFO 10-14 08:46:31 [__init__.py:244] Automatically detected platform cuda.
WARNING 10-14 08:46:34 [config.py:4703] Current vLLM config is not set.
WARNING 10-14 08:46:34 [config.py:4703] Current vLLM config is not set.
INFO 10-14 08:46:34 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
WARNING 10-14 08:46:55 [fused_moe.py:683] Using default MoE config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=64,N=1024,device_name=NVIDIA_H20.json
Forward: 68, Time: 2.1853535175323486, FPS: 31.116247076026422, TPS: 58.114167333166996
Let's break the problem into two parts:

1. **First 4 hours**: Lily runs at 12 km/h  
   Distance = Speed × Time  
   = 12 km/h × 4 h = **48 kilometers**

2. **Next 4 hours**: Lily runs at 6 km/h  
   Distance = 6 km/h × 4 h = **24 kilometers**

Now, add both distances:  
48 km + 24 km = **72 kilometers**

### ✅ Answer: **72 kilometers**<|role_end|>
[rank0]:[W1014 08:46:58.866211905 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

[jiangzizi]

Same result on one single A100.

Also, will you kindly provide script to reproduce the 1100 TPS result on 8xH800? That will help!

BTW, I can only get < 100 TPS using benchmarks/benchmark_dataset_fastdllm.py script with 4* A100-80GB. Is that correct?

sample 0, time: 7.252451419830322, generated: 1023, tpf: 6.053254437869822, tps: 141.05575353497977, fps: 23.30246563774348
Forward: 169, Time: 7.266731262207031, FPS: 23.256668657224917(23.30246563774348), TPS: 140.77068570794538(141.05575353497977), TPF: 6.053254437869822(6.053254437869822)
  0%|                                         | 1/541 [00:07<1:05:24,  7.27s/it]sample 1, time: 16.873101711273193, generated: 1023, tpf: 1.6770491803278689, tps: 60.629042454981295, fps: 36.15221495360566
Forward: 779, Time: 24.140734672546387, FPS: 32.26910715043712(29.72734029567457), TPS: 84.75220193810749(100.84239799498053), TPF: 2.6264441591784338(3.8651518090988457)
  0%|▏                                        | 2/541 [00:24<1:56:02, 12.92s/it]sample 2, time: 13.689119100570679, generated: 1023, tpf: 2.07085020242915, tps: 74.7308860770561, fps: 36.08705544678955
Forward: 1273, Time: 37.83050036430359, FPS: 33.65009554735733(31.847245346046225), TPS: 81.12482413122883(92.13856068900571), TPF: 2.410840534171249(3.2670512735422803)
  1%|▏                                        | 3/541 [00:37<1:58:59, 13.27s/it]

[zheng-da]

• To answer all the questions above, our results on the paper were collected from https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 8x H800.
  This benchmark script takes benchmark data files. We'll release the data files soon.
• You can potentially use use https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_ep.py to have a glance at the inference speed of dInfer. This script generates tokens for a task collected from gsm8k.
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset_fastdllm.py is the script to reproduce Fast-dLLM results
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark.py is to run dInfer with LLaDA, instead of LLaDA-MoE.

[zheng-da]

The demo code in Readme is to show how to use the current dInfer API. We'll provide a higher-level API in the next release for easy reproduction of dInfer performance.

We will also release performance numbers of dInfer on other hardware, such as H20.

[mZhenz]

we benchmarked our framework on 8x h800 GPU. we'll release inference speed on smaller GPUs.

@zheng-da 大佬，论文中qwen2.5-3B的TPS也是8xh800测的吗？vllm配置是tp=8？推理也是batch_size=1吗？

[SolarisAdams]

we benchmarked our framework on 8x h800 GPU. we'll release inference speed on smaller GPUs.

@zheng-da 大佬，论文中qwen2.5-3B的TPS也是8xh800测的吗？vllm配置是tp=8？推理也是batch_size=1吗？

是的

[jiangzizi]

• To answer all the questions above, our results on the paper were collected from https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 8x H800.
  This benchmark script takes benchmark data files. We'll release the data files soon.
• You can potentially use use https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_ep.py to have a glance at the inference speed of dInfer. This script generates tokens for a task collected from gsm8k.
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset_fastdllm.py is the script to reproduce Fast-dLLM results
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark.py is to run dInfer with LLaDA, instead of LLaDA-MoE.

Hi team:
I tried to reproduce the result using https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 4x A100. I use IFEval as dataset.

I use the following script to start inference

python benchmarks/benchmark.py --config 11

Here is the result

Forward: 112340, Time: 1231.8520724773407, FPS: 91.1960149355242(97.88496295541188), TPS: 189.26785545858525(205.0103223574363), TPF: 2.075396118924693(2.1727105518606447)

I use the official code from https://github.com/google-research/google-research/tree/master/instruction_following_eval and get a score of 54.3438.
For reference, the paper reported TPS: 2.32, TPS: 444.51 and score: 58.78. I think this difference might come from GPU number and GPU type. I am convinced that reaching even higher TPS can be possible with dInfer framework.

Thank you for your great work!

[ZinYY]

Hi team,

I’m encountering the same issue. I’m running on 4×H100 GPUs using benchmark_dataset.py with the openai/gsm8k dataset and the LLaDA-MoE-7B-A1B-Instruct-fused model.

Even though I’ve closely followed the paper’s configuration, my performance results are significantly lower. My current configuration is:

args.cache = 'dual'
args.parallel_decoding = 'hierarchy_faster'
args.prefix_look = 16
args.after_look = 16
args.cont_weight = 0.3
args.threshold = 0.92
args.low_threshold = 0.62
args.warmup_times = 4
args.use_credit = True
args.use_tp = True

However, my observed results are:

avg_fps = 37.09, avg_nfe = 94.76, avg_tpf = 3.15, avg_tps = 116.91

These are notably lower than the reported values in Table 1 (LLaDA-MoE + dInfer on GSM8K: TPF = 3.42, TPS = 682.9, especially the TPS).

Could you please advise if there are additional hyperparameters, environment settings, or optimization flags that might explain this gap?

[zheng-da]

• To answer all the questions above, our results on the paper were collected from https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 8x H800.
  This benchmark script takes benchmark data files. We'll release the data files soon.
• You can potentially use use https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_ep.py to have a glance at the inference speed of dInfer. This script generates tokens for a task collected from gsm8k.
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset_fastdllm.py is the script to reproduce Fast-dLLM results
• https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark.py is to run dInfer with LLaDA, instead of LLaDA-MoE.

Hi team: I tried to reproduce the result using https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 4x A100. I use IFEval as dataset.

I use the following script to start inference

python benchmarks/benchmark.py --config 11

Here is the result

Forward: 112340, Time: 1231.8520724773407, FPS: 91.1960149355242(97.88496295541188), TPS: 189.26785545858525(205.0103223574363), TPF: 2.075396118924693(2.1727105518606447)

I use the official code from https://github.com/google-research/google-research/tree/master/instruction_following_eval and get a score of 54.3438. For reference, the paper reported TPS: 2.32, TPS: 444.51 and score: 58.78. I think this difference might come from GPU number and GPU type. I am convinced that reaching even higher TPS can be possible with dInfer framework.

Thank you for your great work!

Thanks for your feedback. We didn't benchmark our framework on A100 GPUs before. A100 is weaker than H800 or H20. I'm not sure how fast the framework can run on this GPU. Let's see if we can find A100 and benchmarks our framework. It seems to me that FPS is reasonable. A100 is much weaker than H20/H800, I'm not surprised that it runs (in terms of FPS) twice as slow as H800.

In terms of TPF, IfEval benchmark is slower than other benchmarks such as humaneval.

One more thing is that we used our internal benchmarks, which is a little different from the public IfEval (the main difference is the prompt). So the score can be a little different.

We are working on releasing some of the benchmarks that can reproduce the scores on the paper.

[zheng-da]

Hi team,

I’m encountering the same issue. I’m running on 4×H100 GPUs using benchmark_dataset.py with the openai/gsm8k dataset and the LLaDA-MoE-7B-A1B-Instruct-fused model.

Even though I’ve closely followed the paper’s configuration, my performance results are significantly lower. My current configuration is:

args.cache = 'dual'
args.parallel_decoding = 'hierarchy_faster'
args.prefix_look = 16
args.after_look = 16
args.cont_weight = 0.3
args.threshold = 0.92
args.low_threshold = 0.62
args.warmup_times = 4
args.use_credit = True
args.use_tp = True
However, my observed results are:

avg_fps = 37.09, avg_nfe = 94.76, avg_tpf = 3.15, avg_tps = 116.91

These are notably lower than the reported values in Table 1 (LLaDA-MoE + dInfer on GSM8K: TPF = 3.42, TPS = 682.9, especially the TPS).

Could you please advise if there are additional hyperparameters, environment settings, or optimization flags that might explain this gap?

your result indeed looks weird. do you mind sharing the log printed by dInfer?

[vaibhavjindal]

• To answer all the questions above, our results on the paper were collected from https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 8x H800.
  This benchmark script takes benchmark data files. We'll release the data files soon.

@zheng-da Wanted to follow up on this. Would it be possible for you to share how to create some dummy data for this? What is expected in the json file used for this?

Thanks a lot for the help!

[zheng-da]

#20 (comment)

does this help?

[lundu28]

• To answer all the questions above, our results on the paper were collected from https://github.com/inclusionAI/dInfer/blob/master/benchmarks/benchmark_dataset.py on 8x H800.
  This benchmark script takes benchmark data files. We'll release the data files soon.

@zheng-da Wanted to follow up on this. Would it be possible for you to share how to create some dummy data for this? What is expected in the json file used for this?

Thanks a lot for the help!

Hi @vaibhavjindal, you can follow this response to build datasets.

Thanks!