Question about benchmarking YAML configurations

[Nobody-Zhang]

Hi, thanks a lot for the great work and for releasing this project!

I noticed that the repository currently only includes two YAML files for benchmarking. I was wondering whether there are additional benchmarking YAML configurations available that were used to reproduce the results reported in the technical report.

If so, would it be possible to share them, or provide guidance on how to configure the benchmarks to match the reported results?

Thanks again for your excellent work!

[Nobody-Zhang]

As a follow-up, I noticed that the MBPP evaluation reports an accuracy of 0 in my experiments.

I’m using the following script:
https://github.com/inclusionAI/dInfer/blob/master/evaluations/eval_llada_mini.sh

Could you confirm whether zero accuracy is expected with this script, or if additional configuration is required for MBPP?

Thanks again!

[lundu28]

Hi @Nobody-Zhang, thank you for raising this issue. we have fixed this issue in master branch.

Thanks.