Could you provide the best parameter Settings for offline evaluation

[zhangwenzong123]

When I was conducting offline benchmarks, I found that the score of Androidcontrol could not reach the score in your paper. Could you provide your test code or parameter configuration?

[BrainWWW]

Hi zhangwenzong,

Could you please share which vLLM version and sample parameters you used, and what the final evaluation results were?

[zhangwenzong123]

Hi zhangwenzong,

Could you please share which vLLM version and sample parameters you used, and what the final evaluation results were?

vllm == 0.10.1，The parameters used follow default parameters used in runner.py and run_navi_7b.sh，The final evaluation result is about 5% lower than yours.

[BrainWWW]

We haven’t evaluated on version 0.10.1. We recommend using our suggested configuration with vLLM 0.8.3.

Could you clarify which test set you are referring to for the evaluation result?

[zhangwenzong123]

We haven’t evaluated on version 0.10.1. We recommend using our suggested configuration with vLLM 0.8.3.

Could you clarify which test set you are referring to for the evaluation result?

Testset of Androidcontrol