Evaluation

[Germany321]

Thank you for your excellent work. I noticed that the evaluation results for the General Baselines are significantly lower than those reported in previous works, such as RoboRefer. Can you explain the evaluation difference with previous works?

[tanhuajie]

Hey @Germany321. Our evaluation team manually verified all samples in the benchmarks. We found that some benchmarks included several incorrect cases (e.g. where2place), which were manually screened and excluded. All our evaluation results can be reproduced using our open-sourced FlagEvalMM evaluation framework. Hope it can help you!

[ChicyChen]

Dear authors, I have a follow-up question. If I use the defaut prompts in FlagEvalMM for RoboSpatial-Home, I got very low accuracy for the model RoboBrain-7B. Did you use other prompts to test the model?

[Yuheng2000]

Hi ChicyChen, could you please confirm whether the prompt explicitly specifies using absolute coordinates? Currently, RoboBrain2.0 only supports the representation of absolute coordinates.