Evaluation on FinSearchComp Benchmark?

[panademo]

Hi MiroMind, thank you so much for open-sourcing such an impressive system! I wanted to congratulate you on being #1 on ByteDance Seed's FutureX benchmark by such a wide margin, and I also wanted to ask if you'd kindly consider evaluating MiroFlow on ByteDance Seed's latest BrowseComp-inspired benchmark: "FinSearchComp, the first expert-level benchmark for financial search & reasoning". It would be great to see how well MiroFlow performs on this expert-driven benchmark, and I'm sure it might even be able to score #1 just like it did on FutureX🤞🙏!

[BinWang28]

Great! We are definitely interested.

[panademo]

@BinWang28 Amazing! Cannot wait to see the results, and hoping MiroMind dominates on FinSearchComp🚀!
And even if early versions of MiroFlow are shown to potentially struggle on this benchmark, I preemptively suspect that it may be due to GPT-5/Grok4 Search having very good PDF-parsing as well as graph/chart ingestion, given that most financial information necessary to score highly on this benchmark is likely found in PDFs(quarterly, annual reports...etc), specifically those from the SEC(edgar) website (at least regarding American securities).
But regardless, the initial results will be great to see when they are released🙏

[JubSteven]

@panademo Hi, could you clarify how we should submit results for evaluation? I’ve checked the website and dataset, and I noticed that questions from subtask T1 (Time-Sensitive Data Fetching) require live testing. However, I wasn’t able to find any instructions on how to submit the corresponding test results. Thanks!

[panademo]

@panademo Hi, could you clarify how we should submit results for evaluation? I’ve checked the website and dataset, and I noticed that questions from subtask T1 (Time-Sensitive Data Fetching) require live testing. However, I wasn’t able to find any instructions on how to submit the corresponding test results. Thanks!

@JubSteven Hi, apologies for the very late response. I’m still not entirely sure how to properly evaluate the T1 portion of FinSearchComp. I did notice that, according to M2’s benchmark results, MiniMax was able to evaluate on FinSearchComp-Global.
However, it may be more practical to follow the approach Moonshot used for Kimi K2 Thinking and evaluate only on the T3 subset. T3 is the most challenging part of FinSearchComp and does not require the live-testing setup that T1 does. Since Kimi has already evaluated several SOTA models on T3, MiroFlow’s results could be directly compared against theirs, providing a clear picture of MiroFlow’s performance. 🙏