Cannot reproduce the results reported in the paper

[KKKKristen]

Hi, thank you very much for the impressive work and for open-sourcing the code and checkpoints!
I am currently trying to reproduce the results reported in the paper, but I am encountering some discrepancies and would appreciate your guidance.

I download the official checkpoint you provided from Huggingface: reasonrag/Qwen2.5-7B-Instruct-ReasonRAG

I ran the code inference.py using this checkpoint. However, the index building instructions do not seem to provide the processed indexes/wiki18.jsonl file.

# Download wikidump
wget https://archive.org/download/enwiki-20181220/enwiki-20181220-pages-articles.xml.bz2

# Build index
python -m flashrag.retriever.index_builder \
  --retrieval_method bge \
  --model_path /BAAI/bge-base-en-v1.5 \
  --corpus_path indexes/wiki18.jsonl \
  --save_dir indexes/ \
  --use_fp16 \
  --max_length 512 \
  --batch_size 256 \
  --pooling_method mean \
  --faiss_type Flat 

I then used the setting from SearchR1 with e5 as the retriever and return topk=3 results for each query (The wiki corpus and index are set as SearchR1 version). The other parameters remain as default (e.g., max_iter=8). Below are my reproduced results:

With this setup, my reproduced results show a non-negligible gap compared to the numbers reported in the paper.

May I kindly ask for the detailed inference parameter settings for reproduce the results in the paper? Is it possible to also provide the code or already processed wiki18.jsonl for bge retriever?

Thanks!

[KKKKristen]

Hi, I noticed that in the appendix of the paper, you mentioned that the corpus/passages from these evaluation datasets are augmented to the wikidump 2018.

This seems to differ from the corpus used in SearchR1, where the retriever index is built on Wiki 2018 only. I guess that may be one reason why it is difficult to reproduce the your paper numbers when directly using the SearchR1 corpus and index (SearchR1 provided wiki18.jsonl and e5 embeddings).
Another possible factor might be a train–test corpus mismatch, i.e., training with (wiki + dataset passages) while evaluating with a wiki-only corpus。

May I kindly ask would it be possible to release the processed corpus file, e.g. (wiki18 + dataset_passages).jsonl, or provide the preprocessing scripts used to construct it?

I guess the corpus might have non-neglectable influence to the performance.

As a small empirical check, I tried running the official SearchR1 checkpoint using a corpus that contains only the passages from the MuSiQue benchmark, which includes all passages in the training and test set (without Wiki 2018). Under this setting, I observed that the exact-match score can exceed 0.3, compared to 0.1X (<0.2) reported in the SearchR1 paper when using Wiki-only corpus.

Thanks again for the impressive work and for your time. Any clarification or guidance on the corpus construction would be greatly appreciated.