Proposal: add a short “RAG / reranking troubleshooting checklist” (docs only) #66
Description
Hi Rankify team, thanks for releasing this toolkit – the combination of 40+ pre-retrieved datasets, multiple retrieval techniques, rerankers and RAG methods is really valuable for research and teaching.
I have been working on a compact failure-mode checklist for RAG-style pipelines and recently contributed a robustness entry to Harvard MIMS Lab’s ToolUniverse. When using Rankify, it is very easy for users to confuse:
- retrieval quality vs. reranking quality vs. generation quality
- configuration issues vs. algorithmic limitations
So I’d like to propose a very small, docs-only addition.
Scope (docs only, no code changes)
- Add one markdown doc under
docs/(or in the place you prefer), for example:troubleshooting_rag_and_reranking.md
- Optionally, add a link from the README “Getting started” or “Usage” section.
Suggested outline (kept minimal)
- Common failure patterns when using Rankify:
- good retrieval, bad reranking
- bad retrieval, good reranking, but still poor final answers
- configuration mistakes (wrong index, wrong dataset split, stale cache)
- evaluation mismatch (metric suggests improvement, but answers look worse)
- For each pattern:
- what to check (dataset, split, index path, model name, seed)
- example commands / configuration snippets to reproduce
- A short checklist for opening issues:
- dataset id, retrieval setting, reranker, RAG method, config + log excerpt
Motivation
- Rankify is often used as a backbone to compare many RAG and reranking methods in one place.
- A shared troubleshooting checklist can reduce duplicated “it does not work” issues and make experiment reports more comparable.
- This is a low-risk documentation change and should be easy to adjust or extend later.
If you think this is useful and in-scope, I’m happy to draft the doc as a PR following your documentation style.
Thank you for considering.