Evaluation
Benchmark Leaderboard
Reproducible ablation results on scifact (50 queries, 5,183 corpus docs). Ranked by MRR.
Embedder: mock-text · Run: 6/18/2026, 8:00:42 PM · Seed: 42
Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks Track.
Mock embedder: dense/hybrid scores are not semantically meaningful. BM25 sparse retrieval reflects real SciFact IR performance. Use --embedder minilm for full dense/hybrid benchmarks.
| Rank | Config | Strategy | MRR | Recall@5 | Faithfulness | p50 (ms) |
|---|---|---|---|---|---|---|
| #1 | bm25-semantic | bm25 | 73.5% | 82.0% | 43.4% | 35.85 |
| #2 | dense-fixed | dense | 0.0% | 0.0% | 40.0% | 28.58 |
| #3 | hybrid-semantic | hybrid | 0.0% | 0.0% | 40.1% | 1109.20 |
| #4 | hybrid-fixed-xenc | hybrid | 0.0% | 0.0% | 40.1% | 1333.73 |
Reproduce: mosaic-benchmark --dataset scifact --max-samples 50 --embedder mock-text --output benchmarks/results