I Benchmarked 19 Retrieval Pipelines Head-to-Head and the Results Were Surprisingly Honest

I got tired of retrieval papers claiming their pipeline is "state-of-the-art" without significance tests, without telling you what dataset they ran on, and without reproducible code. So I built raven-retrieval -- 19 retrieval pipelines, real BEIR datasets, Bonferroni-corrected bootstrap significance tests, all the numbers whether they look good or not. MIT licensed, open source.

What It Does

The core question: does ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval over simpler approaches? To answer that properly you need the same datasets, same chunking, same metrics, same significance tests for every pipeline. Otherwise you're comparing apples to oranges.

The framework implements 19 pipelines and benchmarks them all on BEIR datasets (SciFact, HotpotQA) on equal footing. No cherry-picking.

The 19 Pipelines (Quick Summary)