Study: fine-tuned rerankers beat decision models on Qwen backbones up to 9B

An arXiv paper finds reranker fine-tuning beats decision fine-tuning by 5–29 nDCG@10 points on Qwen models up to 9B, with a tie at 27B.

If you're thinking about swapping your RAG reranker for one of the new decision models, a paper posted to arXiv on October 23, 2026 suggests you test first. In System One models Are Not Better Rerankers, the authors compare zero-shot, decision-fine-tuned and reranker-fine-tuned versions of six Qwen backbones, from 0.6B to 27B, on ten TREC DL and BEIR collections. Their conclusion: "decision fine-tuning never makes the better reranker."

The abstract reports that from 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, including on zero-shot collections. At 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Rivista AI's write-up names the authors as Michael Dinzinger, Kanishka Ghosh Dastidar and Michael Granitzer. It adds that every version reranked the same BM25 top-100 candidates and that the gap is statistically significant in 48 of 60 comparisons. @omarsar0 gives the same figures.

Calibration, the main pitch for decision models, holds up only partly. The paper finds decision models better calibrated than published reranker fine-tunes from 4B on when asked whether a passage answers the query. They lose that edge when the question is phrased slightly differently. Up to 9B, a reranker fine-tune still keeps better passages at a fixed threshold.

The results cover Qwen backbones and public benchmarks, so Rivista AI advises measuring on your own queries and documents before switching. If you want a dedicated reranker to compare against, Acta's guide covers open options such as bge-reranker-v2-m3, mxbai-rerank-v2 and Qwen3-Reranker. SGLang's docs show how to serve rerankers behind a /v1/rerank endpoint.

Sources