HAKARI-Bench：统一条件下比较检索架构与效率设置的轻量级基准

AI 导读

HAKARI-Bench 是一个轻量级检索基准，将现有检索套件重建为小型数据集（Nano-sets），涵盖 35 个基准、551 个任务和 43 种语言，采用统一格式实现模型无关比较。它支持 BM25、稠密、稀疏、晚交互和重排序五种检索家族及其效率变体（降维、量化等）在同一条件下对比。在 55 个模型上，整体排名与 MTEB retrieval v2、MMTEB v2 retrieval 及 English BEIR（完整版）的 Spearman 相关系数均高于 0.97。HAKARI-Bench 不取代全面评测，而是用于快速模型选择、回归检测和探索质量-效率帕累托前沿。代码、数据和排行榜以 MIT 许可证开源。

推荐理由

有了这个轻量级基准，做检索的开发者不用再跑整套 MTEB 就能快速筛选嵌入模型和效率配置，而且排名与完整评测高度一致，是工程选型的高性价比工具。

正文

View PDF HTML (experimental)

Abstract:With the rapid spread of retrieval-augmented generation and semantic search, choosing the right embedding and retrieval configuration is increasingly hard. Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings--dimensionality reduction, quantization, reranking--across many models under identical conditions. We present HAKARI-Bench, a lightweight benchmark that reconstructs existing retrieval suites into small datasets (Nano-sets): 35 benchmarks and 551 tasks across 43 languages in a unified format, enabling same-condition, model-agnostic comparison of five retrieval families (BM25, dense, sparse, late interaction, rerankers) and their efficiency variants. Across 55 models, its overall ranking reproduces the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) at Spearman >0.97. HAKARI-Bench does not replace full evaluation; it enables rapid model selection, regression detection, and reading the quality-efficiency Pareto frontier. Code, data, and leaderboard are released under the MIT license.

Comments:	48 pages. Code and leaderboard: this https URL this https URL
Subjects:	Information Retrieval (cs.IR); Computation and Language (cs.CL)
Cite as:	arXiv:2606.22778 [cs.IR]
	(or arXiv:2606.22778v1 [cs.IR] for this version)
	https://doi.org/10.48550/arXiv.2606.22778 arXiv-issued DOI via DataCite

Submission history

From: Yuichi Tateno [view email]
[v1] Mon, 22 Jun 2026 02:42:06 UTC (924 KB)