HuggingFace Daily Papers（社区热门论文）· HuggingFace Daily Papers（社区热门论文）·· 8 天前AI 评分45
Jev 心里想"我不知道"却不说出口：Sys1Cal-v1 概率校准数据集发布
AI 导读
针对 System One 模型 Jev 的概率校准承诺缺乏公开测试的问题，研究者发布 Sys1Cal-v1 数据集，用已知精确概率 P(A) 的真/假命题，通过 Noul、Choice、Score 三个原语查询并以全变差距离评估。
正文
Abstract:The appearance of Jev marked the era of System One Models, foundation models that return structured decisions with probability distributions rather than text. Aside from low cost and great speed, Jev's central promise is that these probabilities are calibrated: such claim is not backed by any public test and available external benchmarks evaluate confidence calibration, not whether every returned option probability has the right numerical meaning. To tackle this issue, we introduce Sys1Cal-v1, a dataset of True/False questions about a proposition $A$ for which the exact probability $P(A)$ is known by construction. Each item is queried through the three Jev primitives - Noul, Choice and Score - and evaluated by total variation distance from the ground-truth distribution, which can be used to estimate a soft accuracy of System One Models.
We showcase the utility of Sys1Cal-v1 as a benchmark dataset by evaluating Jev and SemIf, an open-source Choice-style baseline. In this work, however, we focus even more deeply on Jev, by studying the calibration of its Score and Choice answers. In particular, we discover a peculiar behaviour that can be explained by assuming that Jev suppresses a third truth value, going beyond True and False. In other words, in \texttt{Choice} answers, $P(A)$ and $P(\neg A)$ are presented as if $P(A)+P(\neg A)=1$, while a term $P(U)\neq0$ is missing in the sum. Recovering $P(U)$ leads to an improvement of median soft accuracy in \texttt{Choice} answers from $0.771$ to $0.978$, suggesting that, even in binary decisions, Jev wants to answer with a third option:``I don't know''.
| Subjects: | Artificial Intelligence (cs.AI); Logic in Computer Science (cs.LO); Systems and Control (eess.SY) |
| Cite as: | arXiv:2609.35342 [cs.AI] |
| (or arXiv:2609.35342v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.35342 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Riccardo Porcedda [view email]
[v1]
Mon, 28 Sep 2026 14:59:07 UTC (695 KB)
来源：HuggingFace Daily Papers（社区热门论文） · arxiv.org