长度惩罚使思维链更难被监控

View PDF HTML (experimental)

Abstract:Recent work trains reasoning models with length penalties to curb overthinking and cut inference cost. We show that these penalties make the chain of thought less monitorable. A length-compressed model still lets misleading hints steer its answers, but it less often verbalizes their influence. We train Qwen3-4B and Qwen3-14B with reinforcement learning under length penalties targeting 60% down to 30% of baseline chain-of-thought length, then evaluate them with nine types of biasing hints on held-out MMLU-Pro-R and four transfer benchmarks. A chain is faithful when an LLM monitor can tell from it that the hint influenced the answer. At the 30% target, accuracy stays near baseline and wrong-answer hints switch answers as often as before. Yet faithfulness drops on every evaluation set for both models, by 39% for Qwen3-14B and 35% for Qwen3-4B on MMLU-Pro-R. A control trained with the same correctness and format rewards but no length penalty leaves faithfulness intact or raises it. Shortening alone does not explain the drop. Compressed chains mention the hint 7 to 35 percentage points less often than the uncompressed model's chains shortened to the same length by random sentence deletion, across both model sizes and all five evaluation sets. Length penalties therefore trade monitorability for inference cost by removing the evidence monitors depend on.

Subjects:	Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
ACM classes:	I.2.7; I.2.6
Cite as:	arXiv:2607.09786 [cs.AI]
	(or arXiv:2607.09786v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2607.09786 arXiv-issued DOI via DataCite

Submission history

From: Bryce Little [view email]
[v1] Wed, 8 Jul 2026 14:18:26 UTC (79 KB)
[v2] Fri, 17 Jul 2026 13:37:42 UTC (79 KB)
[v3] Sun, 2 Aug 2026 18:21:10 UTC (89 KB)
[v4] Mon, 21 Sep 2026 15:43:41 UTC (105 KB)