What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning

doi:arXiv:2412.15904

KMS

	What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning
	Ma, Yiran 1,3; Chen, Zui 2,3; Liu, Tianqiao 3; Tian, Mi 3; Liu, Zhuo 4; Liu, Zitao 5; Luo, Weiqi 5
	2024-12-20
状态	已发表
摘要	Step-level reward models (SRMs) can significantly enhance mathematical reasoning performance through process supervision or step-level preference alignment based on reinforcement learning. The performance of SRMs is pivotal, as they serve as critical guidelines, ensuring that each step in the reasoning process is aligned with desired outcomes. Recently, AlphaZero-like methods, where Monte Carlo Tree Search (MCTS) is employed for automatic step-level preference annotation, have proven particularly effective. However, the precise mechanisms behind the success of SRMs remain largely unexplored. To address this gap, this study delves into the counterintuitive aspects of SRMs, particularly focusing on MCTS-based approaches. Our findings reveal that the removal of natural language descriptions of thought processes has minimal impact on the efficacy of SRMs. Furthermore, we demonstrate that SRMs are adept at assessing the complex logical coherence present in mathematical language while having difficulty in natural language. These insights provide a nuanced understanding of the core elements that drive effective step-level reward modeling in mathematical reasoning. By shedding light on these mechanisms, this study offers valuable guidance for developing more efficient and streamlined SRMs, which can be achieved by focusing on the crucial parts of mathematical reasoning.
语种	英语
DOI	arXiv:2412.15904
相关网址	查看原文
出处	Arxiv
收录类别	PPRN.PPRN
WOS记录号	PPRN:120117525
WOS类目	Computer Science, Artificial Intelligence
资助项目	Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University[2022LSYS003] ; National Key R&D Program of China[2022YFC3303600]
文献类型	预印本
条目标识符	https://kms.shanghaitech.edu.cn/handle/2MSLDSTB/483976
专题	上海科技大学
通讯作者	Liu, Zitao
作者单位	1.Zhejiang Univ, Hangzhou, Peoples R China 2.ShanghaiTech Univ, Shanghai, Peoples R China 3.TAL Educ Grp, Beijing, Peoples R China 4.Univ Rochester, Rochester, NY, USA 5.Jinan Univ, Guangzhou, Peoples R China
推荐引用方式 GB/T 7714	Ma, Yiran,Chen, Zui,Liu, Tianqiao,et al. What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning. 2024.