Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (2025)
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025)
OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning (2024)
System-2 Mathematical Reasoning via Enriched Instruction Tuning (2024)
Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning (2024)
URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics (2025)
HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-02-06T01:33:29.382Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":264}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.746240496635437},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.02508","authors":[{"_id":"67a2d1f9bc9d072d9459e857","user":{"_id":"6553c985a7aded0380b5f928","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg","isPro":false,"fullname":"Maohao Shen","user":"maohaos2","type":"user"},"name":"Maohao Shen","status":"extracted_confirmed","statusLastChangedAt":"2025-02-05T03:00:33.470Z","hidden":false},{"_id":"67a2d1f9bc9d072d9459e858","name":"Guangtao Zeng","hidden":false},{"_id":"67a2d1f9bc9d072d9459e859","name":"Zhenting Qi","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85a","name":"Zhang-Wei Hong","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85b","name":"Zhenfang Chen","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85c","name":"Wei Lu","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85d","name":"Gregory Wornell","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85e","name":"Subhro Das","hidden":false},{"_id":"67a2d1f9bc9d072d9459e85f","name":"David Cox","hidden":false},{"_id":"67a2d1f9bc9d072d9459e860","name":"Chuang Gan","hidden":false}],"publishedAt":"2025-02-04T17:26:58.000Z","submittedOnDailyAt":"2025-02-05T00:25:09.693Z","title":"Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM\n Reasoning via Autoregressive Search","submittedOnDailyBy":{"_id":"60ad0de755f970745d4ec28d","avatarUrl":"/avatars/b0de0222b8ed5fdac8dc7cb0336d2ec7.svg","isPro":true,"fullname":"GtZeng","user":"chaoscodes","type":"user"},"summary":"Large language models (LLMs) have demonstrated remarkable reasoning\ncapabilities across diverse domains. Recent studies have shown that increasing\ntest-time computation enhances LLMs' reasoning capabilities. This typically\ninvolves extensive sampling at inference time guided by an external LLM\nverifier, resulting in a two-player system. Despite external guidance, the\neffectiveness of this system demonstrates the potential of a single LLM to\ntackle complex tasks. Thus, we pose a new research problem: Can we internalize\nthe searching capabilities to fundamentally enhance the reasoning abilities of\na single LLM? This work explores an orthogonal direction focusing on\npost-training LLMs for autoregressive searching (i.e., an extended reasoning\nprocess with self-reflection and self-exploration of new strategies). To\nachieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a\ntwo-stage training paradigm: 1) a small-scale format tuning stage to\ninternalize the COAT reasoning format and 2) a large-scale self-improvement\nstage leveraging reinforcement learning. Our approach results in Satori, a 7B\nLLM trained on open-source models and data. Extensive empirical evaluations\ndemonstrate that Satori achieves state-of-the-art performance on mathematical\nreasoning benchmarks while exhibits strong generalization to out-of-domain\ntasks. Code, data, and models will be fully open-sourced.","upvotes":23,"discussionId":"67a2d1fcbc9d072d9459e91b","ai_summary":"Satori, a 7B LLM enhanced with COAT reasoning and two-stage training, achieves top performance in math benchmarks and generalizes well to unrelated tasks.","ai_keywords":["LLMs","reasoning capabilities","test-time computation","LLM verifier","autoregressive searching","COAT reasoning","format tuning","self-improvement stage","reinforcement learning","mathematical reasoning benchmarks","out-of-domain tasks"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60ad0de755f970745d4ec28d","avatarUrl":"/avatars/b0de0222b8ed5fdac8dc7cb0336d2ec7.svg","isPro":true,"fullname":"GtZeng","user":"chaoscodes","type":"user"},{"_id":"6553c985a7aded0380b5f928","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6553c985a7aded0380b5f928/tAJnmYvmaAse4fbHq0CjA.jpeg","isPro":false,"fullname":"Maohao Shen","user":"maohaos2","type":"user"},{"_id":"650390d0c549d0b61839ca13","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/650390d0c549d0b61839ca13/2bFvv7csIA2xVNbVR9RRa.png","isPro":true,"fullname":"Zhenting Qi","user":"zhenting","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"631e14ac473a6825f285e89d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/631e14ac473a6825f285e89d/K-6QnoeGLg8XFvbTMMdqA.jpeg","isPro":false,"fullname":"Yury Panikov","user":"panikov","type":"user"},{"_id":"6216f8a3a805e5a4d418f4cb","avatarUrl":"/avatars/8860471b3e397ca61505ee2edf73e4a6.svg","isPro":false,"fullname":"Zhenfang Chen","user":"zfchen","type":"user"},{"_id":"675893bc6f24ab221a777c1e","avatarUrl":"/avatars/ac8e04fe7e6e0032503053071410dae2.svg","isPro":false,"fullname":"Guangtao Zeng","user":"GGGGGuangtao","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"66e1080a26e1efe7d7f0f431","avatarUrl":"/avatars/1e229ff545461a719589d8f1b6196eb0.svg","isPro":false,"fullname":"Xuliang Huang","user":"Xulianghuang","type":"user"},{"_id":"65197556b4bfc5d9120c2c10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65197556b4bfc5d9120c2c10/2_d7Ji4ibs6uWG8HbEmSX.jpeg","isPro":false,"fullname":"liu zh","user":"morphism42","type":"user"},{"_id":"67a4281cbd340a42fd4324ae","avatarUrl":"/avatars/26a13e63dd38a8167aa675f286b8218a.svg","isPro":false,"fullname":"Huayu Dai","user":"Huayu6","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2502.02508

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

Published on Feb 4

· Submitted by

GtZeng on Feb 5

Upvote

Authors:

Maohao Shen ,

Abstract

Satori, a 7B LLM enhanced with COAT reasoning and two-stage training, achieves top performance in math benchmarks and generalizes well to unrelated tasks.

AI-generated summary

Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verifier, resulting in a two-player system. Despite external guidance, the effectiveness of this system demonstrates the potential of a single LLM to tackle complex tasks. Thus, we pose a new research problem: Can we internalize the searching capabilities to fundamentally enhance the reasoning abilities of a single LLM? This work explores an orthogonal direction focusing on post-training LLMs for autoregressive searching (i.e., an extended reasoning process with self-reflection and self-exploration of new strategies). To achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning. Our approach results in Satori, a 7B LLM trained on open-source models and data. Extensive empirical evaluations demonstrate that Satori achieves state-of-the-art performance on mathematical reasoning benchmarks while exhibits strong generalization to out-of-domain tasks. Code, data, and models will be fully open-sourced.

View arXiv page View PDF Add to collection

Community

chaoscodes

Paper submitter Feb 5

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search.

This work explores how we can internalize search capabilities within a single LLM to fundamentally enhance its reasoning abilities. We introduce a novel approach to post-training LLMs that enables self-reflection and self-exploration.

Our proposed two-stage training paradigm includes:

Format Tuning: A small-scale stage to teach Chain-of-Action-Thought (COAT) reasoning.
Self-Improvement via Reinforcement Learning: A large-scale stage that internalizes autoregressive searching.

These advancements culminate in Satori, a 7B LLM trained on fully open-source models and data, achieving state-of-the-art performance on mathematical reasoning benchmarks while demonstrating strong generalization to out-of-domain tasks.