Pretraining Language Models to Ponder in Continuous Space. - Boyi Zeng; Shixiang Song; Siyuan Huang; Yixuan Wang; He Li; Ziwei He; Xinbing Wang; Zhiyu Li; Zhouhan Lin (CoRR, 2025)

Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling. - Ning Liao; Xiaoxing Wang; Zehao Lin; Weiyang Guo; Feng Hong 004; Shixiang Song; Geng Yu; Zihua Zhao; Sitao Xie; Longxuan Wei; Xiangqi Jin; Xiaohan Qin; et al. (CoRR, 2025)