Exploring the Benefit of Activation Sparsity in Pre-training. - Zhengyan Zhang; Chaojun Xiao; Qiujieli Qin; Yankai Lin; Zhiyuan Zeng; Xu Han 007; Zhiyuan Liu 001; Ruobing Xie; Maosong Sun 001; Jie Zhou 016 (ICML, 2024)