Finding Skill Neurons in Pre-trained Transformer-based Language Models. - Xiaozhi Wang; Kaiyue Wen; Zhengyan Zhang; Lei Hou 001; Zhiyuan Liu 001; Juanzi Li (EMNLP, 2022)

On Transferability of Prompt Tuning for Natural Language Processing. - Yusheng Su; Xiaozhi Wang; Yujia Qin; Chi-Min Chan; Yankai Lin; Huadong Wang; Kaiyue Wen; Zhiyuan Liu 001; Peng Li 030; Juanzi Li; Lei Hou 001; Maosong Sun 001; et al. (NAACL-HLT, 2022)

Realistic Deep Learning May Not Fit Benignly. - Kaiyue Wen; Jiaye Teng; Jingzhao Zhang (CoRR, 2022)

How Does Sharpness-Aware Minimization Minimize Sharpness? - Kaiyue Wen; Tengyu Ma 001; Zhiyuan Li 005 (CoRR, 2022)

How Sharpness-Aware Minimization Minimizes Sharpness? - Kaiyue Wen; Tengyu Ma 001; Zhiyuan Li 005 (ICLR, 2023)

Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models. - Kaiyue Wen; Jiaye Teng; Jingzhao Zhang (ICLR, 2023)

Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization. - Kaiyue Wen; Zhiyuan Li 005; Tengyu Ma 001 (NeurIPS, 2023)

Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars. - Kaiyue Wen; Yuchen Li 007; Bingbin Liu; Andrej Risteski (NeurIPS, 2023)

Practically Solving LPN in High Noise Regimes Faster Using Neural Networks. - Haozhe Jiang; Kaiyue Wen; Yilei Chen 001 (IACR Cryptol. ePrint Arch., 2023)

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective. - Kaiyue Wen; Zhiyuan Li 005; Jason S. Wang; David Hall 006; Percy Liang; Tengyu Ma 001 (CoRR, 2024)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models. - Zihan Qiu; Zeyu Huang; Bo Zheng 007; Kaiyue Wen; Zekun Wang; Rui Men; Ivan Titov 001; Dayiheng Liu; Jingren Zhou 001; Junyang Lin (ACL (1), 2025)

Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images. - Shengguang Wu; Fan-Yun Sun; Kaiyue Wen; Nick Haber (ACL (1), 2025)

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View. - Kaiyue Wen; Zhiyuan Li 005; Jason S. Wang; David Leo Wright Hall; Percy Liang; Tengyu Ma 001 (ICLR, 2025)

RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval. - Kaiyue Wen; Xingyu Dang; Kaifeng Lyu (ICLR, 2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency. - Kaiyue Wen; Huaqing Zhang 005; Hongzhou Lin; Jingzhao Zhang (ICLR, 2025)

Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks? - Amirhesam Abedsoltan; Huaqing Zhang 005; Kaiyue Wen; Hongzhou Lin; Jingzhao Zhang; Mikhail Belkin (CoRR, 2025)

Overtrained Language Models Are Harder to Fine-Tune. - Jacob Mitchell Springer; Sachin Goyal; Kaiyue Wen; Tanishq Kumar; Xiang Yue; Sadhika Malladi; Graham Neubig; Aditi Raghunathan (CoRR, 2025)

Weight Ensembling Improves Reasoning in Language Models. - Xingyu Dang; Christina Baek; Kaiyue Wen; Zico Kolter; Aditi Raghunathan (CoRR, 2025)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. - Zihan Qiu; Zekun Wang; Bo Zheng 007; Zeyu Huang; Kaiyue Wen; Songlin Yang; Rui Men; Le Yu; Fei Huang; Suozhi Huang; Dayiheng Liu; Jingren Zhou 001; et al. (CoRR, 2025)

PaTH Attention: Position Encoding via Accumulating Householder Transformations. - Songlin Yang; Yikang Shen; Kaiyue Wen; Shawn Tan; Mayank Mishra; Liliang Ren; Rameswar Panda; Yoon Kim (CoRR, 2025)

QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. - Jiazheng Li; Hong Lu; Kaiyue Wen; Zaiwen Yang; Jiaxuan Gao; Hongzhou Lin; Yi Wu; Jingzhao Zhang (CoRR, 2025)