"BNN - BN = ?": Training Binary Neural Networks Without Batch Normalization. - Tianlong Chen 001; Zhenyu Zhang 015; Xu Ouyang; Zechun Liu; Zhiqiang Shen; Zhangyang Wang (CVPR Workshops, 2021)

Robust Overfitting may be mitigated by properly learned smoothening. - Tianlong Chen 001; Zhenyu Zhang 015; Sijia Liu 001; Shiyu Chang; Zhangyang Wang (ICLR, 2021)

Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning. - Tianlong Chen 001; Zhenyu Zhang 015; Sijia Liu 001; Shiyu Chang; Zhangyang Wang (ICLR, 2021)

GANs Can Play Lottery Tickets Too. - Xuxi Chen; Zhenyu Zhang 015; Yongduo Sui; Tianlong Chen 001 (ICLR, 2021)

Efficient Lottery Ticket Finding: Less Data is More. - Zhenyu Zhang 015; Xuxi Chen; Tianlong Chen 001; Zhangyang Wang (ICML, 2021)

You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership. - Xuxi Chen; Tianlong Chen 001; Zhenyu Zhang 015; Zhangyang Wang (NeurIPS, 2021)

The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy. - Tianlong Chen 001; Zhenyu Zhang 015; Yu Cheng 001; Ahmed Awadallah 001; Zhangyang Wang (CVPR, 2022)

Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free. - Tianlong Chen 001; Zhenyu Zhang 015; Yihua Zhang; Shiyu Chang; Sijia Liu 001; Zhangyang Wang (CVPR, 2022)

Sparsity Winning Twice: Better Robust Generalization from More Efficient Training. - Tianlong Chen 001; Zhenyu Zhang 015; Pengjun Wang; Santosh Balachandra; Haoyu Ma; Zehao Wang; Zhangyang Wang (ICLR, 2022)

Data-Efficient Double-Win Lottery Tickets from Robust Pre-training. - Tianlong Chen 001; Zhenyu Zhang 015; Sijia Liu 001; Yang Zhang 001; Shiyu Chang; Zhangyang Wang (ICML, 2022)

Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustness. - Tianlong Chen 001; Huan Zhang 001; Zhenyu Zhang 015; Shiyu Chang; Sijia Liu 001; Pin-Yu Chen; Zhangyang Wang (ICML, 2022)

Randomized Channel Shuffling: Minimal-Overhead Backdoor Attack Detection without Clean Datasets. - Ruisi Cai; Zhenyu Zhang 015; Tianlong Chen 001; Xiaohan Chen; Zhangyang Wang (NeurIPS, 2022)

Sparse Winning Tickets are Data-Efficient Image Recognizers. - Mukund Varma T.; Xuxi Chen; Zhenyu Zhang 015; Tianlong Chen 001; Subhashini Venugopalan; Zhangyang Wang (NeurIPS, 2022)

QuanGCN: Noise-Adaptive Training for Robust Quantum Graph Convolutional Networks. - Kaixiong Zhou; Zhenyu Zhang 015; Shengyuan Chen; Tianlong Chen 001; Xiao Huang 001; Zhangyang Wang; Xia Hu 001 (CoRR, 2022)

Can You Win Everything with A Lottery Ticket? - Tianlong Chen 001; Zhenyu Zhang 015; Jun Wu; Randy Huang; Sijia Liu 001; Shiyu Chang; Zhangyang Wang (Trans. Mach. Learn. Res., 2022)

Accelerable Lottery Tickets with the Mixed-Precision Quantization. - Zhangheng Li; Yu Gong; Zhenyu Zhang 015; Xingyun Xue; Tianlong Chen 001; Yi Liang; Bo Yuan 001; Zhangyang Wang (CVPR Workshops, 2023)

Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers. - Tianlong Chen 001; Zhenyu Zhang 015; Ajay Kumar Jaiswal; Shiwei Liu 003; Zhangyang Wang (ICLR, 2023)

Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! - Shiwei Liu 003; Tianlong Chen 001; Zhenyu Zhang 015; Xuxi Chen; Tianjin Huang; Ajay Kumar Jaiswal; Zhangyang Wang (ICLR, 2023)

Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights? - Ruisi Cai; Zhenyu Zhang 015; Zhangyang Wang (ICML, 2023)

Are Large Kernels Better Teachers than Transformers for ConvNets? - Tianjin Huang; Lu Yin 006; Zhenyu Zhang 015; Li Shen 008; Meng Fang; Mykola Pechenizkiy; Zhangyang Wang; Shiwei Liu 003 (ICML, 2023)

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. - Zhenyu Zhang 015; Ying Sheng 007; Tianyi Zhou 001; Tianlong Chen 001; Lianmin Zheng; Ruisi Cai; Zhao Song 002; Yuandong Tian; Christopher Ré; Clark W. Barrett; Zhangyang Wang; Beidi Chen (NeurIPS, 2023)

QuantumSEA: In-Time Sparse Exploration for Noise Adaptive Quantum Circuits. - Tianlong Chen 001; Zhenyu Zhang 015; Hanrui Wang 002; Jiaqi Gu 002; Zirui Li; David Z. Pan; Frederic T. Chong; Song Han 003; Zhangyang Wang (QCE, 2023)

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. - Zhenyu Zhang 015; Ying Sheng 007; Tianyi Zhou 002; Tianlong Chen 001; Lianmin Zheng; Ruisi Cai; Zhao Song 002; Yuandong Tian; Christopher Ré; Clark W. Barrett; Zhangyang Wang; Beidi Chen (CoRR, 2023)

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. - Lu Yin 006; You Wu 001; Zhenyu Zhang 015; Cheng-Yu Hsieh; Yaqing Wang 007; Yiling Jia; Mykola Pechenizkiy; Yi Liang; Zhangyang Wang; Shiwei Liu 003 (CoRR, 2023)

Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention. - Zhen Tan 001; Tianlong Chen 001; Zhenyu Zhang 015; Huan Liu 001 (AAAI, 2024)

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. - Pingzhi Li; Zhenyu Zhang 015; Prateek Yadav; Yi-Lin Sung; Yu Cheng 001; Mohit Bansal; Tianlong Chen 001 (ICLR, 2024)

JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention. - Yuandong Tian; Yiping Wang; Zhenyu Zhang 015; Beidi Chen; Simon Shaolei Du (ICLR, 2024)

CaM: Cache Merging for Memory-efficient LLMs Inference. - Yuxin Zhang 002; Yuxuan Du; Gen Luo; Yunshan Zhong; Zhenyu Zhang 015; Shiwei Liu 003; Rongrong Ji (ICML, 2024)

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. - Lu Yin 006; You Wu 001; Zhenyu Zhang 015; Cheng-Yu Hsieh; Yaqing Wang 007; Yiling Jia; Gen Li 012; Ajay Kumar Jaiswal; Mykola Pechenizkiy; Yi Liang; Michael Bendersky; Zhangyang Wang; et al. (ICML, 2024)

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference. - Harry Dong; Xinyu Yang 002; Zhenyu Zhang 015; Zhangyang Wang; Yuejie Chi; Beidi Chen (ICML, 2024)

Sparse Cocktail: Every Sparse Pattern Every Sparse Ratio All At Once. - Zhangheng Li; Shiwei Liu 003; Tianlong Chen 001; Ajay Kumar Jaiswal; Zhenyu Zhang 015; Dilin Wang; Raghuraman Krishnamoorthi; Shiyu Chang; Zhangyang Wang (ICML, 2024)

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. - Jiawei Zhao; Zhenyu Zhang 015; Beidi Chen; Zhangyang Wang; Anima Anandkumar; Yuandong Tian (ICML, 2024)

Q-Hitter: A Better Token Oracle for Efficient LLM Inference via Sparse-Quantized KV Cache. - Zhenyu Zhang 015; Shiwei Liu 003; Runjin Chen; Bhavya Kailkhura; Beidi Chen; Atlas Wang (MLSys, 2024)

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding. - Zhenyu Zhang 015; Runjin Chen; Shiwei Liu 003; Zhewei Yao; Olatunji Ruwase; Beidi Chen; Xiaoxia Wu; Zhangyang Wang (NeurIPS, 2024)

From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients. - Ajay Jaiswal; Lu Yin 006; Zhenyu Zhang 015; Shiwei Liu 003; Jiawei Zhao; Yuandong Tian; Zhangyang Wang (CoRR, 2024)

APOLLO: SGD-like Memory, AdamW-level Performance. - Hanqing Zhu; Zhenyu Zhang 015; Wenyan Cong; Xi Liu; Sem Park; Vikas Chandra; Bo Long; David Z. Pan; Zhangyang Wang; Jinwon Lee (CoRR, 2024)

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients. - Zhenyu Zhang 015; Ajay Kumar Jaiswal; Lu Yin 006; Shiwei Liu 003; Jiawei Zhao; Yuandong Tian; Zhangyang Wang (CPAL, 2025)

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference. - Zhenyu Zhang 015; Zechun Liu; Yuandong Tian; Harshit Khaitan; Zhangyang Wang; Steven Li (ICLR, 2025)

Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More. - Xialie Zhuang; Zhikai Jia; Jianjin Li; Zhenyu Zhang 015; Li Shen 008; Zheng Cao; Shiwei Liu 003 (CoRR, 2025)

Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam. - Tianjin Huang; Haotian Hu; Zhenyu Zhang 015; Gaojie Jin; Xiang Li 001; Li Shen 008; Tianlong Chen 001; Lu Liu; Qingsong Wen; Zhangyang Wang; Shiwei Liu 003 (CoRR, 2025)

SEAL: Steerable Reasoning Calibration of Large Language Models for Free. - Runjin Chen; Zhenyu Zhang 015; Junyuan Hong; Souvik Kundu 009; Zhangyang Wang (CoRR, 2025)

On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention. - Yeonju Ro; Zhenyu Zhang 015; Souvik Kundu 009; Zhangyang Wang; Aditya Akella (CoRR, 2025)