DeepNet: Scaling Transformers to 1, 000 Layers. - Hongyu Wang 009; Shuming Ma; Li Dong 004; Shaohan Huang; Dongdong Zhang 001; Furu Wei (CoRR, 2022)

Foundation Transformers. - Hongyu Wang 009; Shuming Ma; Shaohan Huang; Li Dong 004; Wenhui Wang 003; Zhiliang Peng; Yu Wu; Payal Bajaj; Saksham Singhal; Alon Benhaim; Barun Patra; Zhun Liu; et al. (CoRR, 2022)

TorchScale: Transformers at Scale. - Shuming Ma; Hongyu Wang 009; Shaohan Huang; Wenhui Wang 003; Zewen Chi; Li Dong 004; Alon Benhaim; Barun Patra; Vishrav Chaudhary; Xia Song; Furu Wei (CoRR, 2022)

Magneto: A Foundation Transformer. - Hongyu Wang 009; Shuming Ma; Shaohan Huang; Li Dong 004; Wenhui Wang 003; Zhiliang Peng; Yu Wu; Payal Bajaj; Saksham Singhal; Alon Benhaim; Barun Patra; Zhun Liu; et al. (ICML, 2023)

BitNet: Scaling 1-bit Transformers for Large Language Models. - Hongyu Wang 009; Shuming Ma; Li Dong 004; Shaohan Huang; Huaijie Wang; Lingxiao Ma; Fan Yang 024; Ruiping Wang; Yi Wu 013; Furu Wei (CoRR, 2023)

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. - Shuming Ma; Hongyu Wang 009; Lingxiao Ma; Lei Wang 222; Wenhui Wang 003; Shaohan Huang; Li Dong 004; Ruiping Wang; Jilong Xue; Furu Wei (CoRR, 2024)

Q-Sparse: All Large Language Models can be Fully Sparsely-Activated. - Hongyu Wang 009; Shuming Ma; Ruiping Wang; Furu Wei (CoRR, 2024)

DeepNet: Scaling Transformers to 1,000 Layers. - Hongyu Wang 009; Shuming Ma; Li Dong 004; Shaohan Huang; Dongdong Zhang 001; Furu Wei (IEEE Trans. Pattern Anal. Mach. Intell., 2024)

BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs. - Hongyu Wang 009; Shuming Ma; Furu Wei (CoRR, 2025)