International Conference on Computational Science, ICCS 2012. - Pai-Wei Lai; Huaijian Zhang; Samyam Rajbhandari; Edward F. Valeev; Karol Kowalski; P. Sadayappan (ICCS, 2012)

A framework for load balancing of tensor contraction expressions via dynamic task partitioning. - Pai-Wei Lai; Kevin Stock; Samyam Rajbhandari; Sriram Krishnamoorthy; P. Sadayappan (SC, 2013)

CAST: Contraction Algorithm for Symmetric Tensors. - Samyam Rajbhandari; Akshay Nikam; Pai-Wei Lai; Kevin Stock; Sriram Krishnamoorthy; P. Sadayappan (ICPP, 2014)

A Communication-Optimal Framework for Contracting Distributed Tensors. - Samyam Rajbhandari; Akshay Nikam; Pai-Wei Lai; Kevin Stock; Sriram Krishnamoorthy; P. Sadayappan (SC, 2014)

On fusing recursive traversals of K-d trees. - Samyam Rajbhandari; Jinsung Kim; Sriram Krishnamoorthy; Louis-Noël Pouchet; Fabrice Rastello; Robert J. Harrison; P. Sadayappan (CC, 2016)

A domain-specific compiler for a parallel multiresolution adaptive numerical simulation environment. - Samyam Rajbhandari; Jinsung Kim; Sriram Krishnamoorthy; Louis-Noël Pouchet; Fabrice Rastello; Robert J. Harrison; P. Sadayappan (SC, 2016)

Optimizing CNNs on Multicores for Scalability, Performance and Goodput. - Samyam Rajbhandari; Yuxiong He; Olatunji Ruwase; Michael Carbin; Trishul M. Chilimbi (ASPLOS, 2017)

Optimizing the Four-Index Integral Transform Using Data Movement Lower Bounds Analysis. - Samyam Rajbhandari; Fabrice Rastello; Karol Kowalski; Sriram Krishnamoorthy; P. Sadayappan (PPoPP, 2017)

Learning Intrinsic Sparse Structures within Long Short-term Memory. - Wei Wen; Yuxiong He; Samyam Rajbhandari; Wenhan Wang; Fang Liu; Bin Hu; Yiran Chen 001; Hai Li 001 (CoRR, 2017)

Learning Intrinsic Sparse Structures within Long Short-Term Memory. - Wei Wen; Yuxiong He; Samyam Rajbhandari; Minjia Zhang; Wenhan Wang; Fang Liu; Bin Hu; Yiran Chen 001; Hai Li 001 (ICLR (Poster), 2018)

DeepCPU: Serving RNN-based Deep Learning Models 10x Faster. - Minjia Zhang; Samyam Rajbhandari; Wenhan Wang; Yuxiong He (USENIX ATC, 2018)

Fast LSTM Inference by Dynamic Decomposition on Cloud Systems. - Yang You 001; Yuxiong He; Samyam Rajbhandari; Wenhan Wang; Cho-Jui Hsieh; Kurt Keutzer; James Demmel (ICDM, 2019)

Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft. - Minjia Zhang; Samyam Rajbhandari; Wenhan Wang; Elton Zheng; Olatunji Ruwase; Jeff Rasley; Jason Li; Junhua Wang; Yuxiong He (OpML, 2019)

AntMan: Sparse Low-Rank Compression to Accelerate RNN inference. - Samyam Rajbhandari; Harsh Shrivastava 001; Yuxiong He (CoRR, 2019)

ZeRO: Memory Optimization Towards Training A Trillion Parameter Models. - Samyam Rajbhandari; Jeff Rasley; Olatunji Ruwase; Yuxiong He (CoRR, 2019)

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. - Jeff Rasley; Samyam Rajbhandari; Olatunji Ruwase; Yuxiong He (KDD, 2020)

ZeRO: memory optimizations toward training trillion parameter models. - Samyam Rajbhandari; Jeff Rasley; Olatunji Ruwase; Yuxiong He (SC, 2020)

APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm. - Hanlin Tang; Shaoduo Gan; Samyam Rajbhandari; Xiangru Lian; Ji Liu 002; Yuxiong He; Ce Zhang 001 (CoRR, 2020)

Fast LSTM by dynamic decomposition on cloud and distributed systems. - Yang You 001; Yuxiong He; Samyam Rajbhandari; Wenhan Wang; Cho-Jui Hsieh; Kurt Keutzer; James Demmel (Knowl. Inf. Syst., 2020)

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. - Hanlin Tang; Shaoduo Gan; Ammar Ahmad Awan; Samyam Rajbhandari; Conglong Li; Xiangru Lian; Ji Liu 002; Ce Zhang 001; Yuxiong He (ICML, 2021)

SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement. - Heyang Qin; Samyam Rajbhandari; Olatunji Ruwase; Feng Yan 001; Lei Yang 001; Yuxiong He (NeurIPS, 2021)

ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. - Samyam Rajbhandari; Olatunji Ruwase; Jeff Rasley; Shaden Smith; Yuxiong He (SC, 2021)

ZeRO-Offload: Democratizing Billion-Scale Model Training. - Jie Ren 015; Samyam Rajbhandari; Reza Yazdani Aminabadi; Olatunji Ruwase; Shuangyan Yang; Minjia Zhang; Dong Li 001; Yuxiong He (USENIX ATC, 2021)

Scalable and Efficient MoE Training for Multitask Multilingual Models. - Young Jin Kim 001; Ammar Ahmad Awan; Alexandre Muzio; Andrés Felipe Cruz-Salinas; Liyang Lu; Amr Hendy; Samyam Rajbhandari; Yuxiong He; Hany Hassan Awadalla (CoRR, 2021)

1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. - Conglong Li; Ammar Ahmad Awan; Hanlin Tang; Samyam Rajbhandari; Yuxiong He (HIPC, 2022)

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. - Samyam Rajbhandari; Conglong Li; Zhewei Yao; Minjia Zhang; Reza Yazdani Aminabadi; Ammar Ahmad Awan; Jeff Rasley; Yuxiong He (ICML, 2022)

DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. - Reza Yazdani Aminabadi; Samyam Rajbhandari; Ammar Ahmad Awan; Cheng Li 001; Du Li; Elton Zheng; Olatunji Ruwase; Shaden Smith; Minjia Zhang; Jeff Rasley; Yuxiong He (SC, 2022)

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. - Shaden Smith; Mostofa Patwary; Brandon Norick; Patrick LeGresley; Samyam Rajbhandari; Jared Casper; Zhun Liu; Shrimai Prabhumoye; George Zerveas; Vijay Korthikanti; Elton Zheng; Rewon Child; et al. (CoRR, 2022)

A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. - Siddharth Singh; Olatunji Ruwase; Ammar Ahmad Awan; Samyam Rajbhandari; Yuxiong He; Abhinav Bhatele (ICS, 2023)

A Novel Tensor-Expert Hybrid Parallelism Approach to Scale Mixture-of-Experts Training. - Siddharth Singh; Olatunji Ruwase; Ammar Ahmad Awan; Samyam Rajbhandari; Yuxiong He; Abhinav Bhatele (CoRR, 2023)

ZeRO++: Extremely Efficient Collective Communication for Giant Model Training. - Guanhua Wang; Heyang Qin; Sam Ade Jacobs; Connor Holmes; Samyam Rajbhandari; Olatunji Ruwase; Feng Yan 001; Lei Yang 001; Yuxiong He (CoRR, 2023)

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. - Zhewei Yao; Reza Yazdani Aminabadi; Olatunji Ruwase; Samyam Rajbhandari; Xiaoxia Wu; Ammar Ahmad Awan; Jeff Rasley; Minjia Zhang; Conglong Li; Connor Holmes; Zhongzhu Zhou; Michael Wyatt; et al. (CoRR, 2023)

DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention. - Zhewei Yao; Xiaoxia Wu; Conglong Li; Minjia Zhang; Heyang Qin; Olatunji Ruwase; Ammar Ahmad Awan; Samyam Rajbhandari; Yuxiong He (CoRR, 2023)

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. - Sam Ade Jacobs; Masahiro Tanaka; Chengming Zhang 006; Minjia Zhang; Shuaiwen Leon Song; Samyam Rajbhandari; Yuxiong He (CoRR, 2023)

ZeRO++: Extremely Efficient Collective Communication for Large Model Training. - Guanhua Wang; Heyang Qin; Sam Ade Jacobs; Xiaoxia Wu; Connor Holmes; Zhewei Yao; Samyam Rajbhandari; Olatunji Ruwase; Feng Yan 001; Lei Yang 001; Yuxiong He (ICLR, 2024)

System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. - Sam Ade Jacobs; Masahiro Tanaka; Chengming Zhang 006; Minjia Zhang; Reza Yazdani Aminabadi; Shuaiwen Leon Song; Samyam Rajbhandari; Yuxiong He (IPDPS (Workshops), 2024)

System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. - Sam Ade Jacobs; Masahiro Tanaka; Chengming Zhang 006; Minjia Zhang; Reza Yazdani Aminadabi; Shuaiwen Leon Song; Samyam Rajbhandari; Yuxiong He (PODC, 2024)

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. - Connor Holmes; Masahiro Tanaka; Michael Wyatt; Ammar Ahmad Awan; Jeff Rasley; Samyam Rajbhandari; Reza Yazdani Aminabadi; Heyang Qin; Arash Bakhtiari; Lev Kurilenko; Yuxiong He (CoRR, 2024)

SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation. - Aurick Qiao; Zhewei Yao; Samyam Rajbhandari; Yuxiong He (CoRR, 2024)

Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences. - Stas Bekman; Samyam Rajbhandari; Michael Wyatt; Jeff Rasley; Tunji Ruwase; Zhewei Yao; Aurick Qiao; Yuxiong He (CoRR, 2025)

Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI. - Samyam Rajbhandari; Mert Hidayetoglu; Aurick Qiao; Ye Wang; Juncheng Yang; Jeff Rasley; Michael Wyatt; Yuxiong He (CoRR, 2025)