The GPU-based String Matching System in Advanced AC Algorithm. - Jiang-Feng Peng; Hu Chen; Shaohuai Shi (CIT, 2010)

Mixed Precision Method for GPU-based FFT. - Shuhan Qi; Xuan Wang 002; Shaohuai Shi (CSE, 2011)

Benchmarking State-of-the-Art Deep Learning Software Tools. - Shaohuai Shi; Qiang Wang 022; Pengfei Xu 006; Xiaowen Chu 001 (CCBD, 2016)

Performance Evaluation of Deep Learning Tools in Docker Containers. - Pengfei Xu 006; Shaohuai Shi; Xiaowen Chu 001 (BigCom, 2017)

Supervised Learning Based Algorithm Selection for Deep Neural Networks. - Shaohuai Shi; Pengfei Xu 006; Xiaowen Chu 001 (ICPADS, 2017)

Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units. - Shaohuai Shi; Xiaowen Chu 001 (CoRR, 2017)

Improving the Performance of Fully Connected Neural Networks by Out-of-Place Matrix Transpose. - Shaohuai Shi; Pengfei Xu 006; Xiaowen Chu 001 (CoRR, 2017)

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. - Shaohuai Shi; Xiaowen Chu 001 (CoRR, 2017)

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs. - Shaohuai Shi; Qiang Wang 022; Xiaowen Chu 001 (DASC/PiCom/DataCom/CyberSciTech, 2018)

A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning. - Shaohuai Shi; Qiang Wang 022; Xiaowen Chu 001; Bo Li 001 (ICPADS, 2018)

Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs. - Shaohuai Shi; Qiang Wang 022; Xiaowen Chu 001; Bo Li 001 (CoRR, 2018)

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes. - Xianyan Jia; Shutao Song; Wei He; Yangzihao Wang; Haidong Rong; Feihu Zhou; Liqiang Xie; Zhenyu Guo; Yuanzhou Yang; Liwei Yu; Tiegang Chen; Guangxiao Hu; et al. (CoRR, 2018)

Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models. - Xin He; Xiaoming Liu; Zhili Wu; Wu Yu; Xiaowen Chu 001; Shihao Wang; Shaohuai Shi; Zhenheng Tang; Yuxin Wang 003; Zhihao Zhao; Jing Dai; Ronghao Ni; et al. (IEEE BigData, 2019)

A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks. - Shaohuai Shi; Qiang Wang 022; Kaiyong Zhao; Zhenheng Tang; Yuxin Wang 003; Xiang Huang; Xiaowen Chu 001 (ICDCS, 2019)

A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification. - Shaohuai Shi; Kaiyong Zhao; Qiang Wang 022; Zhenheng Tang; Xiaowen Chu 001 (IJCAI, 2019)

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms. - Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (INFOCOM, 2019)

Understanding Top-k Sparsification in Distributed Deep Learning. - Shaohuai Shi; Xiaowen Chu 001; Ka Chun Cheung; Simon See (CoRR, 2019)

Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training. - Yuxin Wang 003; Qiang Wang 022; Shaohuai Shi; Xin He; Zhenheng Tang; Kaiyong Zhao; Xiaowen Chu 001 (CCGRID, 2020)

Layer-Wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees. - Shaohuai Shi; Zhenheng Tang; Qiang Wang 022; Kaiyong Zhao; Xiaowen Chu 001 (ECAI, 2020)

Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection. - Zhenheng Tang; Shaohuai Shi; Xiaowen Chu 001 (ICDCS, 2020)

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format. - Shaohuai Shi; Qiang Wang 022; Xiaowen Chu 001 (ICPADS, 2020)

FADNet: A Fast and Accurate Network for Disparity Estimation. - Qiang Wang 022; Shaohuai Shi; Shizhen Zheng; Kaiyong Zhao; Xiaowen Chu 001 (ICRA, 2020)

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs. - Shaohuai Shi; Qiang Wang 022; Xiaowen Chu 001; Bo Li 001; Yang Qin 001; Ruihao Liu; Xinxiao Zhao (INFOCOM, 2020)

Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs. - Qiang Wang 022; Shaohuai Shi; Canhui Wang; Xiaowen Chu 001 (CoRR, 2020)

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey. - Zhenheng Tang; Shaohuai Shi; Xiaowen Chu 001; Wei Wang 030; Bo Li 001 (CoRR, 2020)

Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges. - Shaohuai Shi; Zhenheng Tang; Xiaowen Chu 001; Chengjian Liu; Wei Wang 030; Bo Li 001 (CoRR, 2020)

Automated Model Design and Benchmarking of Deep Learning Models for COVID-19 Detection with Chest CT Scans. - Xin He; Shihao Wang; Xiaowen Chu 001; Shaohuai Shi; Jiangping Tang; Xin Liu 027; Chenggang Yan 001; Jiyong Zhang; Guiguang Ding (AAAI, 2021)

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks. - Shaohuai Shi; Lin Zhang; Bo Li 001 (ICDCS, 2021)

Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning. - Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (INFOCOM, 2021)

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters. - Shaohuai Shi; Xianhao Zhou; Shutao Song; Xingyao Wang 002; Zilin Zhu; Xue Huang; Xinan Jiang; Feihu Zhou; Zhenyu Guo; Liqiang Xie; Rui Lan; Xianbin Ouyang; et al. (MLSys, 2021)

Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans. - Xin He; Shihao Wang; Xiaowen Chu 001; Shaohuai Shi; Jiangping Tang; Xin Liu 027; Chenggang Yan 001; Jiyong Zhang; Guiguang Ding (CoRR, 2021)

FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks. - Qiang Wang 022; Shaohuai Shi; Shizhen Zheng; Kaiyong Zhao; Xiaowen Chu 001 (CoRR, 2021)

A Quantitative Survey of Communication Optimizations in Distributed Deep Learning. - Shaohuai Shi; Zhenheng Tang; Xiaowen Chu 001; Chengjian Liu; Wei Wang 030; Bo Li 001 (IEEE Netw., 2021)

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning. - Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (IEEE Trans. Parallel Distributed Syst., 2021)

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching. - Qiang Wang 022; Shaohuai Shi; Kaiyong Zhao; Xiaowen Chu 001 (ECCV (32), 2022)

Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning. - Zhenheng Tang; Yonggang Zhang 003; Shaohuai Shi; Xin He; Bo Han 003; Xiaowen Chu 001 (ICML, 2022)

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters. - Yang Xiang 001; Zhihua Wu; Weibao Gong; Siyu Ding; Xianjie Mo; Yuang Liu; Shuohuan Wang; Peng Liu 024; Yongshuai Hou; Long Li; Bin Wang; Shaohuai Shi; et al. (CoRR, 2022)

An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning. - Shaohuai Shi; Qing Yang; Yang Xiang 001; Shuhan Qi; Xuan Wang 002 (CoRR, 2022)

DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining. - Lin Zhang; Shaohuai Shi; Xiaowen Chu 001; Wei Wang 030; Bo Li 001; Chengjian Liu (ICDCS, 2023)

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning. - Lin Zhang; Longteng Zhang; Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (ICDCS, 2023)

Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation. - Lin Zhang; Shaohuai Shi; Bo Li 001 (ICLR, 2023)

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining. - Shaohuai Shi; Xinglin Pan; Xiaowen Chu 001; Bo Li 001 (INFOCOM, 2023)

Accelerating Distributed K-FAC with Efficient Collective Communication and Scheduling. - Lin Zhang; Shaohuai Shi; Bo Li 001 (INFOCOM, 2023)

Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning. - Lin Zhang; Shaohuai Shi; Xiaowen Chu 001; Wei Wang 030; Bo Li 001; Chengjian Liu (CoRR, 2023)

FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training. - Zhenheng Tang; Xiaowen Chu 001; Ryan Yide Ran; Sunwoo Lee 001; Shaohuai Shi; Yonggang Zhang 003; Yuxin Wang 003; Alex Qiaozhong Liang; Salman Avestimehr; Chaoyang He 001 (CoRR, 2023)

A Generic Multi-Player Transformation Algorithm for Solving Large-Scale Zero-Sum Extensive-Form Adversarial Team Games. - Chen Qiu 003; Yulin Wu 001; Weixin Huang; Botao Liu; Shaohuai Shi; Xuan Wang 002 (CoRR, 2023)

Eva: A General Vectorized Approximation Framework for Second-order Optimization. - Lin Zhang; Shaohuai Shi; Bo Li 001 (CoRR, 2023)

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. - Longteng Zhang; Lin Zhang; Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (CoRR, 2023)

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs. - Zhenheng Tang; Yuxin Wang 003; Xin He; Longteng Zhang; Xinglin Pan; Qiang Wang 022; Rongfei Zeng; Kaiyong Zhao; Shaohuai Shi; Bingsheng He; Xiaowen Chu 001 (CoRR, 2023)

Reliable and Efficient In-Memory Fault Tolerance of Large Language Model Pretraining. - Yuxin Wang 003; Shaohuai Shi; Xin He; Zhenheng Tang; Xinglin Pan; Yang Zheng; Xiaoyu Wu; Amelie Chi Zhou; Bingsheng He; Xiaowen Chu 001 (CoRR, 2023)

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models. - Longteng Zhang; Xiang Liu 001; Zeyu Li; Xinglin Pan; Peijie Dong; Ruibo Fan; Rui Guo; Xin Wang; Qiong Luo 001; Shaohuai Shi; Xiaowen Chu 001 (CoRR, 2023)

Scalable K-FAC Training for Deep Neural Networks With Distributed Preconditioning. - Lin Zhang; Shaohuai Shi; Wei Wang 030; Bo Li 001 (IEEE Trans. Cloud Comput., 2023)

GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication. - Zhenheng Tang; Shaohuai Shi; Bo Li 001; Xiaowen Chu 001 (IEEE Trans. Parallel Distributed Syst., 2023)

Performance Analysis and Optimizations of Matrix Multiplications on ARMv8 Processors. - Hucheng Liu; Shaohuai Shi; Xuan Wang 002; Zoe L. Jiang; Qian Chen 028 (DATE, 2024)

ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling. - Shaohuai Shi; Xinglin Pan; Qiang Wang 022; Chengjian Liu; Xiaozhe Ren; Zhongzhe Hu; Yu Yang; Bo Li 001; Xiaowen Chu 001 (EuroSys, 2024)

FedImpro: Measuring and Improving Client Update in Federated Learning. - Zhenheng Tang; Yonggang Zhang 003; Shaohuai Shi; Xinmei Tian 001; Tongliang Liu; Bo Han 003; Xiaowen Chu 001 (ICLR, 2024)

Sparse Gradient Communication with AlltoAll for Accelerating Distributed Deep Learning. - Jing Peng 003; Zihan Li; Shaohuai Shi; Bo Li 001 (ICPP, 2024)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning. - Zichen Tang; Junlin Huang; Rudan Yan; Yuxin Wang 003; Zhenheng Tang; Shaohuai Shi; Amelie Chi Zhou; Xiaowen Chu 001 (ICPP, 2024)

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules. - Xinglin Pan; Wenxiang Lin; Shaohuai Shi; Xiaowen Chu 001; Weinong Sun; Bo Li 001 (INFOCOM, 2024)

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing. - Yizhou Luo; Qiang Wang 022; Shaohuai Shi; Jiaxin Lai; Shuhan Qi; Jiajia Zhang 001; Xuan Wang 002 (IWQoS, 2024)

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression. - Zhenheng Tang; Xueze Kang; Yiming Yin; Xinglin Pan; Yuxin Wang 003; Xin He; Qiang Wang 022; Rongfei Zeng; Kaiyong Zhao; Shaohuai Shi; Amelie Chi Zhou; Bo Li 001; et al. (CoRR, 2024)

ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference. - Xin He; Shunkang Zhang; Yuxin Wang 003; Haiyan Yin; Zihao Zeng; Shaohuai Shi; Zhenheng Tang; Xiaowen Chu 001; Ivor W. Tsang; Yew-Soon Ong (CoRR, 2024)

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems. - Wenxiang Lin; Xinglin Pan; Shaohuai Shi; Xuan Wang 002; Xiaowen Chu 001 (CoRR, 2024)

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models. - Xinglin Pan; Wenxiang Lin; Lin Zhang; Shaohuai Shi; Zhenheng Tang; Rui Wang; Bo Li 001; Xiaowen Chu 001 (ASPLOS (1), 2025)

ScheInfer: Efficient Inference of Large Language Models with Task Scheduling on Moderate GPUs. - Wenxiang Lin; Xinglin Pan; Shaohuai Shi; Xuan Wang 002; Xiaowen Chu 001 (Euro-Par (3), 2025)

SQ-DeAR: Sparsified and Quantized Gradient Compression for Distributed Training. - Xinrui Yang; Shaohuai Shi (Euro-Par (3), 2025)

SP-MoE: Expediting Mixture-of-Experts Training with Optimized Pipelining Planning. - Ne Wang; Wenxiang Lin; Lin Zhang; Shaohuai Shi; Ruiting Zhou; Bo Li 001 (INFOCOM, 2025)

DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization. - Zhenheng Tang; Zichen Tang; Junlin Huang; Xinglin Pan; Rudan Yan; Yuxin Wang 003; Amelie Chi Zhou; Shaohuai Shi; Xiaowen Chu 001; Bo Li 001 (CoRR, 2025)

RoPe-Door: Toward Robust and Persistent Backdoor Data Poisoning Attacks in Federated Learning. - Ye Liu; Shan Chang; Denghui Li; Shaohuai Shi; Bo Li 001 (IEEE Netw., 2025)

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap. - Wenxiang Lin; Xinglin Pan; Lin Zhang; Shaohuai Shi; Xuan Wang 002; Xiaowen Chu 001 (CoRR, 2025)