Towards Wall-Normal Filtering for Large-Eddy Simulation. - Jeremy A. Templeton; Mohammad Shoeybi (Multiscale Model. Simul., 2006)

Stable and accurate schemes for the compressible Navier-Stokes equations. - Ken Mattsson; Magnus Svärd; Mohammad Shoeybi (J. Comput. Phys., 2008)

An adaptive implicit-explicit scheme for the DNS and LES of compressible flows on unstructured grids. - Mohammad Shoeybi; Magnus Svärd; Frank E. Ham; Parviz Moin (J. Comput. Phys., 2010)

Deep Voice: Real-time Neural Text-to-Speech. - Sercan Ömer Arik; Mike Chrzanowski; Adam Coates 002; Gregory Frederick Diamos; Andrew Gibiansky; Yongguo Kang; Xian Li; John Miller 001; Andrew Y. Ng; Jonathan Raiman; Shubho Sengupta; Mohammad Shoeybi (ICML, 2017)

Deep Voice: Real-time Neural Text-to-Speech. - Sercan Ömer Arik; Mike Chrzanowski; Adam Coates 002; Greg Diamos; Andrew Gibiansky; Yongguo Kang; Xian Li; John Miller 001; Jonathan Raiman; Shubho Sengupta; Mohammad Shoeybi (CoRR, 2017)

Trace norm regularization and faster inference for embedded speech recognition RNNs. - Markus Kliegl; Siddharth Goyal; Kexin Zhao; Kavya Srinet; Mohammad Shoeybi (CoRR, 2017)

Unsupervised Video Interpolation Using Cycle Consistency. - Fitsum A. Reda; Deqing Sun; Aysegul Dundar; Mohammad Shoeybi; Guilin Liu; Kevin J. Shih; Andrew Tao; Jan Kautz; Bryan Catanzaro (ICCV, 2019)

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. - Mohammad Shoeybi; Mostofa Patwary; Raul Puri; Patrick LeGresley; Jared Casper; Bryan Catanzaro (CoRR, 2019)

Neural ODEs for Image Segmentation with Level Sets. - Rafael Valle; Fitsum A. Reda; Mohammad Shoeybi; Patrick LeGresley; Andrew Tao; Bryan Catanzaro (CoRR, 2019)

Large Scale Multi-Actor Generative Dialog Modeling. - Alex Boyd; Raul Puri; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (ACL, 2020)

Training Question Answering Models From Synthetic Data. - Raul Puri; Ryan Spring; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (EMNLP (1), 2020)

BioMegatron: Larger Biomedical Domain Language Model. - Hoo-Chang Shin; Yang Zhang 089; Evelina Bakhturina; Raul Puri; Mostofa Patwary; Mohammad Shoeybi; Raghav Mani (EMNLP (1), 2020)

MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. - Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Raul Puri; Pascale Fung; Anima Anandkumar; Bryan Catanzaro (EMNLP (1), 2020)

Style Example-Guided Text Generation using Generative Adversarial Transformers. - Kuo-Hao Zeng; Mohammad Shoeybi; Ming-Yu Liu 001 (CoRR, 2020)

Local Knowledge Powered Conversational Agents. - Sashank Santhanam; Wei Ping; Raul Puri; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (CoRR, 2020)

End-to-End Training of Neural Retrievers for Open-Domain Question Answering. - Devendra Singh Sachan; Mostofa Patwary; Mohammad Shoeybi; Neel Kant; Wei Ping; William L. Hamilton; Bryan Catanzaro (ACL/IJCNLP (1), 2021)

Long-Short Transformer: Efficient Transformers for Language and Vision. - Chen Zhu 001; Wei Ping; Chaowei Xiao; Mohammad Shoeybi; Tom Goldstein; Anima Anandkumar; Bryan Catanzaro (NeurIPS, 2021)

Efficient large-scale language model training on GPU clusters using megatron-LM. - Deepak Narayanan; Mohammad Shoeybi; Jared Casper; Patrick LeGresley; Mostofa Patwary; Vijay Korthikanti; Dmitri Vainbrand; Prethvi Kashinkunti; Julie Bernauer; Bryan Catanzaro; Amar Phanishayee; Matei Zaharia (SC, 2021)

Efficient Large-Scale Language Model Training on GPU Clusters. - Deepak Narayanan; Mohammad Shoeybi; Jared Casper; Patrick LeGresley; Mostofa Patwary; Vijay Korthikanti; Dmitri Vainbrand; Prethvi Kashinkunti; Julie Bernauer; Bryan Catanzaro; Amar Phanishayee; Matei Zaharia (CoRR, 2021)

Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases. - Shrimai Prabhumoye; Rafal Kocielnik; Mohammad Shoeybi; Anima Anandkumar; Bryan Catanzaro (CoRR, 2021)

Multi-Stage Prompting for Knowledgeable Dialogue Generation. - Zihan Liu 001; Mostofa Patwary; Ryan Prenger; Shrimai Prabhumoye; Wei Ping; Mohammad Shoeybi; Bryan Catanzaro (ACL (Findings), 2022)

Evaluating Parameter Efficient Learning for Generation. - Peng Xu 008; Mostofa Patwary; Shrimai Prabhumoye; Virginia Adams; Ryan Prenger; Wei Ping; Nayeon Lee; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2022)

Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. - David Wingate; Mohammad Shoeybi; Taylor Sorensen (EMNLP (Findings), 2022)

Factuality Enhanced Language Models for Open-Ended Text Generation. - Nayeon Lee; Wei Ping; Peng Xu 008; Mostofa Patwary; Pascale Fung; Mohammad Shoeybi; Bryan Catanzaro (NeurIPS, 2022)

Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models. - Boxin Wang; Wei Ping; Chaowei Xiao; Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Bo Li 026; Anima Anandkumar; Bryan Catanzaro (NeurIPS, 2022)

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. - Shaden Smith; Mostofa Patwary; Brandon Norick; Patrick LeGresley; Samyam Rajbhandari; Jared Casper; Zhun Liu; Shrimai Prabhumoye; George Zerveas; Vijay Korthikanti; Elton Zheng; Rewon Child; et al. (CoRR, 2022)

Factuality Enhanced Language Models for Open-Ended Text Generation. - Nayeon Lee; Wei Ping; Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2022)

FP8 Formats for Deep Learning. - Paulius Micikevicius; Dusan Stosic; Neil Burgess; Marius Cornea; Pradeep Dubey; Richard Grisenthwaite; Sangwon Ha; Alexander Heinecke; Patrick Judd; John Kamalu; Naveen Mellempudi; Stuart F. Oberman; et al. (CoRR, 2022)

Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models. - Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EACL, 2023)

Context Generation Improves Open Domain Question Answering. - Dan Su 003; Mostofa Patwary; Shrimai Prabhumoye; Peng Xu 008; Ryan Prenger; Mohammad Shoeybi; Pascale Fung; Anima Anandkumar; Bryan Catanzaro (EACL (Findings), 2023)

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. - Boxin Wang; Wei Ping; Peng Xu 008; Lawrence McAfee; Zihan Liu 001; Mohammad Shoeybi; Yi Dong; Oleksii Kuchaiev; Bo Li 026; Chaowei Xiao; Anima Anandkumar; Bryan Catanzaro (EMNLP, 2023)

Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning. - Zhuolin Yang; Wei Ping; Zihan Liu 001; Vijay Korthikanti; Weili Nie; De-An Huang; Linxi Fan; Zhiding Yu; Shiyi Lan; Bo Li 026; Mohammad Shoeybi; Ming-Yu Liu 001; et al. (EMNLP (Findings), 2023)

Reducing Activation Recomputation in Large Transformer Models. - Vijay Anand Korthikanti; Jared Casper; Sangkug Lym; Lawrence McAfee; Michael Andersch; Mohammad Shoeybi; Bryan Catanzaro (MLSys, 2023)

RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder Language Models. - Jie Huang 009; Wei Ping; Peng Xu 008; Mohammad Shoeybi; Kevin Chen-Chuan Chang; Bryan Catanzaro (CoRR, 2023)

VILA: On Pre-training for Visual Language Models. - Ji Lin 002; Hongxu Yin; Wei Ping; Yao Lu 006; Pavlo Molchanov 001; Andrew Tao; Huizi Mao; Jan Kautz; Mohammad Shoeybi; Song Han 003 (CoRR, 2023)

VILA: On Pre-training for Visual Language Models. - Ji Lin 002; Hongxu Yin; Wei Ping; Pavlo Molchanov 001; Mohammad Shoeybi; Song Han 003 (CVPR, 2024)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Bo Liu; Aastha Jhunjhunwala; Zhilin Wang; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2024)

LLM-Evolve: Evaluation for LLM's Evolving Capability on Benchmarks. - Jiaxuan You; Mingjie Liu; Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2024)

Retrieval meets Long Context Large Language Models. - Peng Xu 008; Wei Ping; Xianchao Wu; Lawrence McAfee; Chen Zhu 001; Zihan Liu 001; Sandeep Subramanian; Evelina Bakhturina; Mohammad Shoeybi; Bryan Catanzaro (ICLR, 2024)

ODIN: Disentangled Reward Mitigates Hacking in RLHF. - Lichang Chen; Chen Zhu 001; Jiuhai Chen; Davit Soselia; Tianyi Zhou 001; Tom Goldstein; Heng Huang; Mohammad Shoeybi; Bryan Catanzaro (ICML, 2024)

InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining. - Boxin Wang; Wei Ping; Lawrence McAfee; Peng Xu 008; Bo Li 026; Mohammad Shoeybi; Bryan Catanzaro (ICML, 2024)

ChatQA: Surpassing GPT-4 on Conversational QA and RAG. - Zihan Liu 001; Wei Ping; Rajarshi Roy 003; Peng Xu 008; Chankyu Lee; Mohammad Shoeybi; Bryan Catanzaro (NeurIPS, 2024)

Compact Language Models via Pruning and Knowledge Distillation. - Saurav Muralidharan; Sharath Turuvekere Sreenivas; Raviraj Joshi; Marcin Chochowski; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jan Kautz; Pavlo Molchanov 001 (NeurIPS, 2024)

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. - Yue Yu; Wei Ping; Zihan Liu 001; Boxin Wang; Jiaxuan You; Chao Zhang; Mohammad Shoeybi; Bryan Catanzaro (NeurIPS, 2024)

ChatQA: Building GPT-4 Level Conversational QA Models. - Zihan Liu 001; Wei Ping; Rajarshi Roy 003; Peng Xu 008; Chankyu Lee; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Nemotron-4 15B Technical Report. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Mostofa Patwary; Sandeep Subramanian; Dan Su 003; Chen Zhu 001; Deepak Narayanan; Aastha Jhunjhunwala; Ayush Dattagupta; Vibhu Jawa; Jiwei Liu; et al. (CoRR, 2024)

An Empirical Study of Mamba-based Language Models. - Roger Waleffe; Wonmin Byeon; Duncan Riach; Brandon Norick; Vijay Korthikanti; Tri Dao; Albert Gu; Ali Hatamizadeh; Sudhakar Singh 003; Deepak Narayanan; Garvit Kulshreshtha; Vartika Singh; et al. (CoRR, 2024)

Nemotron-4 340B Technical Report. - Bo Adler; Niket Agarwal; Ashwath Aithal; Dong H. Anh; Pallab Bhattacharya; Annika Brundyn; Jared Casper; Bryan Catanzaro; Sharon Clay; Jonathan M. Cohen; Sirshak Das; Ayush Dattagupta; et al. (CoRR, 2024)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Bo Li 026; Aastha Jhunjhunwala; Zhilin Wang; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models. - Jupinder Parmar; Sanjeev Satheesh; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities. - Peng Xu 008; Wei Ping; Xianchao Wu; Zihan Liu 001; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

LLM Pruning and Distillation in Practice: The Minitron Approach. - Sharath Turuvekere Sreenivas; Saurav Muralidharan; Raviraj Joshi; Marcin Chochowski; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jan Kautz; Pavlo Molchanov 001 (CoRR, 2024)

NVLM: Open Frontier-Class Multimodal LLMs. - Wenliang Dai; Nayeon Lee; Boxin Wang; Zhuoling Yang; Zihan Liu 001; Jon Barker; Tuomas Rintamaki; Mohammad Shoeybi; Bryan Catanzaro; Wei Ping (CoRR, 2024)

Upcycling Large Language Models into Mixture of Experts. - Ethan He; Abhinav Khattar; Ryan Prenger; Vijay Korthikanti; Zijie Yan; Tong Liu; Shiqing Fan; Ashwath Aithal; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining. - Steven Feng; Shrimai Prabhumoye; Kezhi Kong; Dan Su 003; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. - Dan Su 003; Kezhi Kong; Ying Lin; Joseph Jennings; Brandon Norick; Markus Kliegl; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (ACL (1), 2025)

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling. - Zihan Liu 001; Yang Chen; Mohammad Shoeybi; Bryan Catanzaro; Wei Ping (ACL (Findings), 2025)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs. - Syeda Nahida Akter; Shrimai Prabhumoye; John Kamalu; Sanjeev Satheesh; Eric Nyberg; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (ICLR, 2025)

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. - Chankyu Lee; Rajarshi Roy 003; Mengyao Xu; Jonathan Raiman; Mohammad Shoeybi; Bryan Catanzaro; Wei Ping (ICLR, 2025)

Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS. - Sheng-Chieh Lin; Chankyu Lee; Mohammad Shoeybi; Jimmy Lin; Bryan Catanzaro; Wei Ping (ICLR, 2025)

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities. - Peng Xu 008; Wei Ping; Xianchao Wu; Chejian Xu; Zihan Liu 001; Mohammad Shoeybi; Bryan Catanzaro (ICLR, 2025)

Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning. - Ximing Lu; Seungju Han 002; David Acuna; Hyunwoo Kim 002; Jaehun Jung; Shrimai Prabhumoye; Niklas Muennighoff; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Yejin Choi 001 (CoRR, 2025)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models. - Chejian Xu; Wei Ping; Peng Xu 008; Zihan Liu 001; Boxin Wang; Mohammad Shoeybi; Bo Li 026; Bryan Catanzaro (CoRR, 2025)

Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning. - Ali Taghibakhshi; Sharath Turuvekere Sreenivas; Saurav Muralidharan; Marcin Chochowski; Yashaswi Karnati; Raviraj Joshi; Ameya Sunil Mahabaleshwarkar; Zijia Chen; Yoshi Suhara; Oluwatobi Olabiyi; Daniel Korzekwa; Mostofa Patwary; et al. (CoRR, 2025)

Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning. - Syeda Nahida Akter; Shrimai Prabhumoye; Matvei Novikov; Seungju Han 002; Ying Lin; Evelina Bakhturina; Eric Nyberg; Yejin Choi 001; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2025)

MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core. - Dennis Liu; Zijie Yan; Xin Yao; Tong Liu; Vijay Korthikanti; Evan Wu; Shiqing Fan; Gao Deng; Hongxiao Bai; Jianbin Chang; Ashwath Aithal; Michael Andersch; et al. (CoRR, 2025)

AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning. - Yang Chen; Zhuolin Yang; Zihan Liu 001; Chankyu Lee; Peng Xu 008; Mohammad Shoeybi; Bryan Catanzaro; Wei Ping (CoRR, 2025)

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning. - Jaehun Jung; Seungju Han 002; Ximing Lu; Skyler Hallinan; David Acuna; Shrimai Prabhumoye; Mostafa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Yejin Choi 001 (CoRR, 2025)

AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy. - Zihan Liu 001; Zhuolin Yang; Yang Chen; Chankyu Lee; Mohammad Shoeybi; Bryan Catanzaro; Wei Ping (CoRR, 2025)

Fusing LLM Capabilities with Routing Data. - Tao Feng; Haozhen Zhang; Zijie Lei; Pengrui Han; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jiaxuan You (CoRR, 2025)

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset. - Rabeeh Karimi Mahabadi; Sanjeev Satheesh; Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2025)