DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems. - Adam Rupe; Prabhat; James P. Crutchfield; Nalini Kumar; Vladislav Epifanov; Karthik Kashinath; Oleksandr Pavlyk; Frank Schlimbach; Mostofa Patwary; Sergey Maidanov; Victor W. Lee (MLHPC@SC, 2019)

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. - Mohammad Shoeybi; Mostofa Patwary; Raul Puri; Patrick LeGresley; Jared Casper; Bryan Catanzaro (CoRR, 2019)

Large Scale Multi-Actor Generative Dialog Modeling. - Alex Boyd; Raul Puri; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (ACL, 2020)

Training Question Answering Models From Synthetic Data. - Raul Puri; Ryan Spring; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (EMNLP (1), 2020)

BioMegatron: Larger Biomedical Domain Language Model. - Hoo-Chang Shin; Yang Zhang 089; Evelina Bakhturina; Raul Puri; Mostofa Patwary; Mohammad Shoeybi; Raghav Mani (EMNLP (1), 2020)

MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. - Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Raul Puri; Pascale Fung; Anima Anandkumar; Bryan Catanzaro (EMNLP (1), 2020)

Local Knowledge Powered Conversational Agents. - Sashank Santhanam; Wei Ping; Raul Puri; Mohammad Shoeybi; Mostofa Patwary; Bryan Catanzaro (CoRR, 2020)

End-to-End Training of Neural Retrievers for Open-Domain Question Answering. - Devendra Singh Sachan; Mostofa Patwary; Mohammad Shoeybi; Neel Kant; Wei Ping; William L. Hamilton; Bryan Catanzaro (ACL/IJCNLP (1), 2021)

Efficient large-scale language model training on GPU clusters using megatron-LM. - Deepak Narayanan; Mohammad Shoeybi; Jared Casper; Patrick LeGresley; Mostofa Patwary; Vijay Korthikanti; Dmitri Vainbrand; Prethvi Kashinkunti; Julie Bernauer; Bryan Catanzaro; Amar Phanishayee; Matei Zaharia (SC, 2021)

Efficient Large-Scale Language Model Training on GPU Clusters. - Deepak Narayanan; Mohammad Shoeybi; Jared Casper; Patrick LeGresley; Mostofa Patwary; Vijay Korthikanti; Dmitri Vainbrand; Prethvi Kashinkunti; Julie Bernauer; Bryan Catanzaro; Amar Phanishayee; Matei Zaharia (CoRR, 2021)

Multi-Stage Prompting for Knowledgeable Dialogue Generation. - Zihan Liu 001; Mostofa Patwary; Ryan Prenger; Shrimai Prabhumoye; Wei Ping; Mohammad Shoeybi; Bryan Catanzaro (ACL (Findings), 2022)

Evaluating Parameter Efficient Learning for Generation. - Peng Xu 008; Mostofa Patwary; Shrimai Prabhumoye; Virginia Adams; Ryan Prenger; Wei Ping; Nayeon Lee; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2022)

Keynote Talk 2 Training Large Language Models: Challenges and Opportunities. - Mostofa Patwary (IPDPS Workshops, 2022)

Factuality Enhanced Language Models for Open-Ended Text Generation. - Nayeon Lee; Wei Ping; Peng Xu 008; Mostofa Patwary; Pascale Fung; Mohammad Shoeybi; Bryan Catanzaro (NeurIPS, 2022)

Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models. - Boxin Wang; Wei Ping; Chaowei Xiao; Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Bo Li 026; Anima Anandkumar; Bryan Catanzaro (NeurIPS, 2022)

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. - Shaden Smith; Mostofa Patwary; Brandon Norick; Patrick LeGresley; Samyam Rajbhandari; Jared Casper; Zhun Liu; Shrimai Prabhumoye; George Zerveas; Vijay Korthikanti; Elton Zheng; Rewon Child; et al. (CoRR, 2022)

Factuality Enhanced Language Models for Open-Ended Text Generation. - Nayeon Lee; Wei Ping; Peng Xu 008; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2022)

Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models. - Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EACL, 2023)

Context Generation Improves Open Domain Question Answering. - Dan Su 003; Mostofa Patwary; Shrimai Prabhumoye; Peng Xu 008; Ryan Prenger; Mohammad Shoeybi; Pascale Fung; Anima Anandkumar; Bryan Catanzaro (EACL (Findings), 2023)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Bo Liu; Aastha Jhunjhunwala; Zhilin Wang; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2024)

LLM-Evolve: Evaluation for LLM's Evolving Capability on Benchmarks. - Jiaxuan You; Mingjie Liu; Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (EMNLP, 2024)

Compact Language Models via Pruning and Knowledge Distillation. - Saurav Muralidharan; Sharath Turuvekere Sreenivas; Raviraj Joshi; Marcin Chochowski; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jan Kautz; Pavlo Molchanov 001 (NeurIPS, 2024)

Nemotron-4 15B Technical Report. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Mostofa Patwary; Sandeep Subramanian; Dan Su 003; Chen Zhu 001; Deepak Narayanan; Aastha Jhunjhunwala; Ayush Dattagupta; Vibhu Jawa; Jiwei Liu; et al. (CoRR, 2024)

Nemotron-4 340B Technical Report. - Bo Adler; Niket Agarwal; Ashwath Aithal; Dong H. Anh; Pallab Bhattacharya; Annika Brundyn; Jared Casper; Bryan Catanzaro; Sharon Clay; Jonathan M. Cohen; Sirshak Das; Ayush Dattagupta; et al. (CoRR, 2024)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction. - Jupinder Parmar; Shrimai Prabhumoye; Joseph Jennings; Bo Li 026; Aastha Jhunjhunwala; Zhilin Wang; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models. - Jupinder Parmar; Sanjeev Satheesh; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

LLM Pruning and Distillation in Practice: The Minitron Approach. - Sharath Turuvekere Sreenivas; Saurav Muralidharan; Raviraj Joshi; Marcin Chochowski; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jan Kautz; Pavlo Molchanov 001 (CoRR, 2024)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining. - Steven Feng; Shrimai Prabhumoye; Kezhi Kong; Dan Su 003; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2024)

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. - Dan Su 003; Kezhi Kong; Ying Lin; Joseph Jennings; Brandon Norick; Markus Kliegl; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (ACL (1), 2025)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs. - Syeda Nahida Akter; Shrimai Prabhumoye; John Kamalu; Sanjeev Satheesh; Eric Nyberg; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (ICLR, 2025)

Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning. - Ximing Lu; Seungju Han 002; David Acuna; Hyunwoo Kim 002; Jaehun Jung; Shrimai Prabhumoye; Niklas Muennighoff; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Yejin Choi 001 (CoRR, 2025)

Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning. - Ali Taghibakhshi; Sharath Turuvekere Sreenivas; Saurav Muralidharan; Marcin Chochowski; Yashaswi Karnati; Raviraj Joshi; Ameya Sunil Mahabaleshwarkar; Zijia Chen; Yoshi Suhara; Oluwatobi Olabiyi; Daniel Korzekwa; Mostofa Patwary; et al. (CoRR, 2025)

CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training. - Shizhe Diao; Yu Yang; Yonggan Fu; Xin Dong 009; Dan Su 003; Markus Kliegl; Zijia Chen; Peter Belcak; Yoshi Suhara; Hongxu Yin; Mostofa Patwary; Yingyan Lin; et al. (CoRR, 2025)

Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning. - Syeda Nahida Akter; Shrimai Prabhumoye; Matvei Novikov; Seungju Han 002; Ying Lin; Evelina Bakhturina; Eric Nyberg; Yejin Choi 001; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2025)

Llama-Nemotron: Efficient Reasoning Models. - Akhiad Bercovich; Itay Levy; Izik Golan; Mohammad Dabbah; Ran El-Yaniv; Omri Puny; Ido Galil; Zach Moshe; Tomer Ronen; Najeeb Nabwani; Ido Shahaf; Oren Tropp; et al. (CoRR, 2025)

Fusing LLM Capabilities with Routing Data. - Tao Feng; Haozhen Zhang; Zijie Lei; Pengrui Han; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro; Jiaxuan You (CoRR, 2025)

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset. - Rabeeh Karimi Mahabadi; Sanjeev Satheesh; Shrimai Prabhumoye; Mostofa Patwary; Mohammad Shoeybi; Bryan Catanzaro (CoRR, 2025)