Exploring Low Rank Training of Deep Neural Networks. - Siddhartha Rao Kamalakara; Acyr Locatelli; Bharat Venkitesh; Jimmy Ba; Yarin Gal; Aidan N. Gomez (CoRR, 2022)

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning. - Ted Zadouri; Ahmet Üstün; Arash Ahmadian; Beyza Ermis; Acyr Locatelli; Sara Hooker (ICLR, 2024)

SnapKV: LLM Knows What You are Looking for Before Generation. - Yuhong Li; Yingbing Huang; Bowen Yang; Bharat Venkitesh; Acyr Locatelli; Hanchen Ye; Tianle Cai; Patrick Lewis 002; Deming Chen (NeurIPS, 2024)

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts. - Qizhen (Irene) Zhang; Nikolas Gritsch; Dwaraknath Gnaneshwar; Simon Guo 003; David Cairuz; Bharat Venkitesh; Jakob N. Foerster; Phil Blunsom; Sebastian Ruder; Ahmet Üstün; Acyr Locatelli (NeurIPS, 2024)

Aya 23: Open Weight Releases to Further Multilingual Progress. - Viraat Aryabumi; John Dang; Dwarak Talupuru; Saurabh Dash; David Cairuz; Hangyu Lin; Bharat Venkitesh; Madeline Smith; Jon Ander Campos; Yi Chern Tan; Kelly Marchisio; Max Bartolo; et al. (CoRR, 2024)

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts. - Nikolas Gritsch; Qizhen Zhang; Acyr Locatelli; Sara Hooker; Ahmet Üstün (CoRR, 2024)

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms. - Zhengyan Shi; Sander Land; Acyr Locatelli; Matthieu Geist; Max Bartolo (CoRR, 2024)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models. - Laura Ruis; Maximilian Mozes; Juhan Bae; Siddhartha Rao Kamalakara; Dwarak Talupuru; Acyr Locatelli; Robert Kirk; Tim Rocktäschel; Edward Grefenstette; Max Bartolo (CoRR, 2024)

Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier. - John Dang; Shivalika Singh; Daniel D'souza; Arash Ahmadian; Alejandro Salamanca; Madeline Smith; Aidan Peppin; Sungjin Hong; Manoj Govindassamy; Terrence Zhao; Sandra Kublik; Meor Amer; et al. (CoRR, 2024)

To Code or Not To Code? Exploring Impact of Code in Pre-training. - Viraat Aryabumi; Yixuan Su; Raymond Ma; Adrien Morisot; Ivan Zhang; Acyr Locatelli; Marzieh Fadaee; Ahmet Üstün; Sara Hooker (ICLR, 2025)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models. - Laura Ruis; Maximilian Mozes; Juhan Bae; Siddhartha Rao Kamalakara; Dwaraknath Gnaneshwar; Acyr Locatelli; Robert Kirk; Tim Rocktäschel; Edward Grefenstette; Max Bartolo (ICLR, 2025)

Rope to Nope and Back Again: A New Hybrid Attention Strategy. - Bowen Yang; Bharat Venkitesh; Dwarak Talupuru; Hangyu Lin; David Cairuz; Phil Blunsom; Acyr Locatelli (CoRR, 2025)

Aya Vision: Advancing the Frontier of Multilingual Multimodality. - Saurabh Dash; Yiyang Nan; John Dang; Arash Ahmadian; Shivalika Singh; Madeline Smith; Bharat Venkitesh; Vlad Shmyhlo; Viraat Aryabumi; Walter Beller-Morales; Jeremy Pekmez; Jason Ozuzu; et al. (CoRR, 2025)

One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers. - Diana Abagyan; Alejandro Salamanca; Felipe Cruz-Salinas; Kris Cao; Hangyu Lin; Acyr Locatelli; Marzieh Fadaee; Ahmet Üstün; Sara Hooker (CoRR, 2025)