On the training dynamics of deep networks with $L_2$ regularization. - Aitor Lewkowycz; Guy Gur-Ari (NeurIPS, 2020)

The large learning rate phase of deep learning: the catapult mechanism. - Aitor Lewkowycz; Yasaman Bahri; Ethan Dyer; Jascha Sohl-Dickstein; Guy Gur-Ari (CoRR, 2020)

On the training dynamics of deep networks with L2 regularization. - Aitor Lewkowycz; Guy Gur-Ari (CoRR, 2020)

How to decay your learning rate. - Aitor Lewkowycz (CoRR, 2021)

Show Your Work: Scratchpads for Intermediate Computation with Language Models. - Maxwell I. Nye; Anders Johan Andreassen; Guy Gur-Ari; Henryk Michalewski; Jacob Austin; David Bieber; David Dohan; Aitor Lewkowycz; Maarten Bosma; David Luan; Charles Sutton; Augustus Odena (CoRR, 2021)

Effect of scale on catastrophic forgetting in neural networks. - Vinay Venkatesh Ramasesh; Aitor Lewkowycz; Ethan Dyer (ICLR, 2022)

Exploring Length Generalization in Large Language Models. - Cem Anil; Yuhuai Wu; Anders Andreassen; Aitor Lewkowycz; Vedant Misra; Vinay V. Ramasesh; Ambrose Slone; Guy Gur-Ari; Ethan Dyer; Behnam Neyshabur (NeurIPS, 2022)

Solving Quantitative Reasoning Problems with Language Models. - Aitor Lewkowycz; Anders Andreassen; David Dohan; Ethan Dyer; Henryk Michalewski; Vinay V. Ramasesh; Ambrose Slone; Cem Anil; Imanol Schlag; Theo Gutman-Solo; Yuhuai Wu; Behnam Neyshabur; et al. (NeurIPS, 2022)

Scaling Up Models and Data with t5x and seqio. - Adam Roberts; Hyung Won Chung; Anselm Levskaya; Gaurav Mishra; James Bradbury; Daniel Andor; Sharan Narang; Brian Lester; Colin Gaffney; Afroz Mohiuddin; Curtis Hawthorne; Aitor Lewkowycz; et al. (CoRR, 2022)

Language Model Cascades. - David Dohan; Winnie Xu; Aitor Lewkowycz; Jacob Austin; David Bieber; Raphael Gontijo Lopes; Yuhuai Wu; Henryk Michalewski; Rif A. Saurous; Jascha Sohl-Dickstein; Kevin Murphy 002; Charles Sutton (CoRR, 2022)

PaLM: Scaling Language Modeling with Pathways. - Aakanksha Chowdhery; Sharan Narang; Jacob Devlin; Maarten Bosma; Gaurav Mishra; Adam Roberts; Paul Barham 001; Hyung Won Chung; Charles Sutton; Sebastian Gehrmann; Parker Schuh; Kensen Shi; et al. (J. Mach. Learn. Res., 2023)

Scaling Up Models and Data with t5x and seqio. - Adam Roberts; Hyung Won Chung; Gaurav Mishra; Anselm Levskaya; James Bradbury; Daniel Andor; Sharan Narang; Brian Lester; Colin Gaffney; Afroz Mohiuddin; Curtis Hawthorne; Aitor Lewkowycz; et al. (J. Mach. Learn. Res., 2023)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. - Aarohi Srivastava; Abhinav Rastogi; Abhishek Rao; Abu Awal Md Shoeb; Abubakar Abid; Adam Fisch; Adam R. Brown; Adam Santoro; Aditya Gupta; Adrià Garriga-Alonso; Agnieszka Kluska; Aitor Lewkowycz; et al. (Trans. Mach. Learn. Res., 2023)