ShopTalk: A System for Conversational Faceted Search. - Gurmeet Manku; James Lee-Thorp; Bhargav Kanagal; Joshua Ainslie; Jingchen Feng; Zach Pearson; Ebenezer Anjorin; Sudeep Gandhe; Ilya Eckstein; Jim Rosswog; Sumit Sanghai; Michael Pohl; et al. (CoRR, 2021)

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. - James Lee-Thorp; Joshua Ainslie (EMNLP (Findings), 2022)

FNet: Mixing Tokens with Fourier Transforms. - James Lee-Thorp; Joshua Ainslie; Ilya Eckstein; Santiago Ontañón (NAACL-HLT, 2022)

Scaling Up Models and Data with t5x and seqio. - Adam Roberts; Hyung Won Chung; Anselm Levskaya; Gaurav Mishra; James Bradbury; Daniel Andor; Sharan Narang; Brian Lester; Colin Gaffney; Afroz Mohiuddin; Curtis Hawthorne; Aitor Lewkowycz; et al. (CoRR, 2022)

CoLT5: Faster Long-Range Transformers with Conditional Computation. - Joshua Ainslie; Tao Lei; Michiel de Jong; Santiago Ontañón; Siddhartha Brahma; Yury Zemlyanskiy; David C. Uthus; Mandy Guo; James Lee-Thorp; Yi Tay; Yun-Hsuan Sung; Sumit Sanghai (EMNLP, 2023)

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. - Joshua Ainslie; James Lee-Thorp; Michiel de Jong; Yury Zemlyanskiy; Federico Lebrón; Sumit Sanghai (EMNLP, 2023)

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. - Aran Komatsuzaki; Joan Puigcerver; James Lee-Thorp; Carlos Riquelme Ruiz; Basil Mustafa; Joshua Ainslie; Yi Tay; Mostafa Dehghani 001; Neil Houlsby (ICLR, 2023)

Scaling Up Models and Data with t5x and seqio. - Adam Roberts; Hyung Won Chung; Gaurav Mishra; Anselm Levskaya; James Bradbury; Daniel Andor; Sharan Narang; Brian Lester; Colin Gaffney; Afroz Mohiuddin; Curtis Hawthorne; Aitor Lewkowycz; et al. (J. Mach. Learn. Res., 2023)

Memory Augmented Language Models through Mixture of Word Experts. - Cícero Nogueira dos Santos; James Lee-Thorp; Isaac Noble; Chung-Ching Chang; David C. Uthus (NAACL-HLT, 2024)