ETC: Encoding Long and Structured Inputs in Transformers. - Joshua Ainslie; Santiago Ontañón; Chris Alberti; Vaclav Cvicek; Zachary Fisher; Philip Pham; Anirudh Ravula; Sumit Sanghai; Qifan Wang 001; Li Yang (EMNLP (1), 2020)

Big Bird: Transformers for Longer Sequences. - Manzil Zaheer; Guru Guruganesh; Kumar Avinava Dubey; Joshua Ainslie; Chris Alberti; Santiago Ontañón; Philip Pham; Anirudh Ravula; Qifan Wang 001; Li Yang; Amr Ahmed 001 (NeurIPS, 2020)

ETC: Encoding Long and Structured Data in Transformers. - Joshua Ainslie; Santiago Ontañón; Chris Alberti; Philip Pham; Anirudh Ravula; Sumit Sanghai (CoRR, 2020)

RealFormer: Transformer Likes Residual Attention. - Ruining He; Anirudh Ravula; Bhargav Kanagal; Joshua Ainslie (ACL/IJCNLP (Findings), 2021)

Improving Compositional Generalization in Classification Tasks via Structure Annotations. - Juyong Kim 002; Pradeep Ravikumar; Joshua Ainslie; Santiago Ontañón (ACL/IJCNLP (2), 2021)

ReadTwice: Reading Very Large Documents with Memories. - Yury Zemlyanskiy; Joshua Ainslie; Michiel de Jong; Philip Pham; Ilya Eckstein; Fei Sha (NAACL-HLT, 2021)

ShopTalk: A System for Conversational Faceted Search. - Gurmeet Manku; James Lee-Thorp; Bhargav Kanagal; Joshua Ainslie; Jingchen Feng; Zach Pearson; Ebenezer Anjorin; Sudeep Gandhe; Ilya Eckstein; Jim Rosswog; Sumit Sanghai; Michael Pohl; et al. (CoRR, 2021)

Iterative Decoding for Compositional Generalization in Transformers. - Luana Ruiz; Joshua Ainslie; Santiago Ontañón (CoRR, 2021)

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction. - Chen-Yu Lee; Chun-Liang Li; Timothy Dozat; Vincent Perot; Guolong Su; Nan Hua; Joshua Ainslie; Renshen Wang; Yasuhisa Fujii; Tomas Pfister (ACL (1), 2022)

Making Transformers Solve Compositional Tasks. - Santiago Ontañón; Joshua Ainslie; Zachary Fisher; Vaclav Cvicek (ACL (1), 2022)

Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing. - Yury Zemlyanskiy; Michiel de Jong; Joshua Ainslie; Panupong Pasupat; Peter Shaw 004; Linlu Qiu; Sumit Sanghai; Fei Sha (COLING, 2022)

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. - James Lee-Thorp; Joshua Ainslie (EMNLP (Findings), 2022)

LongT5: Efficient Text-To-Text Transformer for Long Sequences. - Mandy Guo; Joshua Ainslie; David C. Uthus; Santiago Ontañón; Jianmo Ni; Yun-Hsuan Sung; Yinfei Yang (NAACL-HLT (Findings), 2022)

FNet: Mixing Tokens with Fourier Transforms. - James Lee-Thorp; Joshua Ainslie; Ilya Eckstein; Santiago Ontañón (NAACL-HLT, 2022)

LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models. - Santiago Ontañón; Joshua Ainslie; Vaclav Cvicek; Zachary Fisher (CoRR, 2022)

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference. - Michiel de Jong; Yury Zemlyanskiy; Joshua Ainslie; Nicholas FitzGerald; Sumit Sanghai; Fei Sha; William W. Cohen (ACL (Findings), 2023)

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction. - Chen-Yu Lee; Chun-Liang Li; Hao Zhang; Timothy Dozat; Vincent Perot; Guolong Su; Xiang Zhang; Kihyuk Sohn; Nikolay Glushnev; Renshen Wang; Joshua Ainslie; Shangbang Long; et al. (ACL (1), 2023)

CoLT5: Faster Long-Range Transformers with Conditional Computation. - Joshua Ainslie; Tao Lei; Michiel de Jong; Santiago Ontañón; Siddhartha Brahma; Yury Zemlyanskiy; David C. Uthus; Mandy Guo; James Lee-Thorp; Yi Tay; Yun-Hsuan Sung; Sumit Sanghai (EMNLP, 2023)

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. - Joshua Ainslie; James Lee-Thorp; Michiel de Jong; Yury Zemlyanskiy; Federico Lebrón; Sumit Sanghai (EMNLP, 2023)

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. - Andrea Burns; Krishna Srinivasan; Joshua Ainslie; Geoff Brown; Bryan A. Plummer; Kate Saenko; Jianmo Ni; Mandy Guo (EMNLP, 2023)

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences. - David C. Uthus; Santiago Ontañón; Joshua Ainslie; Mandy Guo (EMNLP (Findings), 2023)

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. - Aran Komatsuzaki; Joan Puigcerver; James Lee-Thorp; Carlos Riquelme Ruiz; Basil Mustafa; Joshua Ainslie; Yi Tay; Mostafa Dehghani 001; Neil Houlsby (ICLR, 2023)

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute. - Michiel de Jong; Yury Zemlyanskiy; Nicholas FitzGerald; Joshua Ainslie; Sumit Sanghai; Fei Sha; William W. Cohen (ICML, 2023)

Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. - Tao Lei; Junwen Bai; Siddhartha Brahma; Joshua Ainslie; Kenton Lee; Yanqi Zhou; Nan Du 002; Vincent Y. Zhao; Yuexin Wu; Bo Li 028; Yu Zhang 033; Ming-Wei Chang (NeurIPS, 2023)

WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset. - Andrea Burns; Krishna Srinivasan; Joshua Ainslie; Geoff Brown; Bryan A. Plummer; Kate Saenko; Jianmo Ni; Mandy Guo (CoRR, 2023)

GLIMMER: generalized late-interaction memory reranker. - Michiel de Jong; Yury Zemlyanskiy; Nicholas FitzGerald; Sumit Sanghai; William W. Cohen; Joshua Ainslie (CoRR, 2023)

Functional Interpolation for Relative Positions improves Long Context Transformers. - Shanda Li; Chong You; Guru Guruganesh; Joshua Ainslie; Santiago Ontañón; Manzil Zaheer; Sumit Sanghai; Yiming Yang; Sanjiv Kumar; Srinadh Bhojanapalli (ICLR, 2024)

MEMORY-VQ: Compression for Tractable Internet-Scale Memory. - Yury Zemlyanskiy; Michiel de Jong; Luke Vilnis; Santiago Ontañón; William W. Cohen; Sumit Sanghai; Joshua Ainslie (NAACL (Short Papers), 2024)

Linear Transformer Topological Masking with Graph Random Features. - Isaac Reid; Kumar Avinava Dubey; Deepali Jain; William F. Whitney; Amr Ahmed 001; Joshua Ainslie; Alex Bewley; Mithun George Jacob; Aranyak Mehta; David Rendleman; Connor Schenck; Richard E. Turner; et al. (ICLR, 2025)

Learning the RoPEs: Better 2D and 3D Position Encodings with STRING. - Connor Schenck; Isaac Reid; Mithun George Jacob; Alex Bewley; Joshua Ainslie; David Rendleman; Deepali Jain; Mohit Sharma 001; Avinava Dubey; Ayzaan Wahid; Sumeet Singh; René Wagner; et al. (CoRR, 2025)

Gemini Robotics: Bringing AI into the Physical World. - Gemini Robotics Team; Saminda Abeyruwan; Joshua Ainslie; Jean-Baptiste Alayrac; Montserrat Gonzalez Arenas; Travis Armstrong; Ashwin Balakrishna; Robert Baruch; Maria Bauzá 001; Michiel Blokzijl; Steven Bohez; Konstantinos Bousmalis; et al. (CoRR, 2025)

Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation. - Biao Zhang; Fedor Moiseev; Joshua Ainslie; Paul Suganthan; Min Ma; Surya Bhupatiraju; Federico Lebrón; Orhan Firat; Armand Joulin; Zhe Dong (CoRR, 2025)