Joshua Ainslie

According to our database, Joshua Ainslie authored at least 28 papers between 2020 and 2023.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of three.

2023

Functional Interpolation for Relative Positions Improves Long Context Transformers.

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli

CoRR, 2023

MEMORY-VQ: Compression for Tractable Internet-Scale Memory.

Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontañón, William W. Cohen, Sumit Sanghai, Joshua Ainslie

CoRR, 2023

GLIMMER: generalized late-interaction memory reranker.

Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie

CoRR, 2023

WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset.

Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo

CoRR, 2023

Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference.

Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute.

Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William W. Cohen

Proceedings of the International Conference on Machine Learning, 2023

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints.

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby

Proceedings of the Eleventh International Conference on Learning Representations, 2023

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences.

David C. Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo

Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, 2023

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding.

Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

CoLT5: Faster Long-Range Transformers with Conditional Computation.

Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction.

Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolay Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference.

Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William W. Cohen

Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, 2023

2022

LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models.

Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher

CoRR, 2022

FNet: Mixing Tokens with Fourier Transforms.

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022

LongT5: Efficient Text-To-Text Transformer for Long Sequences.

Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang

Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2022, 2022

Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT.

James Lee-Thorp, Joshua Ainslie

Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022, 2022

Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing.

Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha

Proceedings of the 29th International Conference on Computational Linguistics, 2022

Making Transformers Solve Compositional Tasks.

Santiago Ontañón, Joshua Ainslie, Zachary Fisher, Vaclav Cvicek

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction.

Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022

2021

Iterative Decoding for Compositional Generalization in Transformers.

Luana Ruiz, Joshua Ainslie, Santiago Ontañón

CoRR, 2021

ShopTalk: A System for Conversational Faceted Search.

Gurmeet Manku, James Lee-Thorp, Bhargav Kanagal, Joshua Ainslie, Jingchen Feng, Zach Pearson, Ebenezer Anjorin, Sudeep Gandhe, Ilya Eckstein, Jim Rosswog, Sumit Sanghai, Michael Pohl, Larry Adams, D. Sivakumar

CoRR, 2021

ReadTwice: Reading Very Large Documents with Memories.

Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021

Improving Compositional Generalization in Classification Tasks via Structure Annotations.

Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021

RealFormer: Transformer Likes Residual Attention.

Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie

Proceedings of the Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, 2021

2020

ETC: Encoding Long and Structured Data in Transformers.

Joshua Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, Sumit Sanghai

CoRR, 2020

Big Bird: Transformers for Longer Sequences.

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

ETC: Encoding Long and Structured Inputs in Transformers.

Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020