The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations. - Nikita Nangia; Adina Williams; Angeliki Lazaridou; Samuel R. Bowman (RepEval@EMNLP, 2017)

Learning to parse from a semantic objective: It works. Is it syntax? - Adina Williams; Andrew Drozdov; Samuel R. Bowman (CoRR, 2017)

XNLI: Evaluating Cross-lingual Sentence Representations. - Alexis Conneau; Ruty Rinott; Guillaume Lample; Adina Williams; Samuel R. Bowman; Holger Schwenk; Veselin Stoyanov (EMNLP, 2018)

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. - Adina Williams; Nikita Nangia; Samuel R. Bowman (NAACL-HLT, 2018)

Verb Argument Structure Alternations in Word and Sentence Embeddings. - Katharina Kann; Alex Warstadt; Adina Williams; Samuel R. Bowman (CoRR, 2018)

Do latent tree learning models identify meaningful structure in sentences? - Adina Williams; Andrew Drozdov; Samuel R. Bowman (Trans. Assoc. Comput. Linguistics, 2018)

Quantifying the Semantic Core of Gender Systems. - Adina Williams; Damián E. Blasi; Lawrence Wolf-Sonkin; Hanna M. Wallach; Ryan Cotterell (EMNLP/IJCNLP (1), 2019)

On the Idiosyncrasies of the Mandarin Chinese Classifier System. - Shijia Liu; Hongyuan Mei; Adina Williams; Ryan Cotterell (NAACL-HLT (1), 2019)

Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition. - Paloma Jeretic; Alex Warstadt; Suvrat Bhooshan; Adina Williams (ACL, 2020)

A Tale of a Probe and a Parser. - Rowan Hall Maudslay; Josef Valvoda; Tiago Pimentel; Adina Williams; Ryan Cotterell (ACL, 2020)

Adversarial NLI: A New Benchmark for Natural Language Understanding. - Yixin Nie; Adina Williams; Emily Dinan; Mohit Bansal; Jason Weston; Douwe Kiela (ACL, 2020)

Information-Theoretic Probing for Linguistic Structure. - Tiago Pimentel; Josef Valvoda; Rowan Hall Maudslay; Ran Zmigrod; Adina Williams; Ryan Cotterell (ACL, 2020)

Predicting Declension Class from Form and Meaning. - Adina Williams; Tiago Pimentel; Hagen Blix; Arya D. McCarthy; Eleanor Chodroff; Ryan Cotterell (ACL, 2020)

Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation. - Emily Dinan; Angela Fan; Adina Williams; Jack Urbanek; Douwe Kiela; Jason Weston (EMNLP (1), 2020)

Multi-Dimensional Gender Bias Classification. - Emily Dinan; Angela Fan; Ledell Wu; Jason Weston; Douwe Kiela; Adina Williams (EMNLP (1), 2020)

Intrinsic Probing through Dimension Selection. - Lucas Torroba Hennigen; Adina Williams; Ryan Cotterell (EMNLP (1), 2020)

Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions. - Arya D. McCarthy; Adina Williams; Shijia Liu; David Yarowsky; Ryan Cotterell (EMNLP (1), 2020)

Pareto Probing: Trading Off Accuracy for Complexity. - Tiago Pimentel; Naomi Saphra; Adina Williams; Ryan Cotterell (EMNLP (1), 2020)

SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection. - Ekaterina Vylomova; Jennifer C. White; Elizabeth Salesky; Sabrina J. Mielke; Shijie Wu; Edoardo Maria Ponti; Rowan Hall Maudslay; Ran Zmigrod; Josef Valvoda; Svetlana Toldova; Francis M. Tyers; Elena Klyachko; et al. (SIGMORPHON, 2020)

ANLIzing the Adversarial Natural Language Inference Dataset. - Adina Williams; Tristan Thrush; Douwe Kiela (CoRR, 2020)

UnNatural Language Inference. - Koustuv Sinha; Prasanna Parthasarathi; Joelle Pineau; Adina Williams (ACL/IJCNLP (1), 2021)

To what extent do human explanations of model behavior align with actual model behavior? - Grusha Prasad; Yixin Nie; Mohit Bansal; Robin Jia; Douwe Kiela; Adina Williams (BlackboxNLP@EMNLP, 2021)

Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network. - Verna Dankers; Anna Langedijk; Kate McCurdy; Adina Williams; Dieuwke Hupkes (CoNLL, 2021)

Sometimes We Want Ungrammatical Translations. - Prasanna Parthasarathi; Koustuv Sinha; Joelle Pineau; Adina Williams (EMNLP (Findings), 2021)

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little. - Koustuv Sinha; Robin Jia; Dieuwke Hupkes; Joelle Pineau; Adina Williams; Douwe Kiela (EMNLP (1), 2021)

Dynabench: Rethinking Benchmarking in NLP. - Douwe Kiela; Max Bartolo; Yixin Nie; Divyansh Kaushik; Atticus Geiger; Zhengxuan Wu; Bertie Vidgen; Grusha Prasad; Amanpreet Singh; Pratik Ringshia; Zhiyi Ma; Tristan Thrush; et al. (NAACL-HLT, 2021)

Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking. - Zhiyi Ma; Kawin Ethayarajh; Tristan Thrush; Somya Jain; Ledell Wu; Robin Jia; Christopher Potts; Adina Williams; Douwe Kiela (NeurIPS, 2021)

Sometimes We Want Translationese. - Prasanna Parthasarathi; Koustuv Sinha; Joelle Pineau; Adina Williams (CoRR, 2021)

Investigating Failures of Automatic Translation in the Case of Unambiguous Gender. - Adithya Renduchintala; Adina Williams (CoRR, 2021)

Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models. - Eric Michael Smith; Adina Williams (CoRR, 2021)

A Word on Machine Ethics: A Response to Jiang et al. (2021). - Zeerak Talat; Hagen Blix; Josef Valvoda; Maya Indira Ganesh; Ryan Cotterell; Adina Williams (CoRR, 2021)

On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs. - Adina Williams; Lawrence Wolf-Sonkin; Damián E. Blasi; Hanna M. Wallach; Ryan Cotterell (Trans. Assoc. Comput. Linguistics, 2021)

Investigating Failures of Automatic Translationin the Case of Unambiguous Gender. - Adi Renduchintala; Adina Williams (ACL (1), 2022)

Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks. - Tristan Thrush; Kushal Tirumala; Anmol Gupta; Max Bartolo; Pedro Rodriguez 001; Tariq Kane; William Gaviria Rojas; Peter Mattson; Adina Williams; Douwe Kiela (ACL (demo), 2022)

Analyzing Dynamic Adversarial Training Data in the Limit. - Eric Wallace; Adina Williams; Robin Jia; Douwe Kiela (ACL (Findings), 2022)

Evaluating locality in NMT models. - Itay Itzhak; Koustuv Sinha; Brenden M. Lake; Adina Williams; Dieuwke Hupkes (CogSci, 2022)

Benchmarking Compositionality with Formal Languages. - Josef Valvoda; Naomi Saphra; Jonathan Rawski; Adina Williams; Ryan Cotterell (COLING, 2022)

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. - Tristan Thrush; Ryan Jiang; Max Bartolo; Amanpreet Singh; Adina Williams; Douwe Kiela; Candace Ross (CVPR, 2022)

Perturbation Augmentation for Fairer NLP. - Rebecca Qian; Candace Ross; Jude Fernandes; Eric Michael Smith; Douwe Kiela; Adina Williams (EMNLP, 2022)

The Curious Case of Absolute Position Embeddings. - Koustuv Sinha; Amirhossein Kazemnejad; Siva Reddy; Joelle Pineau; Dieuwke Hupkes; Adina Williams (EMNLP (Findings), 2022)

"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset. - Eric Michael Smith; Melissa Hall; Melanie Kambadur; Eleonora Presani; Adina Williams (EMNLP, 2022)

On the Machine Learning of Ethical Judgments from Natural Language. - Zeerak Talat; Hagen Blix; Josef Valvoda; Maya Indira Ganesh; Ryan Cotterell; Adina Williams (NAACL-HLT, 2022)

"I'm sorry to hear that": finding bias in language models with a holistic descriptor dataset. - Eric Michael Smith; Melissa Hall; Melanie Kambadur; Eleonora Presani; Adina Williams (CoRR, 2022)

DataPerf: Benchmarks for Data-Centric AI Development. - Mark Mazumder; Colby R. Banbury; Xiaozhe Yao; Bojan Karlas; William Gaviria Rojas; Sudnya Frederick Diamos; Greg Diamos; Lynn He; Douwe Kiela; David Jurado; David Kanter; Rafael Mosquera; et al. (CoRR, 2022)

Learning Transductions to Test Systematic Compositionality. - Josef Valvoda; Naomi Saphra; Jonathan Rawski; Ryan Cotterell; Adina Williams (CoRR, 2022)

A Latent-Variable Model for Intrinsic Probing. - Karolina Stanczak; Lucas Torroba Hennigen; Adina Williams; Ryan Cotterell; Isabelle Augenstein (AAAI, 2023)

Language model acceptability judgements are not always robust to context. - Koustuv Sinha; Jon Gauthier; Aaron Mueller; Kanishka Misra; Keren Fuentes; Roger Levy; Adina Williams (ACL (1), 2023)

The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks. - Kaiser Sun; Adina Williams; Dieuwke Hupkes (CoNLL, 2023)

ROBBIE: Robust Bias Evaluation of Large Generative Language Models. - David Esiobu; Xiaoqing Ellen Tan; Saghar Hosseini; Megan Ung; Yuchen Zhang; Jude Fernandes; Jane Dwivedi-Yu; Eleonora Presani; Adina Williams; Eric Michael Smith (EMNLP, 2023)

Robustness of Named-Entity Replacements for In-Context Learning. - Saeed Goodarzi; Nikhil Kagita; Dennis Minn; Shufan Wang; Roberto Dessì; Shubham Toshniwal; Adina Williams; Jack Lanchantin; Koustuv Sinha (EMNLP (Findings), 2023)

DataPerf: Benchmarks for Data-Centric AI Development. - Mark Mazumder; Colby R. Banbury; Xiaozhe Yao; Bojan Karlas; William Gaviria Rojas; Sudnya Frederick Diamos; Greg Diamos; Lynn He; Alicia Parrish; Hannah Rose Kirk; Jessica Quaye; Charvi Rastogi; et al. (NeurIPS, 2023)

The Gender-GAP Pipeline: A Gender-Aware Polyglot Pipeline for Gender Characterisation in 55 Languages. - Benjamin Muller; Belen Alastruey; Prangthip Hansanti; Elahe Kalbassi; Christophe Ropers; Eric Michael Smith; Adina Williams; Luke Zettlemoyer; Pierre Andrews; Marta R. Costa-jussà (WMT, 2023)

Call for Papers - The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. - Alex Warstadt; Leshem Choshen; Aaron Mueller; Adina Williams; Ethan Wilcox; Chengxu Zhuang (CoRR, 2023)

Weisfeiler and Lehman Go Measurement Modeling: Probing the Validity of the WL Test. - Arjun Subramonian; Adina Williams; Maximilian Nickel; Yizhou Sun; Levent Sagun (CoRR, 2023)

Llama 2: Open Foundation and Fine-Tuned Chat Models. - Hugo Touvron; Louis Martin; Kevin Stone; Peter Albert; Amjad Almahairi; Yasmine Babaei; Nikolay Bashlykov; Soumya Batra; Prajjwal Bhargava; Shruti Bhosale; Dan Bikel; Lukas Blecher; et al. (CoRR, 2023)

Grammatical Gender's Influence on Distributional Semantics: A Causal Perspective. - Karolina Stanczak; Kevin Du; Adina Williams; Isabelle Augenstein; Ryan Cotterell (CoRR, 2023)

Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality. - Da Ju; Karen Ullrich; Adina Williams (ACL (Findings), 2024)

EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models. - Maureen de Seyssel; Antony D'Avirro; Adina Williams; Emmanuel Dupoux (EMNLP, 2024)

Towards Geographic Inclusion in the Evaluation of Text-to-Image Models. - Melissa Hall; Samuel J. Bell; Candace Ross; Adina Williams; Michal Drozdzal; Adriana Romero-Soriano (FAccT, 2024)

The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. - Hannah Rose Kirk; Alexander Whitefield; Paul Röttger; Andrew M. Bean; Katerina Margatina; Rafael Mosquera Gómez; Juan Ciro; Max Bartolo; Adina Williams; He He 001; Bertie Vidgen; Scott Hale (NeurIPS, 2024)

The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More. - Ouail Kitouni; Niklas Nolte; Adina Williams; Michael Rabbat; Diane Bouchacourt; Mark Ibrahim (NeurIPS, 2024)

Compositional learning of functions in humans and machines. - Yanli Zhou; Brenden M. Lake; Adina Williams (CoRR, 2024)

[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. - Leshem Choshen; Ryan Cotterell; Michael Y. Hu; Tal Linzen; Aaron Mueller; Candace Ross; Alex Warstadt; Ethan Wilcox; Adina Williams; Chengxu Zhuang (CoRR, 2024)

Introducing v0.5 of the AI Safety Benchmark from MLCommons. - Bertie Vidgen; Adarsh Agrawal; Ahmed M. Ahmed 004; Victor Akinwande; Namir Al-Nuaimi; Najla Alfaraj; Elie Alhajjar; Lora Aroyo; Trupti Bavalatti; Borhane Blili-Hamelin; Kurt D. Bollacker; Rishi Bomassani; et al. (CoRR, 2024)

The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. - Hannah Rose Kirk; Alexander Whitefield; Paul Röttger; Andrew M. Bean; Katerina Margatina; Juan Ciro; Rafael Mosquera; Max Bartolo; Adina Williams; He He 001; Bertie Vidgen; Scott A. Hale (CoRR, 2024)

Decomposed evaluations of geographic disparities in text-to-image models. - Abhishek Sureddy; Dishant Padalia; Nandhinee Periyakaruppa; Oindrila Saha; Adina Williams; Adriana Romero-Soriano; Megan Richards; Polina Kirichenko; Melissa Hall (CoRR, 2024)

Changing Answer Order Can Decrease MMLU Accuracy. - Vipul Gupta; David Pantoja; Candace Ross; Adina Williams; Megan Ung (CoRR, 2024)

Are Female Carpenters like Blue Bananas? A Corpus Investigation of Occupation Gender Typicality. - Da Ju; Karen Ulrich; Adina Williams (CoRR, 2024)

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models. - Anaelia Ovalle; Krunoslav Lehman Pavasovic; Louis Martin; Luke Zettlemoyer; Eric Michael Smith; Adina Williams; Levent Sagun (CoRR, 2024)

Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models. - Da Ju; Adina Williams; Brian Karrer; Maximilian Nickel (CoRR, 2024)

Transformers Can Navigate Mazes With Multi-Step Prediction. - Niklas Nolte; Ouail Kitouni; Adina Williams; Mike Rabbat; Mark Ibrahim (CoRR, 2024)

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. - Michael Y. Hu; Aaron Mueller; Candace Ross; Adina Williams; Tal Linzen; Chengxu Zhuang; Ryan Cotterell; Leshem Choshen; Alex Warstadt; Ethan Gotlieb Wilcox (CoRR, 2024)

What makes a good metric? Evaluating automatic metrics for text-to-image consistency. - Candace Ross; Melissa Hall; Adriana Romero-Soriano; Adina Williams (CoRR, 2024)

Chained Tuning Leads to Biased Forgetting. - Megan Ung; Alicia Sun; Samuel J. Bell; Bhaktipriya Radharapu; Levent Sagun; Adina Williams (CoRR, 2024)

DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity. - Melissa Hall; Candace Ross; Adina Williams; Nicolas Carion; Michal Drozdzal; Adriana Romero-Soriano (Trans. Mach. Learn. Res., 2024)

Improving Text-to-Image Consistency via Automatic Prompt Optimization. - Oscar Mañas; Pietro Astolfi; Melissa Hall; Candace Ross; Jack Urbanek; Adina Williams; Aishwarya Agrawal; Adriana Romero-Soriano; Michal Drozdzal (Trans. Mach. Learn. Res., 2024)

Domain Regeneration: How well do LLMs match syntactic properties of text domains? - Da Ju; Hagen Blix; Adina Williams (ACL (Findings), 2025)

Arbiters of Ambivalence: Challenges of using LLMs in No-Consensus tasks. - Bhaktipriya Radharapu; Manon Revel; Megan Ung; Sebastian Ruder; Adina Williams (ACL (Findings), 2025)

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models. - Anaelia Ovalle; Krunoslav Lehman Pavasovic; Louis Martin; Luke Zettlemoyer; Eric Michael Smith; Kai-Wei Chang; Adina Williams; Levent Sagun (FAccT, 2025)

On the Role of Speech Data in Reducing Toxicity Detection Bias. - Samuel J. Bell; Mariano Coria Meglioli; Megan Richards; Eduardo Sánchez; Christophe Ropers; Skyler Wang; Adina Williams; Levent Sagun; Marta R. Costa-jussà (NAACL (Long Papers), 2025)

Improving Model Evaluation using SMART Filtering of Benchmark Datasets. - Vipul Gupta; Candace Ross; David Pantoja; Rebecca J. Passonneau; Megan Ung; Adina Williams (NAACL (Long Papers), 2025)

BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop. - Lucas Charpentier; Leshem Choshen; Ryan Cotterell; Mustafa Omer Gul; Michael Y. Hu; Jaap Jumelet; Tal Linzen; Jing Liu; Aaron Mueller; Candace Ross; Raj Sanjay Shah; Alex Warstadt; et al. (CoRR, 2025)

AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons. - Shaona Ghosh; Heather Frase; Adina Williams; Sarah Luger; Paul Röttger; Fazl Barez; Sean McGregor; Kenneth Fricklas; Mala Kumar; Quentin Feuillade-Montixi; Kurt Bollacker; Felix Friedrich; et al. (CoRR, 2025)

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. - Alex Warstadt; Aaron Mueller; Leshem Choshen; Ethan Wilcox; Chengxu Zhuang; Juan Ciro; Rafael Mosquera; Bhargavi Paranjape; Adina Williams; Tal Linzen; Ryan Cotterell (CoRR, 2025)

Do different prompting methods yield a common task representation in language models? - Guy Davidson; Todd M. Gureckis; Brenden M. Lake; Adina Williams (CoRR, 2025)

IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments. - Florian Bordes; Quentin Garrido; Justine T. Kao; Adina Williams; Michael Rabbat; Emmanuel Dupoux (CoRR, 2025)

FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality. - Mingda Chen; Yang Li; Xilun Chen 002; Adina Williams; Gargi Ghosh; Scott Yih (CoRR, 2025)