Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs. - Alex Warstadt; Yu Cao; Ioana Grosu; Wei Peng 013; Hagen Blix; Yining Nie; Anna Alsop; Shikha Bordia; Haokun Liu; Alicia Parrish; Sheng-Fu Wang; Jason Phang; et al. (EMNLP/IJCNLP (1), 2019)

BLiMP: The Benchmark of Linguistic Minimal Pairs for English. - Alex Warstadt; Alicia Parrish; Haokun Liu; Anhad Mohananey; Wei Peng 013; Sheng-Fu Wang; Samuel R. Bowman (Trans. Assoc. Comput. Linguistics, 2020)

Erratum: "BLiMP: The Benchmark of Linguistic Minimal Pairs for English". - Alex Warstadt; Alicia Parrish; Haokun Liu; Anhad Mohananey; Wei Peng 013; Sheng-Fu Wang; Samuel R. Bowman (Trans. Assoc. Comput. Linguistics, 2020)

NOPE: A Corpus of Naturally-Occurring Presuppositions in English. - Alicia Parrish; Sebastian Schuster; Alex Warstadt; Omar Agha; Soo-Hwan Lee; Zhuoye Zhao; Samuel R. Bowman; Tal Linzen (CoNLL, 2021)

Does Putting a Linguist in the Loop Improve NLU Data Collection? - Alicia Parrish; William Huang; Omar Agha; Soo-Hwan Lee; Nikita Nangia; Alex Warstadt; Karmanya Aggarwal; Emily Allaway; Tal Linzen; Samuel R. Bowman (EMNLP (Findings), 2021)

BBQ: A hand-built bias benchmark for question answering. - Alicia Parrish; Angelica Chen; Nikita Nangia; Vishakh Padmakumar; Jason Phang; Jana Thompson; Phu Mon Htut; Samuel R. Bowman (ACL (Findings), 2022)

QuALITY: Question Answering with Long Input Texts, Yes! - Richard Yuanzhe Pang; Alicia Parrish; Nitish Joshi; Nikita Nangia; Jason Phang; Angelica Chen; Vishakh Padmakumar; Johnny Ma; Jana Thompson; He He 001; Samuel R. Bowman (NAACL-HLT, 2022)

Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions. - Alicia Parrish; Harsh Trivedi; Ethan Perez; Angelica Chen; Nikita Nangia; Jason Phang; Samuel R. Bowman (CoRR, 2022)

Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions. - Alicia Parrish; Harsh Trivedi; Nikita Nangia; Vishakh Padmakumar; Jason Phang; Amanpreet Singh Saimbhi; Samuel R. Bowman (CoRR, 2022)

What Do NLP Researchers Believe? Results of the NLP Community Metasurvey. - Julian Michael; Ari Holtzman; Alicia Parrish; Aaron Mueller; Alex Wang; Angelica Chen; Divyam Madaan; Nikita Nangia; Richard Yuanzhe Pang; Jason Phang; Samuel R. Bowman (ACL (1), 2023)

DICES Dataset: Diversity in Conversational AI Evaluation for Safety. - Lora Aroyo; Alex S. Taylor; Mark Díaz; Christopher Homan; Alicia Parrish; Gregory Serapio-García; Vinodkumar Prabhakaran; Ding Wang (NeurIPS, 2023)

DataPerf: Benchmarks for Data-Centric AI Development. - Mark Mazumder; Colby R. Banbury; Xiaozhe Yao; Bojan Karlas; William Gaviria Rojas; Sudnya Frederick Diamos; Greg Diamos; Lynn He; Alicia Parrish; Hannah Rose Kirk; Jessica Quaye; Charvi Rastogi; et al. (NeurIPS, 2023)

Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models. - Alicia Parrish; Hannah Rose Kirk; Jessica Quaye; Charvi Rastogi; Max Bartolo; Oana Inel; Juan Ciro; Rafael Mosquera; Addison Howard; Will Cukierski; D. Sculley; Vijay Janapa Reddi; et al. (CoRR, 2023)

Intersectionality in Conversational AI Safety: How Bayesian Multilevel Models Help Understand Diverse Perceptions of Safety. - Christopher M. Homan; Greg Serapio-García; Lora Aroyo; Mark Diaz; Alicia Parrish; Vinodkumar Prabhakaran; Alex S. Taylor; Ding Wang (CoRR, 2023)

"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision models. - Alicia Parrish; Sarah Laszlo; Lora Aroyo (CoRR, 2023)

A Framework to Assess (Dis)agreement Among Diverse Rater Groups. - Vinodkumar Prabhakaran; Christopher Homan; Lora Aroyo; Alicia Parrish; Alex S. Taylor; Mark Díaz; Ding Wang (CoRR, 2023)

DMLR: Data-centric Machine Learning Research - Past, Present and Future. - Luis Oala; Manil Maskey; Lilith Bat-Leah; Alicia Parrish; Nezihe Merve Gürel; Tzu-Sheng Kuo; Yang Liu 018; Rotem Dror; Danilo Brajovic; Xiaozhe Yao; Max Bartolo; William Gaviria Rojas; et al. (CoRR, 2023)

Inverse Scaling: When Bigger Isn't Better. - Ian R. McKenzie; Alexander Lyzhov; Michael Pieler; Alicia Parrish; Aaron Mueller; Ameya Prabhu; Euan McLean; Aaron Kirtland; Alexis Ross; Alisa Liu; Andrew Gritsevskiy; Daniel Wurgaft; et al. (Trans. Mach. Learn. Res., 2023)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. - Aarohi Srivastava; Abhinav Rastogi; Abhishek Rao; Abu Awal Md Shoeb; Abubakar Abid; Adam Fisch; Adam R. Brown; Adam Santoro; Aditya Gupta; Adrià Garriga-Alonso; Agnieszka Kluska; Aitor Lewkowycz; et al. (Trans. Mach. Learn. Res., 2023)

Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation. - Jessica Quaye; Alicia Parrish; Oana Inel; Charvi Rastogi; Hannah Rose Kirk; Minsuk Kahng; Erin van Liemt; Max Bartolo; Jess Tsang; Justin White; Nathan Clement; Rafael Mosquera; et al. (FAccT, 2024)

GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives. - Vinodkumar Prabhakaran; Christopher Homan; Lora Aroyo; Aida Mostafazadeh Davani; Alicia Parrish; Alex S. Taylor; Mark Diaz; Ding Wang; Gregory Serapio-García (NAACL-HLT, 2024)

A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models. - Stephen R. Pfohl; Heather Cole-Lewis; Rory Sayres; Darlene Neal; Mercy Nyamewaa Asiedu; Awa Dieng; Nenad Tomasev; Qazi Mamunur Rashid; Shekoofeh Azizi; Negar Rostamzadeh; Liam G. McCoy; Leo Anthony Celi; et al. (CoRR, 2024)

Introducing v0.5 of the AI Safety Benchmark from MLCommons. - Bertie Vidgen; Adarsh Agrawal; Ahmed M. Ahmed 004; Victor Akinwande; Namir Al-Nuaimi; Najla Alfaraj; Elie Alhajjar; Lora Aroyo; Trupti Bavalatti; Borhane Blili-Hamelin; Kurt D. Bollacker; Rishi Bomassani; et al. (CoRR, 2024)

Gemma 2: Improving Open Language Models at a Practical Size. - Morgane Rivière; Shreya Pathak; Pier Giuseppe Sessa; Cassidy Hardin; Surya Bhupatiraju; Léonard Hussenot; Thomas Mesnard; Bobak Shahriari; Alexandre Ramé; Johan Ferret; Peter Liu; Pouya Tafti; et al. (CoRR, 2024)

Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups. - Charvi Rastogi; Tian Huey Teh; Pushkar Mishra; Roma Patel; Zoe Ashwood; Aida Mostafazadeh Davani; Mark Diaz; Michela Paganini; Alicia Parrish; Ding Wang; Vinodkumar Prabhakaran; Lora Aroyo; et al. (CoRR, 2024)

Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs. - Angelica Chen; Jason Phang; Alicia Parrish; Vishakh Padmakumar; Chen Zhao; Samuel R. Bowman; Kyunghyun Cho (Trans. Mach. Learn. Res., 2024)

MSTS: A Multimodal Safety Test Suite for Vision-Language Models. - Paul Röttger; Giuseppe Attanasio; Felix Friedrich; Janis Goldzycher; Alicia Parrish; Rishabh Bhardwaj; Chiara Di Bonaventura; Roman Eng; Gaia El Khoury Geagea; Subhra S. Goswami; Jieun Han; Dirk Hovy; et al. (CoRR, 2025)

Nuanced Safety for Generative AI: How Demographics Shape Responsiveness to Severity. - Pushkar Mishra; Charvi Rastogi; Stephen R. Pfohl; Alicia Parrish; Roma Patel; Mark Diaz; Ding Wang; Michela Paganini; Vinodkumar Prabhakaran; Lora Aroyo; Verena Rieser (CoRR, 2025)

AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons. - Shaona Ghosh; Heather Frase; Adina Williams; Sarah Luger; Paul Röttger; Fazl Barez; Sean McGregor; Kenneth Fricklas; Mala Kumar; Quentin Feuillade-Montixi; Kurt Bollacker; Felix Friedrich; et al. (CoRR, 2025)

Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Models. - Charvi Rastogi; Tian Huey Teh; Pushkar Mishra; Roma Patel; Ding Wang; Mark Díaz; Alicia Parrish; Aida Mostafazadeh Davani; Zoe Ashwood; Michela Paganini; Vinodkumar Prabhakaran; Verena Rieser; et al. (CoRR, 2025)

"Just a strange pic": Evaluating 'safety' in GenAI Image safety annotation tasks from diverse annotators' perspectives. - Ding Wang; Mark Díaz; Charvi Rastogi; Aida Mostafazadeh Davani; Vinodkumar Prabhakaran; Pushkar Mishra; Roma Patel; Alicia Parrish; Zoe Ashwood; Michela Paganini; Tian Huey Teh; Verena Rieser; et al. (CoRR, 2025)

From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models. - Jessica Quaye; Charvi Rastogi; Alicia Parrish; Oana Inel; Minsuk Kahng; Lora Aroyo; Vijay Janapa Reddi (CoRR, 2025)