Pitfalls in Machine Learning Research: Reexamining the Development Cycle. - Stella Biderman; Walter J. Scheirer (ICBINB@NeurIPS, 2020)

Magic: the Gathering is as Hard as Arithmetic. - Stella Biderman (CoRR, 2020)

Magic: The Gathering Is Turing Complete. - Alex Churchill; Stella Biderman; Austin Herrick (FUN, 2021)

The Pile: An 800GB Dataset of Diverse Text for Language Modeling. - Leo Gao; Stella Biderman; Sid Black; Laurence Golding; Travis Hoppe; Charles Foster; Jason Phang; Horace He; Anish Thite; Noa Nabeshima; Shawn Presser; Connor Leahy (CoRR, 2021)

Towards a Formal Model of Narratives. - Louis Castricato; Stella Biderman; Rogelio E. Cardona-Rivera; David Thue (CoRR, 2021)

Cut the CARP: Fishing for zero-shot story evaluation. - Shahbuland Matiana; J. R. Smith; Ryan Teehan; Louis Castricato; Stella Biderman; Leo Gao; Spencer Frazier (CoRR, 2021)

Multitask Prompted Training Enables Zero-Shot Task Generalization. - Victor Sanh; Albert Webson; Colin Raffel; Stephen H. Bach; Lintang Sutawika; Zaid Alyafeai; Antoine Chaffin; Arnaud Stiegler; Teven Le Scao; Arun Raja; Manan Dey; M. Saiful Bari; et al. (CoRR, 2021)

Fooling MOSS Detection with Pretrained Language Models. - Stella Biderman; Edward Raff (CIKM, 2022)

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. - Katherine Crowson; Stella Biderman; Daniel Kornis; Dashiell Stander; Eric Hallahan; Louis Castricato; Edward Raff (ECCV (37), 2022)

What Language Model to Train if You Have One Million GPU Hours? - Teven Le Scao; Thomas Wang; Daniel Hesslow; Lucile Saulnier; Stas Bekman; M. Saiful Bari; Stella Biderman; Hady Elsahar; Niklas Muennighoff; Jason Phang; Ofir Press; Colin Raffel; et al. (EMNLP (Findings), 2022)

Data Governance in the Age of Large-Scale Data-Driven Language Technology. - Yacine Jernite; Huu Nguyen; Stella Biderman; Anna Rogers; Maraim Masoud; Valentin Danchev; Samson Tan; Alexandra Sasha Luccioni; Nishant Subramani; Isaac Johnson 001; Gérard Dupont; Jesse Dodge; et al. (FAccT, 2022)

Multitask Prompted Training Enables Zero-Shot Task Generalization. - Victor Sanh; Albert Webson; Colin Raffel; Stephen H. Bach; Lintang Sutawika; Zaid Alyafeai; Antoine Chaffin; Arnaud Stiegler; Arun Raja; Manan Dey; M Saiful Bari; Canwen Xu; et al. (ICLR, 2022)

BigBio: A Framework for Data-Centric Biomedical Natural Language Processing. - Jason A. Fries; Leon Weber; Natasha Seelam; Gabriel Altay; Debajyoti Datta; Samuele Garda; Sunny Kang; Rosaline Su; Wojciech Kusa; Samuel Cahyawijaya; Fabio Barth; Simon Ott; et al. (NeurIPS, 2022)

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. - Hugo Laurençon; Lucile Saulnier; Thomas Wang; Christopher Akiki; Albert Villanova del Moral; Teven Le Scao; Leandro von Werra; Chenghao Mou; Eduardo González Ponferrada; Huu Nguyen; Jörg Frohberg; Mario Sasko; et al. (NeurIPS, 2022)

Datasheet for the Pile. - Stella Biderman; Kieran Bicheno; Leo Gao (CoRR, 2022)

Neural Language Models are Effective Plagiarists. - Stella Biderman; Edward Raff (CoRR, 2022)

Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources. - Angelina McMillan-Major; Zaid Alyafeai; Stella Biderman; Kimbo Chen; Francesco De Toni; Gérard Dupont; Hady Elsahar; Chris Emezue; Alham Fikri Aji; Suzana Ilic; Nurulaqilla Khamis; Colin Leong; et al. (CoRR, 2022)

GPT-NeoX-20B: An Open-Source Autoregressive Language Model. - Sid Black; Stella Biderman; Eric Hallahan; Quentin Anthony; Leo Gao; Laurence Golding; Horace He; Connor Leahy; Kyle McDonell; Jason Phang; Michael Pieler; USVSN Sai Prashanth; et al. (CoRR, 2022)

Data Governance in the Age of Large-Scale Data-Driven Language Technology. - Yacine Jernite; Huu Nguyen; Stella Biderman; Anna Rogers; Maraim Masoud; Valentin Danchev; Samson Tan; Alexandra Sasha Luccioni; Nishant Subramani; Gérard Dupont; Jesse Dodge; Kyle Lo; et al. (CoRR, 2022)

BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing. - Jason Alan Fries; Leon Weber; Natasha Seelam; Gabriel Altay; Debajyoti Datta; Samuele Garda; Myungsun Kang; Ruisi Su; Wojciech Kusa; Samuel Cahyawijaya; Fabio Barth; Simon Ott; et al. (CoRR, 2022)

EleutherAI: Going Beyond "Open Science" to "Science in the Open". - Jason Phang; Herbie Bradley; Leo Gao; Louis Castricato; Stella Biderman (CoRR, 2022)

Large language models are not zero-shot communicators. - Laura Ruis; Akbir Khan; Stella Biderman; Sara Hooker; Tim Rocktäschel; Edward Grefenstette (CoRR, 2022)

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. - Teven Le Scao; Angela Fan; Christopher Akiki; Ellie Pavlick; Suzana Ilic; Daniel Hesslow; Roman Castagné; Alexandra Sasha Luccioni; François Yvon; Matthias Gallé; Jonathan Tow; Alexander M. Rush; et al. (CoRR, 2022)

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting. - Zheng Xin Yong; Hailey Schoelkopf; Niklas Muennighoff; Alham Fikri Aji; David Ifeoluwa Adelani; Khalid Almubarak; M. Saiful Bari; Lintang Sutawika; Jungo Kasai; Ahmed Baruwa; Genta Indra Winata; Stella Biderman; et al. (CoRR, 2022)

MP-NeRF: A massively parallel method for accelerating protein structure reconstruction from internal coordinates. - Eric Alcaide; Stella Biderman; Amalio Telenti; M. Cyrus Maher (J. Comput. Chem., 2022)

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. - Julia Kreutzer; Isaac Caswell; Lisa Wang; Ahsan Wahab; Daan van Esch; Nasanbayar Ulzii-Orshikh; Allahsera Tapo; Nishant Subramani; Artem Sokolov; Claytone Sikasote; Monang Setyawan; Supheakmungkol Sarin; et al. (Trans. Assoc. Comput. Linguistics, 2022)

Crosslingual Generalization through Multitask Finetuning. - Niklas Muennighoff; Thomas Wang; Lintang Sutawika; Adam Roberts; Stella Biderman; Teven Le Scao; M. Saiful Bari; Sheng Shen 001; Zheng Xin Yong; Hailey Schoelkopf; Xiangru Tang; Dragomir Radev; et al. (ACL (1), 2023)

GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration. - Aleksandra Piktus; Odunayo Ogundepo; Christopher Akiki; Akintunde Oladipo; Xinyu Zhang 018; Hailey Schoelkopf; Stella Biderman; Martin Potthast; Jimmy Lin (ACL (demo), 2023)

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting. - Zheng Xin Yong; Hailey Schoelkopf; Niklas Muennighoff; Alham Fikri Aji; David Ifeoluwa Adelani; Khalid Almubarak; M. Saiful Bari; Lintang Sutawika; Jungo Kasai; Ahmed Baruwa; Genta Indra Winata; Stella Biderman; et al. (ACL (1), 2023)

trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback. - Alexander Havrilla; Maksym Zhuravinskyi; Duy Phung; Aman Tiwari; Jonathan Tow; Stella Biderman; Quentin Anthony; Louis Castricato (EMNLP, 2023)

RWKV: Reinventing RNNs for the Transformer Era. - Bo Peng; Eric Alcaide; Quentin Anthony; Alon Albalak; Samuel Arcadinho; Stella Biderman; Huanqi Cao; Xin Cheng; Michael Chung; Leon Derczynski; Xingjian Du; Matteo Grella; et al. (EMNLP (Findings), 2023)

Recasting Self-Attention with Holographic Reduced Representations. - Mohammad Mahmudul Alam; Edward Raff; Stella Biderman; Tim Oates 001; James Holt (ICML, 2023)

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. - Stella Biderman; Hailey Schoelkopf; Quentin Gregory Anthony; Herbie Bradley; Kyle O'Brien; Eric Hallahan; Mohammad Aflah Khan; Shivanshu Purohit; USVSN Sai Prashanth; Edward Raff; Aviya Skowron; Lintang Sutawika; et al. (ICML, 2023)

LEACE: Perfect linear concept erasure in closed form. - Nora Belrose; David Schneider-Joseph; Shauli Ravfogel; Ryan Cotterell; Edward Raff; Stella Biderman (NeurIPS, 2023)

Emergent and Predictable Memorization in Large Language Models. - Stella Biderman; USVSN Sai Prashanth; Lintang Sutawika; Hailey Schoelkopf; Quentin Anthony; Shivanshu Purohit; Edward Raff (NeurIPS, 2023)

The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs. - Laura Ruis; Akbir Khan; Stella Biderman; Sara Hooker; Tim Rocktäschel; Edward Grefenstette (NeurIPS, 2023)

Eliciting Latent Predictions from Transformers with the Tuned Lens. - Nora Belrose; Zach Furman; Logan Smith; Danny Halawi; Igor Ostrovsky; Lev McKinney; Stella Biderman; Jacob Steinhardt (CoRR, 2023)

Can Transformers Learn to Solve Problems Recursively? - Shizhuo Dylan Zhang; Curt Tigges; Stella Biderman; Maxim Raginsky; Talia Ringer (CoRR, 2023)

Stay on topic with Classifier-Free Guidance. - Guillaume Sanchez; Honglu Fan; Alexander Spangher; Elad Levi; Pawan Sasanka Ammanamanchi; Stella Biderman (CoRR, 2023)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. - Aarohi Srivastava; Abhinav Rastogi; Abhishek Rao; Abu Awal Md Shoeb; Abubakar Abid; Adam Fisch; Adam R. Brown; Adam Santoro; Aditya Gupta; Adrià Garriga-Alonso; Agnieszka Kluska; Aitor Lewkowycz; et al. (Trans. Mach. Learn. Res., 2023)

Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection. - Mohammad Mahmudul Alam; Edward Raff; Stella Biderman; Tim Oates 001; James Holt (AISTATS, 2024)

Re-Evaluating Evaluation for Multilingual Summarization. - Jessica Forde; Ruochen Zhang; Lintang Sutawika; Alham Fikri Aji; Samuel Cahyawijaya; Genta Indra Winata; Minghao Wu; Carsten Eickhoff; Stella Biderman; Ellie Pavlick (EMNLP, 2024)

Llemma: An Open Language Model for Mathematics. - Zhangir Azerbayev; Hailey Schoelkopf; Keiran Paster; Marco Dos Santos; Stephen Marcus McAleer; Albert Q. Jiang; Jia Deng; Stella Biderman; Sean Welleck (ICLR, 2024)

Position: On the Societal Impact of Open Foundation Models. - Sayash Kapoor; Rishi Bommasani; Kevin Klyman; Shayne Longpre; Ashwin Ramaswami; Peter Cihon; Aspen K. Hopkins; Kevin Bankston; Stella Biderman; Miranda Bogen; Rumman Chowdhury; Alex Engler; et al. (ICML, 2024)

Stay on Topic with Classifier-Free Guidance. - Guillaume Sanchez; Alexander Spangher; Honglu Fan; Elad Levi; Stella Biderman (ICML, 2024)

Grokking Group Multiplication with Cosets. - Dashiell Stander; Qinan Yu; Honglu Fan; Stella Biderman (ICML, 2024)

The Case for Co-Designing Model Architectures with Hardware. - Quentin Anthony; Jacob Hatef; Deepak Narayanan; Stella Biderman; Stas Bekman; Junqi Yin; Aamir Shafi; Hari Subramoni; Dhabaleswar K. Panda 001 (ICPP, 2024)

A Walsh Hadamard Derived Linear Vector Symbolic Architecture. - Mohammad Mahmudul Alam; Alexander Oberle; Edward Raff; Stella Biderman; Tim Oates 001; James Holt (NeurIPS, 2024)

Consent in Crisis: The Rapid Decline of the AI Data Commons. - Shayne Longpre; Robert Mahari; Ariel Lee; Campbell Lund; Hamidah Oderinwale; William Brannon; Nayan Saxena; Naana Obeng-Marnu; Tobin South; Cole Hunter; Kevin Klyman; Christopher Klamm; et al. (NeurIPS, 2024)

LLM Circuit Analyses Are Consistent Across Training and Scale. - Curt Tigges; Michael Hanna 001; Qinan Yu; Stella Biderman (NeurIPS, 2024)

Suppressing Pink Elephants with Direct Principle Feedback. - Louis Castricato; Nathan Lile; Suraj Anand; Hailey Schoelkopf; Siddharth Verma; Stella Biderman (CoRR, 2024)

On the Societal Impact of Open Foundation Models. - Sayash Kapoor; Rishi Bommasani; Kevin Klyman; Shayne Longpre; Ashwin Ramaswami; Peter Cihon; Aspen K. Hopkins; Kevin Bankston; Stella Biderman; Miranda Bogen; Rumman Chowdhury; Alex Engler; et al. (CoRR, 2024)

Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence. - Bo Peng; Daniel Goldstein; Quentin Anthony; Alon Albalak; Eric Alcaide; Stella Biderman; Eugene Cheah; Xingjian Du; Teddy Ferdinan; Haowen Hou; Przemyslaw Kazienko; Kranthi Kiran GV; et al. (CoRR, 2024)

Lessons from the Trenches on Reproducible Evaluation of Language Models. - Stella Biderman; Hailey Schoelkopf; Lintang Sutawika; Leo Gao; Jonathan Tow; Baber Abbasi; Alham Fikri Aji; Pawan Sasanka Ammanamanchi; Sidney Black; Jordan Clive; Anthony DiPofi; Julen Etxaniz; et al. (CoRR, 2024)

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? - Rylan Schaeffer; Hailey Schoelkopf; Brando Miranda; Gabriel Mukobi; Varun Madan; Adam Ibrahim; Herbie Bradley; Stella Biderman; Sanmi Koyejo (CoRR, 2024)

Consent in Crisis: The Rapid Decline of the AI Data Commons. - Shayne Longpre; Robert Mahari; Ariel Lee; Campbell Lund; Hamidah Oderinwale; William Brannon; Nayan Saxena; Naana Obeng-Marnu; Tobin South; Cole Hunter; Kevin Klyman; Christopher Klamm; et al. (CoRR, 2024)

Bridging the Data Provenance Gap Across Text, Speech and Video. - Shayne Longpre; Nikhil Singh 003; Manuel Cherep; Kushagra Tiwary; Joanna Materzynska; William Brannon; Robert Mahari; Manan Dey; Mohammed Hamdy; Nayan Saxena; Ahmad Mustafa Anis; Emad A. Alghamdi; et al. (CoRR, 2024)

The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources. - Shayne Longpre; Stella Biderman; Alon Albalak; Hailey Schoelkopf; Daniel McDuff; Sayash Kapoor; Kevin Klyman; Kyle Lo; Gabriel Ilharco; Nay San; Maribeth Rauh; Aviya Skowron; et al. (Trans. Mach. Learn. Res., 2024)

Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion. - Shizhuo Dylan Zhang; Curt Tigges; Zory Zhang; Stella Biderman; Maxim Raginsky; Talia Ringer (Trans. Mach. Learn. Res., 2024)

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon. - USVSN Sai Prashanth; Alvin Deng; Kyle O'Brien; Jyothir S. V; Mohammad Aflah Khan; Jaydeep Borkar; Christopher A. Choquette-Choo; Jacob Ray Fuehne; Stella Biderman; Tracy Ke; Katherine Lee; Naomi Saphra (ICLR, 2025)

PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs. - Oskar van der Wal; Pietro Lesci; Max Müller-Eberstein; Naomi Saphra; Hailey Schoelkopf; Willem H. Zuidema; Stella Biderman (ICLR, 2025)

KMMLU: Measuring Massive Multitask Language Understanding in Korean. - Guijin Son; Hanwool Lee; Sungdong Kim; Seungone Kim; Niklas Muennighoff; Taekyoon Choi; Cheonbok Park; Kang Min Yoo; Stella Biderman (NAACL (Long Papers), 2025)

Towards Best Practices for Open Datasets for LLM Training. - Stefan Baack; Stella Biderman; Kasia Odrozek; Aviya Skowron; Ayah Bdeir; Jillian Bommarito; Jennifer Ding; Maximilian Gahntz; Paul Keller; Pierre-Carl Langlais; Greg Lindahl; Sebastian Majstorovic; et al. (CoRR, 2025)

Open Problems in Mechanistic Interpretability. - Lee Sharkey; Bilal Chughtai; Joshua Batson; Jack Lindsey; Jeff Wu 003; Lucius Bushnaq; Nicholas Goldowsky-Dill; Stefan Heimersheim; Alejandro Ortega; Joseph Isaac Bloom; Stella Biderman; Adrià Garriga-Alonso; et al. (CoRR, 2025)

When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research. - Guijin Son; Jiwoo Hong; Honglu Fan; Heejeong Nam; Hyunwoo Ko; Seungwon Lim; Jinyeop Song; Jinha Choi; Gonçalo Paulo; Youngjae Yu; Stella Biderman (CoRR, 2025)

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text. - Nikhil Kandpal; Brian Lester; Colin Raffel; Sebastian Majstorovic; Stella Biderman; Baber Abbasi; Luca Soldaini; Enrico Shippole; A. Feder Cooper; Aviya Skowron; John Kirchenbauer; Shayne Longpre; et al. (CoRR, 2025)

Scaling Self-Supervised Representation Learning for Symbolic Piano Performance. - Louis Bradshaw; Honglu Fan; Alexander Spangher; Stella Biderman; Simon Colton (CoRR, 2025)

Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. - Kyle O'Brien; Stephen Casper; Quentin Anthony; Tomek Korbak; Robert Kirk; Xander Davies; Ishan Mishra; Geoffrey Irving; Yarin Gal; Stella Biderman (CoRR, 2025)