The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only. - Guilherme Penedo; Quentin Malartic; Daniel Hesslow; Ruxandra Cojocaru; Hamza Alobeidli; Alessandro Cappelli; Baptiste Pannier; Ebtesam Almazrouei; Julien Launay (NeurIPS, 2023)

AlGhafa Evaluation Benchmark for Arabic Language Models. - Ebtesam Almazrouei; Ruxandra Cojocaru; Michele Baldo; Quentin Malartic; Hamza Alobeidli; Daniele Mazzotta; Guilherme Penedo; Giulia Campesan; Mugariya Farooq; Maitha Alhammadi; Julien Launay; Badreddine Noune (ArabicNLP, 2023)

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. - Guilherme Penedo; Quentin Malartic; Daniel Hesslow; Ruxandra Cojocaru; Alessandro Cappelli; Hamza Alobeidli; Baptiste Pannier; Ebtesam Almazrouei; Julien Launay (CoRR, 2023)

The Falcon Series of Open Language Models. - Ebtesam Almazrouei; Hamza Alobeidli; Abdulaziz Alshamsi; Alessandro Cappelli; Ruxandra Cojocaru; Mérouane Debbah; Étienne Goffinet; Daniel Hesslow; Julien Launay; Quentin Malartic; Daniele Mazzotta; Badreddine Noune; et al. (CoRR, 2023)

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. - Guilherme Penedo; Hynek Kydlícek; Loubna Ben Allal; Anton Lozhkov; Margaret Mitchell; Colin A. Raffel; Leandro von Werra; Thomas Wolf 008 (NeurIPS, 2024)

Towards Best Practices for Open Datasets for LLM Training. - Stefan Baack; Stella Biderman; Kasia Odrozek; Aviya Skowron; Ayah Bdeir; Jillian Bommarito; Jennifer Ding; Maximilian Gahntz; Paul Keller; Pierre-Carl Langlais; Greg Lindahl; Sebastian Majstorovic; et al. (CoRR, 2025)

SmolLM2: When Smol Goes Big - Data-Centric Training of a Small Language Model. - Loubna Ben Allal; Anton Lozhkov; Elie Bakouch; Gabriel Martín Blázquez; Guilherme Penedo; Lewis Tunstall; Andrés Marafioti; Hynek Kydlícek; Agustín Piqueres Lajarín; Vaibhav Srivastav; Joshua Lochner; Caleb Fahlgren; et al. (CoRR, 2025)

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text. - Nikhil Kandpal; Brian Lester; Colin Raffel; Sebastian Majstorovic; Stella Biderman; Baber Abbasi; Luca Soldaini; Enrico Shippole; A. Feder Cooper; Aviya Skowron; John Kirchenbauer; Shayne Longpre; et al. (CoRR, 2025)

FineWeb2: One Pipeline to Scale Them All - Adapting Pre-Training Data Processing to Every Language. - Guilherme Penedo; Hynek Kydlícek; Vinko Sabolcec; Bettina Messmer; Negar Foroutan; Amir Hossein Kargaran; Colin Raffel; Martin Jaggi; Leandro von Werra; Thomas Wolf 008 (CoRR, 2025)