POS-tagging of Historical Dutch. - Dieuwke Hupkes; Rens Bod (LREC, 2016)

Diagnostic Classifiers Revealing how Neural Networks Process Hierarchical Structure. - Sara Veldhoen; Dieuwke Hupkes; Willem H. Zuidema (CoCo@NIPS, 2016)

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information. - Mario Giulianelli; Jack Harding; Florian Mohnert; Dieuwke Hupkes; Willem H. Zuidema (BlackboxNLP@EMNLP, 2018)

Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue. - Dieuwke Hupkes; Sanne Bouwmeester; Raquel Fernández (BlackboxNLP@EMNLP, 2018)

Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items. - Jaap Jumelet; Dieuwke Hupkes (BlackboxNLP@EMNLP, 2018)

Visualisation and 'Diagnostic Classifiers' Reveal how Recurrent and Recursive Neural Networks Process Hierarchical Structure (Extended Abstract). - Dieuwke Hupkes; Willem H. Zuidema (IJCAI, 2018)

Learning compositionally through attentive guidance. - Dieuwke Hupkes; Anand Singh; Kris Korrel; Germán Kruszewski; Elia Bruni (CoRR, 2018)

Visualisation and 'Diagnostic Classifiers' Reveal How Recurrent and Recursive Neural Networks Process Hierarchical Structure. - Dieuwke Hupkes; Sara Veldhoen; Willem H. Zuidema (J. Artif. Intell. Res., 2018)

On the Realization of Compositionality in Neural Networks. - Joris Baan; Jana Leible; Mitja Nikolaus; David Rau; Dennis Ulmer; Tim Baumgärtner; Dieuwke Hupkes; Elia Bruni (BlackboxNLP@ACL, 2019)

Transcoding Compositionally: Using Attention to Find More Generalizable Solutions. - Kris Korrel; Dieuwke Hupkes; Verna Dankers; Elia Bruni (BlackboxNLP@ACL, 2019)

Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment. - Jaap Jumelet; Willem H. Zuidema; Dieuwke Hupkes (CoNLL, 2019)

The Fast and the Flexible: Training Neural Networks to Learn to Follow Instructions from Small Data. - Rezka Leonandya; Dieuwke Hupkes; Elia Bruni; Germán Kruszewski (IWCS (1), 2019)

The emergence of number and syntax units in LSTM language models. - Yair Lakretz; Germán Kruszewski; Theo Desbordes; Dieuwke Hupkes; Stanislas Dehaene; Marco Baroni (NAACL-HLT (1), 2019)

Assessing Incrementality in Sequence-to-Sequence Models. - Dennis Ulmer; Dieuwke Hupkes; Elia Bruni (RepL4NLP@ACL, 2019)

Formal models of Structure Building in Music, Language and Animal Songs. - Willem H. Zuidema; Dieuwke Hupkes; Geraint A. Wiggins; Constance Scharff; Martin Rohrmeier (CoRR, 2019)

The compositionality of neural networks: integrating symbolism and connectionism. - Dieuwke Hupkes; Verna Dankers; Mathijs Mul; Elia Bruni (CoRR, 2019)

Location Attention for Extrapolation to Longer Sequences. - Yann Dubois; Gautier Dagan; Dieuwke Hupkes; Elia Bruni (ACL, 2020)

Internal and External Pressures on Language Emergence: Least Effort, Object Constancy and Frequency. - Diana Rodríguez Luna; Edoardo Maria Ponti; Dieuwke Hupkes; Elia Bruni (EMNLP (Findings), 2020)

The Grammar of Emergent Languages. - Oskar van der Wal; Silvan de Boer; Elia Bruni; Dieuwke Hupkes (EMNLP (1), 2020)

Compositionality Decomposed: How do Neural Networks Generalise? (Extended Abstract). - Dieuwke Hupkes; Verna Dankers; Mathijs Mul; Elia Bruni (IJCAI, 2020)

Exploring Processing of Nested Dependencies in Neural-Network Language Models and Humans. - Yair Lakretz; Dieuwke Hupkes; Alessandra Vergallito; Marco Marelli; Marco Baroni; Stanislas Dehaene (CoRR, 2020)

Compositionality Decomposed: How do Neural Networks Generalise? - Dieuwke Hupkes; Verna Dankers; Mathijs Mul; Elia Bruni (J. Artif. Intell. Res., 2020)

Attention vs non-attention for a Shapley-based explanation method. - Tom Kersten; Hugh Mee Wong; Jaap Jumelet; Dieuwke Hupkes (DeeLIO@NAACL-HLT, 2021)

Language Models Use Monotonicity to Assess NPI Licensing. - Jaap Jumelet; Milica Denic; Jakub Szymanik; Dieuwke Hupkes; Shane Steinert-Threlkeld (ACL/IJCNLP (Findings), 2021)

Generalising to German Plural Noun Classes, from the Perspective of a Recurrent Neural Network. - Verna Dankers; Anna Langedijk; Kate McCurdy; Adina Williams; Dieuwke Hupkes (CoNLL, 2021)

Co-evolution of language and agents in referential games. - Gautier Dagan; Dieuwke Hupkes; Elia Bruni (EACL, 2021)

Language Modelling as a Multi-Task Problem. - Lucas Weber; Jaap Jumelet; Elia Bruni; Dieuwke Hupkes (EACL, 2021)

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little. - Koustuv Sinha; Robin Jia; Dieuwke Hupkes; Joelle Pineau; Adina Williams; Douwe Kiela (EMNLP (1), 2021)

How BPE Affects Memorization in Transformers. - Eugene Kharitonov; Marco Baroni; Dieuwke Hupkes (CoRR, 2021)

Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans. - Yair Lakretz; Theo Desbordes; Dieuwke Hupkes; Stanislas Dehaene (CoRR, 2021)

Towards Interactive Language Modeling. - Maartje ter Hoeve; Evgeny Kharitonov; Dieuwke Hupkes; Emmanuel Dupoux (CoRR, 2021)

The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study. - Verna Dankers; Elia Bruni; Dieuwke Hupkes (ACL (1), 2022)

Sparse Interventions in Language Models with Differentiable Masking. - Nicola De Cao; Leon Schmid; Dieuwke Hupkes; Ivan Titov 001 (BlackboxNLP@EMNLP, 2022)

Evaluating locality in NMT models. - Itay Itzhak; Koustuv Sinha; Brenden M. Lake; Adina Williams; Dieuwke Hupkes (CogSci, 2022)

Can Transformers Process Recursive Nested Constructions, Like Humans? - Yair Lakretz; Théo Desbordes; Dieuwke Hupkes; Stanislas Dehaene (COLING, 2022)

The Curious Case of Absolute Position Embeddings. - Koustuv Sinha; Amirhossein Kazemnejad; Siva Reddy; Joelle Pineau; Dieuwke Hupkes; Adina Williams (EMNLP (Findings), 2022)

Text Characterization Toolkit (TCT). - Daniel Simig; Tianlu Wang; Verna Dankers; Peter Henderson 002; Khuyagbaatar Batsuren; Dieuwke Hupkes; Mona T. Diab (AACL/IJCNLP (System Demonstrations), 2022)

Text Characterization Toolkit. - Daniel Simig; Tianlu Wang; Verna Dankers; Peter Henderson 002; Khuyagbaatar Batsuren; Dieuwke Hupkes; Mona T. Diab (CoRR, 2022)

State-of-the-art generalisation research in NLP: a taxonomy and review. - Dieuwke Hupkes; Mario Giulianelli; Verna Dankers; Mikel Artetxe; Yanai Elazar; Tiago Pimentel; Christos Christodoulopoulos 001; Karim Lasri; Naomi Saphra; Arabella Sinclair; Dennis Ulmer; Florian Schottmann; et al. (CoRR, 2022)

The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks. - Kaiser Sun; Adina Williams; Dieuwke Hupkes (CoNLL, 2023)

Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning. - Lucas Weber; Elia Bruni; Dieuwke Hupkes (CoNLL, 2023)

Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation. - Verna Dankers; Ivan Titov 001; Dieuwke Hupkes (EMNLP, 2023)

Neural Agents Struggle to Take Turns in Bidirectional Emergent Communication. - Valentin Taillandier; Dieuwke Hupkes; Benoît Sagot; Emmanuel Dupoux; Paul Michel (ICLR, 2023)

Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses. - Xenia Ohmer; Elia Bruni; Dieuwke Hupkes (CoRR, 2023)

Curriculum Learning with Adam: The Devil Is in the Wrong Details. - Lucas Weber; Jaap Jumelet; Paul Michel; Elia Bruni; Dieuwke Hupkes (CoRR, 2023)

WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models. - Youssef Benchekroun; Megi Dervishi; Mark Ibrahim; Jean-Baptiste Gaya; Xavier Martinet; Grégoire Mialon; Thomas Scialom; Emmanuel Dupoux; Dieuwke Hupkes; Pascal Vincent (CoRR, 2023)

The ICL Consistency Test. - Lucas Weber; Elia Bruni; Dieuwke Hupkes (CoRR, 2023)

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. - Aarohi Srivastava; Abhinav Rastogi; Abhishek Rao; Abu Awal Md Shoeb; Abubakar Abid; Adam Fisch; Adam R. Brown; Adam Santoro; Aditya Gupta; Adrià Garriga-Alonso; Agnieszka Kluska; Aitor Lewkowycz; et al. (Trans. Mach. Learn. Res., 2023)

A taxonomy and review of generalization research in NLP. - Dieuwke Hupkes; Mario Giulianelli; Verna Dankers; Mikel Artetxe; Yanai Elazar; Tiago Pimentel; Christos E. Christodoulopoulos; Karim Lasri; Naomi Saphra; Arabella Sinclair; Dennis Ulmer; Florian Schottmann; et al. (Nat. Mac. Intell., 2023)

Interpretability of Language Models via Task Spaces. - Lucas Weber; Jaap Jumelet; Elia Bruni; Dieuwke Hupkes (ACL (1), 2024)

From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency. - Xenia Ohmer; Elia Bruni; Dieuwke Hupkes (Comput. Linguistics, 2024)

Quantifying Variance in Evaluation Benchmarks. - Lovish Madaan; Aaditya K. Singh; Rylan Schaeffer; Andrew Poulton; Sanmi Koyejo; Pontus Stenetorp; Sharan Narang; Dieuwke Hupkes (CoRR, 2024)

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. - Aman Singh Thakur; Kartik Choudhary; Venkat Srinik Ramayapally; Sankaran Vaidyanathan; Dieuwke Hupkes (CoRR, 2024)

The Llama 3 Herd of Models. - Abhimanyu Dubey; Abhinav Jauhri; Abhinav Pandey; Abhishek Kadian; Ahmad Al-Dahle; Aiesha Letman; Akhil Mathur; Alan Schelten; Amy Yang; Angela Fan; Anirudh Goyal; Anthony Hartshorn; et al. (CoRR, 2024)

Evaluation data contamination in LLMs: how do we measure it and (when) does it matter? - Aaditya K. Singh; Muhammed Yusuf Kocyigit; Andrew Poulton; David Esiobu; Maria Lomeli; Gergely Szilvasy; Dieuwke Hupkes (CoRR, 2024)

Compute Optimal Scaling of Skills: Knowledge vs Reasoning. - Nicholas Roberts; Niladri S. Chatterji; Sharan Narang; Mike Lewis; Dieuwke Hupkes (ACL (Findings), 2025)

Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models. - Lovish Madaan; David Esiobu; Pontus Stenetorp; Barbara Plank; Dieuwke Hupkes (NAACL (Long Papers), 2025)

MLGym: A New Framework and Benchmark for Advancing AI Research Agents. - Deepak Nathani; Lovish Madaan; Nicholas Roberts; Nikolay Bashlykov; Ajay Menon; Vincent Moens; Amar Budhiraja; Despoina Magka; Vladislav Vorotilov; Gaurav Chaurasia; Dieuwke Hupkes; Ricardo Silveira Cabral; et al. (CoRR, 2025)

Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks. - Rylan Schaeffer; Punit Singh Koura; Binh Tang; Ranjan Subramanian; Aaditya K. Singh; Todor Mihaylov; Prajjwal Bhargava; Lovish Madaan; Niladri S. Chatterji; Vedanuj Goswami; Sergey Edunov; Dieuwke Hupkes; et al. (CoRR, 2025)

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages. - Dieuwke Hupkes; Nikolay Bogoychev (CoRR, 2025)