Random initialisations performing above chance and how to find them. - Frederik Benzing; Simon Schug; Robert Meier; Johannes von Oswald; Yassir Akram; Nicolas Zucchet; Laurence Aitchison; Angelika Steger (CoRR, 2022)

Gated recurrent neural networks discover attention. - Nicolas Zucchet; Seijin Kobayashi; Yassir Akram; Johannes von Oswald; Maxime Larcher; Angelika Steger; João Sacramento (CoRR, 2023)

Discovering modular solutions that generalize compositionally. - Simon Schug; Seijin Kobayashi; Yassir Akram; Maciej Wolczyk; Alexandra Proca; Johannes von Oswald; Razvan Pascanu; João Sacramento; Angelika Steger (ICLR, 2024)

Weight decay induces low-rank attention layers. - Seijin Kobayashi; Yassir Akram; Johannes von Oswald (NeurIPS, 2024)

When can transformers compositionally generalize in-context? - Seijin Kobayashi; Simon Schug; Yassir Akram; Florian Redhardt; Johannes von Oswald; Razvan Pascanu; Guillaume Lajoie; João Sacramento (CoRR, 2024)

Learning Randomized Algorithms with Transformers. - Johannes von Oswald; Seijin Kobayashi; Yassir Akram; Angelika Steger (ICLR, 2025)

Attention as a Hypernetwork. - Simon Schug; Seijin Kobayashi; Yassir Akram; João Sacramento; Razvan Pascanu (ICLR, 2025)

Scale leads to compositional generalization. - Florian Redhardt; Yassir Akram; Simon Schug (CoRR, 2025)