SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient. - Aaron Mishkin; Frederik Kunstner; Didrik Nielsen; Mark Schmidt 001; Mohammad Emtiyaz Khan (NeurIPS, 2018)

Limitations of the empirical Fisher approximation for natural gradient descent. - Frederik Kunstner; Philipp Hennig; Lukas Balles (NeurIPS, 2019)

Limitations of the Empirical Fisher Approximation. - Frederik Kunstner; Lukas Balles; Philipp Hennig (CoRR, 2019)

BackPACK: Packing more into Backprop. - Felix Dangel; Frederik Kunstner; Philipp Hennig (ICLR, 2020)

Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search). - Sharan Vaswani; Frederik Kunstner; Issam H. Laradji; Si Yi Meng; Mark Schmidt 001; Simon Lacoste-Julien (CoRR, 2020)

Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent. - Frederik Kunstner; Raunak Kumar; Mark Schmidt 001 (AISTATS, 2021)

Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent - an Open Problem. - Rémi Le Priol; Frederik Kunstner; Damien Scieur; Simon Lacoste-Julien (CoRR, 2021)

Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent (Extended Abstract). - Frederik Kunstner; Raunak Kumar; Mark Schmidt 001 (IJCAI, 2022)

Noise Is Not the Main Factor Behind the Gap Between Sgd and Adam on Transformers, But Sign Descent Might Be. - Frederik Kunstner; Jacques Chen; Jonathan Wilder Lavington; Mark Schmidt 001 (ICLR, 2023)

Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking. - Frederik Kunstner; Victor Sanches Portella; Mark Schmidt 001; Nicholas J. A. Harvey (NeurIPS, 2023)

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models. - Frederik Kunstner; Alan Milligan; Robin Yadav; Mark Schmidt 001; Alberto Bietti (NeurIPS, 2024)

Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law. - Frederik Kunstner; Francis Bach 001 (CoRR, 2025)