Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. - Richard Socher; Jeffrey Pennington; Eric H. Huang; Andrew Y. Ng; Christopher D. Manning (EMNLP, 2011)

Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. - Richard Socher; Eric H. Huang; Jeffrey Pennington; Andrew Y. Ng; Christopher D. Manning (NIPS, 2011)

Glove: Global Vectors for Word Representation. - Jeffrey Pennington; Richard Socher; Christopher D. Manning (EMNLP, 2014)

Spherical Random Features for Polynomial Kernels. - Jeffrey Pennington; Felix X. Yu; Sanjiv Kumar (NIPS, 2015)

Clinical Data Research Network Lessons Learned. - Kevin Murphy; Jeffrey Pennington; Aaron N. Browne; Byron Ruth; Ritu Khare; LeMar Davidson; Patricia K. Morris; Levon H. Utidjian; Charles Bailey 001 (CRI, 2016)

Geometry of Neural Network Loss Surfaces via Random Matrix Theory. - Jeffrey Pennington; Yasaman Bahri (ICML, 2017)

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. - Jeffrey Pennington; Samuel S. Schoenholz; Surya Ganguli (NIPS, 2017)

Nonlinear random matrix theory for deep learning. - Jeffrey Pennington; Pratik Worah (NIPS, 2017)

A Correspondence Between Random Neural Networks and Statistical Field Theory. - Samuel S. Schoenholz; Jeffrey Pennington; Jascha Sohl-Dickstein (CoRR, 2017)

The emergence of spectral universality in deep networks. - Jeffrey Pennington; Samuel S. Schoenholz; Surya Ganguli (AISTATS, 2018)

Deep Neural Networks as Gaussian Processes. - Jaehoon Lee 001; Yasaman Bahri; Roman Novak; Samuel S. Schoenholz; Jeffrey Pennington; Jascha Sohl-Dickstein (ICLR (Poster), 2018)

Sensitivity and Generalization in Neural Networks: an Empirical Study. - Roman Novak; Yasaman Bahri; Daniel A. Abolafia; Jeffrey Pennington; Jascha Sohl-Dickstein (ICLR (Poster), 2018)

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. - Minmin Chen; Jeffrey Pennington; Samuel S. Schoenholz (ICML, 2018)

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. - Lechao Xiao; Yasaman Bahri; Jascha Sohl-Dickstein; Samuel S. Schoenholz; Jeffrey Pennington (ICML, 2018)

The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network. - Jeffrey Pennington; Pratik Worah (NeurIPS, 2018)

Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes. - Roman Novak; Lechao Xiao; Jaehoon Lee 001; Yasaman Bahri; Daniel A. Abolafia; Jeffrey Pennington; Jascha Sohl-Dickstein (CoRR, 2018)

KAMA-NNs: Low-dimensional Rotation Based Neural Networks. - Krzysztof Choromanski; Aldo Pacchiano; Jeffrey Pennington; Yunhao Tang (AISTATS, 2019)

Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes. - Roman Novak; Lechao Xiao; Yasaman Bahri; Jaehoon Lee 001; Greg Yang; Jiri Hron; Daniel A. Abolafia; Jeffrey Pennington; Jascha Sohl-Dickstein (ICLR (Poster), 2019)

A Mean Field Theory of Batch Normalization. - Greg Yang; Jeffrey Pennington; Vinay Rao; Jascha Sohl-Dickstein; Samuel S. Schoenholz (ICLR (Poster), 2019)

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. - Jaehoon Lee 001; Lechao Xiao; Samuel S. Schoenholz; Yasaman Bahri; Roman Novak; Jascha Sohl-Dickstein; Jeffrey Pennington (NeurIPS, 2019)

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs. - Dar Gilboa; Bo Chang 002; Minmin Chen; Greg Yang; Samuel S. Schoenholz; Ed H. Chi; Jeffrey Pennington (CoRR, 2019)

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. - Jaehoon Lee 001; Lechao Xiao; Samuel S. Schoenholz; Yasaman Bahri; Jascha Sohl-Dickstein; Jeffrey Pennington (CoRR, 2019)

A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning. - Ben Adlam; Jake Levinson; Jeffrey Pennington (CoRR, 2019)

Disentangling trainability and generalization in deep learning. - Lechao Xiao; Jeffrey Pennington; Samuel S. Schoenholz (CoRR, 2019)

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. - Wei Hu; Lechao Xiao; Jeffrey Pennington (ICLR, 2020)

The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. - Ben Adlam; Jeffrey Pennington (ICML, 2020)

Disentangling Trainability and Generalization in Deep Neural Networks. - Lechao Xiao; Jeffrey Pennington; Samuel Stern Schoenholz (ICML, 2020)

Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition. - Ben Adlam; Jeffrey Pennington (NeurIPS, 2020)

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. - Wei Hu; Lechao Xiao; Ben Adlam; Jeffrey Pennington (NeurIPS, 2020)

Finite Versus Infinite Neural Networks: an Empirical Study. - Jaehoon Lee 001; Samuel S. Schoenholz; Jeffrey Pennington; Ben Adlam; Lechao Xiao; Roman Novak; Jascha Sohl-Dickstein (NeurIPS, 2020)

Exact posterior distributions of wide Bayesian neural networks. - Jiri Hron; Yasaman Bahri; Roman Novak; Jeffrey Pennington; Jascha Sohl-Dickstein (CoRR, 2020)

Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit. - Ben Adlam; Jaehoon Lee 001; Lechao Xiao; Jeffrey Pennington; Jasper Snoek (ICLR, 2021)

Overparameterization Improves Robustness to Covariate Shift in High Dimensions. - Nilesh Tripuraneni; Ben Adlam; Jeffrey Pennington (NeurIPS, 2021)

Covariate Shift in High-Dimensional Random Feature Regression. - Nilesh Tripuraneni; Ben Adlam; Jeffrey Pennington (CoRR, 2021)

A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions. - Ben Adlam; Jake A. Levinson; Jeffrey Pennington (AISTATS, 2022)

Online education for data science: Opportunities and challenges. - Jeffrey Pennington; Rose Hartman; Ashwini Davison; Ali Shokoufandeh; Joy Payton; Daniel Chen; André Dietrich (AMIA, 2022)

Anisotropic Random Feature Regression in High Dimensions. - Gabriel Mel; Jeffrey Pennington (ICLR, 2022)

Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling. - Jiri Hron; Roman Novak; Jeffrey Pennington; Jascha Sohl-Dickstein (ICML, 2022)

Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm. - Lechao Xiao; Jeffrey Pennington (ICML, 2022)

Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions. - Courtney Paquette; Elliot Paquette; Ben Adlam; Jeffrey Pennington (NeurIPS, 2022)

Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression. - Lechao Xiao; Hong Hu; Theodor Misiakiewicz; Yue Lu; Jeffrey Pennington (NeurIPS, 2022)

Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression. - Lechao Xiao; Jeffrey Pennington (CoRR, 2022)

Second-order regression models exhibit progressive sharpening to the edge of stability. - Atish Agarwala; Fabian Pedregosa; Jeffrey Pennington (ICML, 2023)

Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5? - C. Daniel Freeman; Laura Culp; Aaron Parisi; Maxwell L. Bileschi; Gamaleldin F. Elsayed; Alex Rizkowsky; Isabelle Simpson; Alex Alemi; Azade Nova; Ben Adlam; Bernd Bohnet; Gaurav Mishra; et al. (CoRR, 2023)

Temperature check: theory and practice for training models with softmax-cross-entropy losses. - Atish Agarwala; Samuel Stern Schoenholz; Jeffrey Pennington; Yann N. Dauphin (Trans. Mach. Learn. Res., 2023)

Small-scale proxies for large-scale Transformer training instabilities. - Mitchell Wortsman; Peter J. Liu; Lechao Xiao; Katie E. Everett; Alexander A. Alemi; Ben Adlam; John D. Co-Reyes; Izzeddin Gur; Abhishek Kumar; Roman Novak; Jeffrey Pennington; Jascha Sohl-Dickstein; et al. (ICLR, 2024)

Scaling Exponents Across Parameterizations and Optimizers. - Katie E. Everett; Lechao Xiao; Mitchell Wortsman; Alexander A. Alemi; Roman Novak; Peter J. Liu; Izzeddin Gur; Jascha Sohl-Dickstein; Leslie Pack Kaelbling; Jaehoon Lee 001; Jeffrey Pennington (ICML, 2024)

4+3 Phases of Compute-Optimal Neural Scaling Laws. - Elliot Paquette; Courtney Paquette; Lechao Xiao; Jeffrey Pennington (NeurIPS, 2024)

High dimensional analysis reveals conservative sharpening and a stochastic edge of stability. - Atish Agarwala; Jeffrey Pennington (CoRR, 2024)

Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability. - Jiri Hron; Laura Culp; Gamaleldin F. Elsayed; Rosanne Liu; Ben Adlam; Maxwell L. Bileschi; Bernd Bohnet; JD Co-Reyes; Noah Fiedel; C. Daniel Freeman; Izzeddin Gur; Kathleen Kenealy; et al. (CoRR, 2024)

Training LLMs over Neurally Compressed Text. - Brian Lester; Jaehoon Lee 001; Alexander A. Alemi; Jeffrey Pennington; Adam Roberts; Jascha Sohl-Dickstein; Noah Constant (Trans. Mach. Learn. Res., 2024)

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. - Avi Singh; John D. Co-Reyes; Rishabh Agarwal; Ankesh Anand; Piyush Patil; Xavier Garcia; Peter J. Liu; James Harrison; Jaehoon Lee 001; Kelvin Xu; Aaron T. Parisi; Abhishek Kumar; et al. (Trans. Mach. Learn. Res., 2024)

Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks. - Shikai Qiu; Lechao Xiao; Andrew Gordon Wilson; Jeffrey Pennington; Atish Agarwala (CoRR, 2025)