Etai Littwin

According to our database, Etai Littwin authored at least 29 papers between 2015 and 2024.

Collaborative distances:

• Dijkstra number of four.
• Erdős number of three.

2024

The Slingshot Effect: A Late-Stage Optimization Anomaly in Adaptive Gradient Methods.

Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Joshua M. Susskind

Trans. Mach. Learn. Res., 2024

How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks.

Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang, Joshua M. Susskind

CoRR, 2024

What Algorithms can Transformers Learn? A Study in Length Generalization.

Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Joshua M. Susskind, Samy Bengio, Preetum Nakkiran

Proceedings of the Twelfth International Conference on Learning Representations, 2024

LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures.

Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, Etai Littwin

Proceedings of the Twelfth International Conference on Learning Representations, 2024

Vanishing Gradients in Reinforcement Finetuning of Language Models.

Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, Etai Littwin

Proceedings of the Twelfth International Conference on Learning Representations, 2024

When can transformers reason with abstract symbols?

Enric Boix-Adserà, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, Joshua M. Susskind

Proceedings of the Twelfth International Conference on Learning Representations, 2024

2023

Tight conditions for when the NTK approximation is valid.

Enric Boix-Adserà, Etai Littwin

Trans. Mach. Learn. Res., 2023

Adaptivity and Modularity for Efficient Generalization Over Task Complexity.

Samira Abnar, Omid Saremi, Laurent Dinh, Shantel Wilson, Miguel Ángel Bautista, Chen Huang, Vimal Thilak, Etai Littwin, Jiatao Gu, Josh M. Susskind, Samy Bengio

CoRR, 2023

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit.

Greg Yang, Etai Littwin

CoRR, 2023

The NTK approximation is valid for longer than you think.

Enric Boix-Adserà, Etai Littwin

CoRR, 2023

Transformers learn through gradual rank increase.

Emmanuel Abbe, Samy Bengio, Enric Boix-Adserà, Etai Littwin, Joshua M. Susskind

Proceedings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023

Stabilizing Transformer Training by Preventing Attention Entropy Collapse.

Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, Joshua M. Susskind

Proceedings of the International Conference on Machine Learning, 2023

Adaptive Optimization in the ∞-Width Limit.

Etai Littwin, Greg Yang

Proceedings of the Eleventh International Conference on Learning Representations, 2023

2022

The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon.

Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, Joshua M. Susskind

CoRR, 2022

Learning Representation from Neural Fisher Kernel with Low-rank Approximation.

Ruixiang Zhang, Shuangfei Zhai, Etai Littwin, Joshua M. Susskind

Proceedings of the Tenth International Conference on Learning Representations, 2022

2021

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks.

Shih-Yu Sun, Vimal Thilak, Etai Littwin, Omid Saremi, Joshua M. Susskind

CoRR, 2021

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks.

Etai Littwin, Omid Saremi, Shuangfei Zhai, Vimal Thilak, Hanlin Goh, Joshua M. Susskind, Greg Yang

CoRR, 2021

On random kernels of residual architectures.

Etai Littwin, Tomer Galanti, Lior Wolf

Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, 2021

Tensor Programs IIb: Architectural Universality Of Neural Tangent Kernel Training Dynamics.

Greg Yang, Etai Littwin

Proceedings of the 38th International Conference on Machine Learning, 2021

2020

On the Optimization Dynamics of Wide Hypernetworks.

Etai Littwin, Tomer Galanti, Lior Wolf

CoRR, 2020

Residual Tangent Kernels.

Etai Littwin, Lior Wolf

CoRR, 2020

On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width.

Etai Littwin, Lior Wolf

CoRR, 2020

Collegial Ensembles.

Etai Littwin, Ben Myara, Sima Sabah, Joshua M. Susskind, Shuangfei Zhai, Oren Golan

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

On Infinite-Width Hypernetworks.

Etai Littwin, Tomer Galanti, Lior Wolf, Greg Yang

Proceedings of the Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020

2018

Regularizing by the Variance of the Activations' Sample-Variances.

Etai Littwin, Lior Wolf

Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, 2018

2016

The Loss Surface of Residual Networks: Ensembles and the Role of Batch Normalization.

Etai Littwin, Lior Wolf

CoRR, 2016

Complexity of multiverse networks and their multilayer generalization.

Etai Littwin, Lior Wolf

Proceedings of the 23rd International Conference on Pattern Recognition, 2016

The Multiverse Loss for Robust Transfer Learning.

Etai Littwin, Lior Wolf

Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016

2015

Spherical embedding of inlier silhouette dissimilarities.

Etai Littwin, Hadar Averbuch-Elor, Daniel Cohen-Or

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015