Can we optimize large-scale language models faster than Adam? This Harvard AI paper reveals SOAP, which improves and stabilizes shampoo in deep learning.

Efficient optimization of large-scale deep learning models remains a critical challenge as the cost of training large-scale language models (LLMs) continues to increase. As models grow larger, the computational burden and time required for training increases significantly, creating a demand for more efficient optimization tools that can reduce both training time and resources. This challenge is particularly important for reducing the overhead of real-world AI applications and making large-scale model training more feasible.

Current optimization methods include the following first-order optimization programs: Adam and the second method, etc. shampoo. while doing Adam Widely used for its computational efficiency, it often converges more slowly, especially in large-batch systems. In contrast, shampoo Although it provides good performance using hierarchical Kronecker factor preconditioners, it has high computational complexity because it requires frequent eigendecompositions and introduces several additional hyperparameters. This limits Shampoo’s scalability and efficiency, especially for large-scale and real-time applications.

Researchers at Harvard University suggest that soap (ShampooO with Adam in its Preconditioner’s unique criteria) integrates SOAP to overcome Shampoo’s limitations. Adam and shampoo While running Adam Reduce computational overhead by making Shampoo’s preconditioner a unique criterion. This approach minimizes the need for frequent matrix operations and reduces the number of hyperparameters, with SOAP introducing only one additional hyperparameter, the precondition frequency, compared to Adam. This new method improves both training efficiency and performance without compromising accuracy.

SOAP modifies the traditional Shampoo optimizer by updating the preconditioner less frequently and running Adam’s updates in the rotation space defined by the Shampoo preconditioner. It maintains two preconditioners for the weight matrices of each layer and updates them according to the optimized preconditioning frequency. In our experimental setup, SOAP is tested on models with 360M and 660M parameters in large-scale batch learning tasks. The preconditioning frequency and other hyperparameters are optimized so that SOAP can maximize both performance and efficiency, significantly reducing the computational overhead while maintaining high accuracy.

SOAP shows significant improvements in performance and efficiency, reducing the number of training iterations by 40% and the wall clock time by 35% compared to AdamW. It also outperforms Shampoo by 20% on both metrics. These improvements are consistent across a range of model sizes, with SOAP maintaining or exceeding the test loss scores of AdamW and Shampoo. This highlights SOAP’s ability to balance training efficiency and model performance, making it a powerful tool for large-scale deep learning optimization.

as a result, soap It demonstrates significant progress in deep learning optimization by combining computational efficiency. Adam With secondary benefits shampoo. By reducing computational overhead and minimizing hyperparameter complexity, SOAP provides a scalable and efficient solution for training large-scale models. This method, which can reduce both the number of training iterations and the wall clock time without sacrificing performance, highlights its potential to become a practical standard for optimizing large-scale AI models, contributing to more efficient and feasible deep learning training.

Check it out paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us. twitter And join us Telegram Channel and LinkedIn grWhoop. If you like our work, you’ll like us. newsletter..

Don’t forget to join us 50,000+ ML subreddits

⏩ ⏩ Free AI Webinar: ‘SAM 2 for Video: How to Fine-tune Your Data’ (Wednesday, September 25, 4:00-4:45 a.m. EST)

Aswin AK is a consulting intern at MarkTechPost. He holds a dual degree from Indian Institute of Technology, Kharagpur. He is passionate about data science and machine learning and has a strong academic background and hands-on experience solving real-world cross-domain challenges.