Large neural network models dominate natural language processing and computer vision, but their initialization and learning rates often depend on heuristic methods, resulting in inconsistency across studies and model sizes. The µ-Parameterization (µP) proposes scaling rules for these parameters, facilitating zero-shot hyperparameter transfer from small to large models. Nevertheless, despite its potential, widespread adoption of µP is hindered by implementation complexity, quite a few variations, and complicated theoretical underpinnings.

Although promising, empirical evidence on the effectiveness of µP at large scales is lacking, raising concerns about hyperparameter preservation and compatibility with existing techniques like decoupled weight decay. While some recent works have adopted µP, open questions remain unresolved, prompting further investigation.

The µP proposed inside the Tensor Programs series demonstrated zero-shot hyperparameter transfer, yet concerns arose regarding stability and scalability for large-scale transformers. Recent works explored hyperparameter tuning with µP but lacked evidence of its efficacy for big models. Some suggest using µ-Transfer to avoid large-scale experiments, while others propose alternative methods like scaling laws based on computing budget or architectural adjustments. Automatic Gradient Descent and Hypergradients offer complex alternatives for learning rate tuning but may lack affordability in comparison with µP.

The researcher investigates µP for transformers concerning width. The µP enables hyperparameter transfer from small to large models, specializing in width for transformers. It presents scaling rules for initialization variance and Adam learning rates. The paper assumes specific values for model parameters and follows scaling rules based on the bottom learning rate α.  Also, it adjusts the eye scale τ−1 for simplicity, observing its impact on performance and transfer. Overall, µP offers a scientific approach to parameter scaling in neural networks.

The RMSNorm ablation tests the efficacy of trainable scale vectors (‘gains’) and their impact on learning rate transferability under µP. Results show unreliable transfer of optimal learning rates with Θ(1) scaling for gains, negatively affecting model quality in large µP models. Zero-initialized query projections enhance transfer and barely improve loss. Using the usual attention scale harms performance. Multiplicative nonlinearities allow transfer despite potential interference. Lion optimizer fails to transfer base learning rates, while multi-query attention stays compatible. Large-scale experiments confirm µ-Transfer’s effectiveness, predicting optimal learning rates even at significantly larger scales, suggesting minimal interference from emergent outliers.

To conclude, This research evaluated µ-Transfer’s reliability in transferring learning rates for transformers. µP succeeded in most scenarios, including various architectural modifications and batch sizes. Nevertheless, it did not transfer when using trainable gain parameters or excessively large attention scales. The easy µP approach outperformed the usual parameterization for transformers. Notably, µ-Transfer accurately predicted optimal learning rates from a small to a vastly larger model. These findings contribute to hyperparameter transfer research, potentially inspiring further exploration in the sector.

Try the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

When you like our work, you’ll love our newsletter..

Don’t Forget to hitch our 40k+ ML SubReddit

Wish to get in front of 1.5 Million AI Audience? Work with us here