Pretrained language models (LMs) are commonly finetuned to adapt them to latest domains or tasks, a process often called finetuning. While finetuning allows for adaptation to numerous functions with small amounts of in-domain data, it may be prohibitively expensive for big LMs. 

Parameter-efficient finetuning (PEFT) methods offer an answer by updating only a fraction of the weights, reducing memory usage and training time. Adapters, a standard PEFT approach, learn edits that might be added to a subset of model weights or operate alongside the frozen base model. Recent advancements like LoRA and its variants reduce the variety of trainable parameters through the use of low-rank approximations during adapter training.

Nonetheless, a big aspect of current PEFT methods is their concentrate on modifying weights reasonably than representations, despite prior research indicating that representations encode wealthy semantic information. Representation Finetuning (ReFT) methods have been proposed in response to this by a team of researchers from Stanford and Pr(Ai)2R Group.

As an alternative of adapting model weights, ReFT methods train interventions to control a small fraction of model representations, steering model behaviors to unravel downstream tasks at inference time. Their approach draws inspiration from recent work in LM interpretability, which intervenes on representations to discover causal mechanisms and steer model behaviors at inference time.

One notable instance of the ReFT family is the Low-rank Linear Subspace ReFT (LoReFT), which intervenes on hidden representations within the linear subspace spanned by a low-rank projection matrix. LoReFT builds directly on existing methods like distributed alignment search (DAS), demonstrating state-of-the-art performance on various benchmarks while using significantly fewer parameters than traditional PEFT methods. Their results suggest that ReFT methods offer more efficient and effective alternatives to weight-based PEFTs, deserving further exploration across different model families and domains.

Future research directions for ReFT include exploring its effectiveness on other model families and vision-language models and automating hyperparameter search. Moreover, investigating more practical interventions for specific tasks and exploring the facility of learned orthogonal subspaces are areas of interest. ReFT advances neural network interpretability research and contributes insights back to the sphere, difficult traditional approaches to interpreting individual neurons in isolation.

By way of evaluation practices, it’s essential to determine benchmarks that allow for fair comparisons of PEFTs and ReFTs, including compute- or time-matched hyperparameter-tuning comparisons and disallowing tuning or model selection based on the test set to mitigate overfitting and ensure real-world performance assessment.

Try the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

For those who like our work, you’ll love our newsletter..

Don’t Forget to hitch our 40k+ ML SubReddit

Need to get in front of 1.5 Million AI Audience? Work with us here