Inspiration: the change in weights during model adaptation have a low “intrinsic rank”
Description: Change small matrices A and B when fine-tune, adding A * B to weight W, which significantly reduce the trainable number of parameters because r << d
Novelty: Reducing Fine-tuen cost without additional latency and quality reduction or changing input consturction at the same time
1) Efficiently switch models for different target tasks through switching LoRA matrices 2) Reduce consuming time and hardware requirement when Fine-tune 3) Without any inference lantency 4) Orthogonal with other adaptation method
1) LoRA have better scalability and performances
2) Adapting more weight matrices is preferable than adapting a single type of weights with a larger rank, 2 or 4 is a great option
3) Similiarity of LoRA vectors among different rank(r) is higher when dimention is small, which prove (1)low-demension consists more information, (2) small r is enough
4) Matrices of LoRA is similiar with original weight mtraices, LoRA is a amplification of original information.
1) Purpose of auther? reduce the cost of fine-tuen without any loss
2) Key of new method? change of adaptation have a low “intrinsic rank”
3) What is useful for me? large Matrices in LLM have a low “intrinsic rank”? New fine-tune method
4) What references is necessary to read? Where “intrinsic rank” comes from?
1-4 from Andrew NG
5) new idea
rand-deficiency of delta_w suggests that w could be rank-deficient as well, which can be a source of inspiration of future of works.
This post is licensed under CC BY 4.0 by the author.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。