论文略读：MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning - 代码天地

论文略读：MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

物联网 2024-11-01 14:15:44 阅读次数: 0

202404 arxiv

LoRA的改进

1 Lora的问题

过低的秩会严重限制模型学习和记忆新知识的能力，尤其在需要获取大量领域知识的任务上

2 mora

MoRA的关键在于使用方阵M取代LoRA的低秩矩阵A和B，以提升rank
- 假设原权重矩阵W的维度为d×k
  - Lora的参数量为(d+k)r，rank为r
  - 相同参数量下，MoRA的rank为 $\sqrt{(d+k)r}$

至于压缩算子，论文给出了多种方法
- 截断：直接截取部分维度
- 共享：维度合并，共享同一个方阵M的值
- 解耦：将输入reshape为矩阵，然后与M做矩阵乘法
- 旋转：在解耦的基础上引入旋转矩阵，增强表达能力

3 结果

猜你喜欢

转载自blog.csdn.net/qq_40206371/article/details/143250103

论文略读：MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

【论文笔记】BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

轻量化微调 Parameter-Efficient Fine-Tuning

Arixv 2403 | Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

预训练大语言模型的三种微调技术总结：fine-tuning、parameter-efficient fine-tuning和prompt-tuning

LLMs PEFT技术1：LoRA Parameter efficient fine-tuning PEFT techniques 1: LoRA Low rank Adaptation

[论文阅读笔记74]The Power of Scale for Parameter-Efficient Prompt Tuning

论文Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-into Attention论文解读

Adapter Tuning：详细解读Parameter-Efficient Transfer Learning for NLP

[论文阅读72]Parameter-Efficient Transfer Learning for NLP

LLMs参数高效微调（PEFT） Parameter efficient fine-tuning (PEFT)

Lora升级！ReLoRa！最新论文 High-Rank Training Through Low-Rank Updates

论文分享|《Universal Language Model Fine-tuning for Text Classificatio》

HRank: Filter Pruning using High-Rank Feature Map论文笔记

LLMs 参数高效微调技术2：软提示 Parameter efficient fine-tuning PEFT techniques 2: Soft prompts

论文解读：P-Tuning v2: Prompt Tuning Can BeComparable to Fine-tuning Universally Across Scales and Tasks

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

论文略读：LoRA+: Efficient Low Rank Adaptation of Large Models

【论文笔记】ULMFiT——Universal Language Model Fine-tuning for Text Classification

《论文阅读》BERT-ERC: Fine-tuning BERT is Enough for Emotion Recognition in Conversation

【论文翻译】Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Ma

论文笔记：Composable Sparse Fine-Tuning for Cross-Lingual Transfer

Re73 读论文：ULMFiT Universal Language Model Fine-tuning for Text Classification

Fine-tuning

什么是fine-tuning？

fine-tuning介绍

论文阅读：Towards a Unified View of Parameter-Efficient Transfer Learning对参数高效迁移学习的统一看法

【(强推)李宏毅2021/2022春机器学习课程】2022-如何有效的使用自监督模型-Data-Efficient &Parameter-Efficient Tuning

模型微调fine-tuning

今日推荐

周排行

win7之session理解

如何快速上手 AB Testing ？阿里技术专家秘方公开

数据库自助查询平台

【CCF 2017-12-2】游戏问题

Fire!(BFS)

修复GRUB引导故障

老倪祖膏药分销系统

MySQL慢查询日志-捕获慢SQL

批归一化(Batch Normalization)

ssm整合问题-------一

每日归档

更多

2025-03-02(0)

2025-03-01(0)

2025-02-28(0)

2025-02-27(0)

2025-02-26(0)

2025-02-25(0)

2025-02-24(0)

2025-02-23(0)

2025-02-22(0)

2025-02-21(0)