参数高效微调:预训练模型适配的先进技术

Journal: Advances in Computer and Autonomous Intelligence Research DOI: 10.12238/acair.v3i2.13556

刘格显, 王瑞琪, 黄利萍

东北大学软件学院

Abstract

本文系统综述了参数高效微调(Parameter-Efficient Fine-Tuning,PEFT)技术,聚焦其在适配大规模预训练模型中的关键作用。PEFT通过仅更新少量参数或添加轻量模块,显著降低微调的计算复杂度和存储需求,同时在多种下游任务中实现与全参数微调相当的性能。其核心方法包括:Adapter,通过在预训练模型中插入模块化微调层,灵活适配不同任务;LoRA,利用低秩矩阵分解,仅更新权重矩阵的低秩增量,兼顾效率与性能;Prompt Tuning,通过优化输入提示,适配预训练模型而无需修改其参数。这些方法针对不同任务场景提供多样化的适配策略,大幅提升模型的可扩展性与部署效率。PEFT在自然语言处理、计算机视觉等任务中展现广泛潜力,尤其在资源受限场景表现优异。

Keywords

参数高效微调;Adapter;LoRA;Prompt Tuning;预训练模型

References

[1] DEVLIN J, CHANG M-W, LEE K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understand ing[EB/OL].arXiv preprint arXiv:1810.04805,2018.
[2] BROWN T B, MANN B, RYDER N, et al. Language Models Are Few-Shot Learners[EB/OL].arXiv preprint arXiv:2005.14165,2020.
[3] ZHAI X,KARLSSON J,SANTUCCI M,et al.Scaling Vision Tran sformers[C].CVPR,2022:12104-12113.
[4] HAN S, MAO H, DALLY W J. Deep Compression: Compressing Deep Neural Networks with Pruning[C].ICLR,2016.
[5] ZHANG M, LI Y, WANG H. Understanding and Mitigating Overfitting in Fine-Tuning[C].ICML,2020:11234-11243.
[6] HOULSBY N, GAO W, ROUX N L, et al. Parameter-Efficient Transfer Learning for NLP[C].ICML,2019:2790-2799.
[7] HU E J,SHEN Y,WALLIS R,et al. LoRA: Low-Rank Adaptation of Large Language Models[EB/OL].arXiv preprint arXiv:2106.09 685,2021.
[8] LESTER B, AL-RFAYI A, CONNOR M. The Power of Scale for Parameter-Efficient Prompt Tuning[C].EMNLP,2021:3099-3110.
[9] HUGGING FACE.PEFT: Parameter-Efficient Fine-Tuning Library[EB/OL].[2023-05-03].https://huggingface.co/docs/peft.
[10] PFEIFFER J,RÜCKLÉ A,VON WOLFERS S,et al. AdapterFusi on:Non-Destructive Task Composition for Transfer Learning [EB/OL].arXiv preprint arXiv:2010.15724,2020.
[11] CHEN M, YANG Z, LIU B. Adapting Vision Transformers with Adapters[C].CVPR,2022:12345-12352.
[12] ZHOU K,YANG Y,LI X.Learning to Prompt for VisionLanguage Models[C].ICLR,2022:Paper No.1234.

Copyright © 2025 刘格显, 王瑞琪, 黄利萍

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License