chore(deps): update dependency peft to v0.20.0 - #275
Open
dreadnode-renovate-bot[bot] wants to merge 1 commit into
Open
chore(deps): update dependency peft to v0.20.0#275dreadnode-renovate-bot[bot] wants to merge 1 commit into
dreadnode-renovate-bot[bot] wants to merge 1 commit into
Conversation
| datasource | package | from | to | | ---------- | ------- | ------ | ------ | | pypi | peft | 0.19.1 | 0.20.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
| Package | Change | Age | Confidence |
|
Generated Summary:
peftdependency from version0.19.1to0.20.0.peft 0.20.0.torchversion at2.13.0.This summary was generated with ❤️ by rigging
| peft |
| 
|
==0.19.1→==0.20.0|Release Notes
huggingface/peft (peft)
v0.20.0Compare Source
Highlights
This release adds no less than nine new PEFT methods and puts a lot of work into the surrounding infrastructure, for example adding a new image generation benchmark for the method comparison suite and greatly improving the documentation structure.
New Methods
HiRA
@hqsiswiliam added "HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models" to PEFT (#2668). Instead of adding the low-rank product
BAto the base weight, HiRA multiplies it elementwise (Hadamard product) with the frozen base weight. Because the base weight itself is full rank, the resulting update is no longer constrained to be low rank, while the trainable parameter count stays the same as LoRA's.GLoRA
@not-lain contributed GLoRA: "One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning" in #3098. It is a flexible PEFT method that extends LoRA with configurable weight, activation, and bias adaptation, delivering richer fine-tuning with no extra inference cost. Use it when you need per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is already sufficient and simplicity matters.
BEFT
@whubaichuan added "BEFT: Bias-Efficient Fine-Tuning of Language Models" in #3195. BEFT builds on the observation that fine-tuning bias terms alone can be competitive in low-data regimes, but goes further: rather than training all biases, it targets the value projection by default, as the authors found this to be most efficient. This brings the trainable parameter count down to roughly 0.01% of the total parameters.
MonteCLoRA
@victor7246 integrated MonteCLoRA, "Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation" in #2943. LoRA is known to be sensitive to hyperparameters like learning rate and batch size. This new LoRA variant addresses this by treating the low-rank parameters as a distribution rather than a point estimate, using Monte Carlo estimation to obtain a low-variance posterior estimate. This should make training noticeably more robust, at the cost of only O(rank) additional parameters.
VeLoRA
@roymiles added VeLoRA: "Memory Efficient Training using Rank-1 Sub-Token Projections" in #3159. Unlike most PEFT methods, this LoRA variant targets activation memory rather than parameter count: intermediate activations are split into sub-tokens and compressed with a fixed rank-1 projection before being cached for the backward pass, then reconstructed during backpropagation. Conceptually, it is similar to gradient checkpointing, with a lower memory saving but running faster.
Uni-LoRA
@KaiyangLi1992 contributed Uni-LoRA: "One Vector is All You Need" in #3257. The paper shows that parameter-efficient LoRA variants like VeRA and VB-LoRA can all be described as projecting the full LoRA parameter space down to a much smaller subspace, differing only in the choice of projection. Uni-LoRA uses a single global projection across the whole model instead of layer-wise ones, which allows cross-layer parameter sharing and thus very low parameter counts.
FRoD
@Bane-Elvin added FRoD, "Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence", in #3270. Instead of adding low-rank deltas like LoRA, it reconstructs selected weights with shared rotational subspaces and sparse trainable coefficients. It is especially useful when fast convergence and a higher full-rank capacity ceiling are important, and its large sparse rotational subspace may also be promising for model merging. The main tradeoffs are the costly joint-decomposition initialization and slightly slower forward/backward passes than LoRA, so it may be less attractive for a one-off single-task fine-tune.
MiCA
@sr-networks contributed MiCA, "MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning", in #3260. MiCA is a LoRA variant that initializes
Bfrom the SVD of the base weight, taking the left singular vectors belonging to the smallest singular values, and then trains onlyA. The idea is that adapting these underused directions integrates new knowledge more effectively than adapting the dominant subspace, with less interference with existing capabilities.DEFT
@MAXNORM8650 added DEFT, "Decompositional Efficient Fine-Tuning for Text-to-Image Models", in #3342. This PEFT method splits a weight update into two learned low-rank parts: a projection that removes a sub-space of the frozen weight, and a low-rank update that injects new content in its place. Use DEFT for personalizing a text-to-image model from a few images while retaining the base model's instruction-following/editability with minimal forgetting. It is less of a fit if you don't need to preserve the base model's other capabilities (a plain additive adapter is simpler) or for layers beyond Linear/Conv1D. The PaRa method is also supported as a special case of DEFT.
Enhancements
Automatic LoRA target selection
@oswaldoludwig added
KappaTuneSelectorin #3106 based on "The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units". The PR adds a new function,find_kappa_target_modules, which automatically identifies which modules best to target based on the condition number of the base weight matrices. This removes some of the guesswork from choosingtarget_moduleswhen configuring LoRA or other PEFT methods.Broader quantization support
#3117 adds generic quantization support to BOFT, MiSS, and VeRA, and #3321 does the same for SHiRA. Previously, quantization support had to be implemented per method and per backend; more methods now work with the common quantization backends out of the box. Note that as a consequence, VeRA now uses the generic quantization backend instead of its previous bitsandbytes quantization backend. If you encounter issues because of that, please let us know.
Multiple adapters with
target_parameterstarget_parameters, which allows targetingnn.Parameters directly (useful for MoE models that pack experts into a single parameter), now supports adding and loading multiple adapters (#3350).AdaLoRA for Conv2d
@Anai-Guo added
SVDConv2din #3196, so AdaLoRA can now targettorch.nn.Conv2dlayers.Documentation restructure
The PEFT documentation was reorganized in #3300 and #3325, which should make it considerably easier to navigate now that PEFT supports so many methods. Each PEFT method also provides an overview of how well it fares in the PEFT benchmarks. There is also a long tail of documentation improvements from many contributors this release, thanks to everyone who worked on those.
Changes
AutoGPTQ deprecation
Following the switch to GPT-QModel as the backend in the previous release, AutoGPTQ is now formally deprecated (#3190) thanks to @Qubitium.
Benchmarks
Image generation benchmark
The method comparison suite now has an image generation benchmark (#3082). This complements the existing MetaMathQA benchmark and allows to compare PEFT methods on a completely different task. This gives us a more complete picture of the capabilities of different PEFT methods. To check the results, visit the Gradio Space and on the top left, select the task "image-gen".
A look at the results from the image generation benchmark
MetaMathQA
Since the last release, we also made several improvements to the MetaMathQA benchmark. You should check the updated results in the Gradio Space. With the help of the community, we now cover a much broader range of experiments. Moreover, we discovered a bug in the forgetting metric -- the new numbers now accurately reflect how much fine-tuning retains of the original knowledge of the base model.
PEFT Shop
#3317 adds a new Gradio Space with the goal of making it very easy to compare different PEFT methods not only with regard to performance but also capabilities (e.g. if quantization is supported). Give this Space a visit to find out which PEFT method is the best fit for your use case.
Browse the PEFT shop for the best PEFT method for your use case
All Changes
test_lora_seq2seq_lm_multi_gpu_inferenceby @kaixuanliu in #3226New Contributors
Full Changelog: huggingface/peft@v0.19.0...v0.20.0
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Mend Renovate CLI.