Fix: Implement LoRA on Custom Model with Transformer Encoder #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request addresses the issue of implementing LoRA on a custom model that includes a transformer encoder from PyTorch. The main challenge was targeting the q, k, and v projection weights in the self-attention block of the transformer encoder layer.
Changes made:
LoRALayer
class to inherit fromnn.Module
, which is necessary for integrating LoRA with PyTorch modules.This change allows for the correct application of LoRA to the specified projection weights, facilitating the desired functionality in the custom model. This should resolve the issue of not being able to find module names corresponding to the q, k, and v projections in the PyTorch transformer encoder.