feat(lokr, loha): add 1x1 Conv2d and Conv1d support

[grewalsk]

What does this PR do?
This PR enhances the LoKr and LoHa adapter implementations within PEFT by adding proper support for:

1. 1x1 Convolutions (nn.Conv2d with kernel_size=(1,1))
2. nn.Conv1d layers (specifically including kernel_size=1).

This allows LoKr/LoHa adapters to be correctly applied to a wider range of modern architectures that heavily utilize these layer types (e.g., ResNet bottlenecks, MobileNet pointwise convolutions, various Transformer blocks). The implementation aims for optimized handling, inspired by LoRA's 1x1 optimization, while maintaining consistency with existing LyCORIS patterns in PEFT. Parts of the implementation logic, particularly for parameter factorization and layer adaptation, were adapted from the KohakuBlueleaf/LyCORIS library (e.g., lycoris/modules/loha.py), consistent with existing acknowledgements within the PEFT codebase.

This includes:

• New Conv1d adapter layer classes for both LoKr and LoHa, mirroring Conv2d.
• Updated layers_mapping in LoKrModel and LoHaModel to recognize Conv1d.
• Enhanced create_adapter_parameters methods in LoKr/LoHa to correctly initialize parameters based on Conv1d weight shapes.
• Refactored update_layer methods in LoKr/LoHa to:
  • Detect Conv1d layers.
  • Implement specific logic for 1x1 Conv2d and kernel_size=1 Conv1d, notably disabling use_effective_conv2d where appropriate for direct matrix handling.
  • Ensure correct shape calculations for factorization.
• Added detection flags (is_1x1_conv2d, is_1_conv1d) in get_delta_weight methods as hooks for potential future computation optimizations (without altering current paths).
• Maintained backward compatibility; changes are additive and do not affect existing functionality for other layer types or kernel sizes.
• Followed established PEFT/LyCORIS coding patterns for consistency.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guidelines?
• Did you write any new necessary tests?
• Did you run the tests?
• Did you run the pre-commit hooks?
• Did you check the documentation builds correctly?

Fixes #1935

[BenjaminBossan]

Thanks a lot for this great PR, it is much appreciated.

I saw some small changes still being required, please check the comments. Also, to accept Conv1d, it also needs to be added here:

peft/src/peft/tuners/lycoris_utils.py

Line 263 in d3ff133

if isinstance(target_base_layer, torch.nn.Conv2d):

Then, it's important to add unit tests to ensure that 1x1 conv and Conv1d work as expected. For this, let's add the following test cases:

    (
        "Conv1D LoHa",
        "Conv1d",
        LoHaConfig,
        {"target_modules": ["conv1d"]},
    ),
    ("Conv2d 1x1 LoHa", "Conv2d1x1", LoHaConfig, {"target_modules": ["conv2d"]}),

...

    (
        "Conv1D LoKr",
        "Conv1d",
        LoKrConfig,
        {"target_modules": ["conv1d"]},
    ),
    ("Conv2d 1x1 LoKr", "Conv2d1x1", LoKrConfig, {"target_modules": ["conv2d"]}),

here and here.

We also need to define the ModelConv2d1x1, e.g. like this:

class ModelConv2D1x1(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv2d = nn.Conv2d(1, 10, 3)
        self.relu = nn.ReLU()
        self.flat = nn.Flatten()
        self.lin0 = nn.Linear(10, 2)
        self.sm = nn.LogSoftmax(dim=-1)
        self.dtype = torch.float

    def forward(self, X):
        X = X.to(self.dtype)
        X = X.reshape(-1, 5, 3, 3)
        X = self.conv2d(X)
        X = self.relu(X)
        X = self.flat(X)
        X = self.lin0(X)
        X = self.sm(X)
        return X

Ensure to add this snippet here:

        if model_id == "Conv2d1x1":
            return ModelConv2D1x1().to(torch_dtype)

[BenjaminBossan]

gentle ping @grewalsk

[grewalsk]

Hi @BenjaminBossan, could you take a quick look?

I removed the conflicting ModelConv2DGroups class and cleared the merge-markers locally, but it still shows up in the PR diff.

[githubnemo]

Thanks for the update, I'll take over the review for now :)

I've had some questions/remarks but nothing major. The merge conflict markers are not visible to me, so that seems resolved.

[grewalsk]

does this address ur concerns or should I add something?

[githubnemo]

Thank you so much for the quick update. I think this looks good to go :)

Let's see what the CI has to say but if there are no further issues, we can proceed merging this.

Took over review

[githubnemo]

@grewalsk could you run make style from the top of the repository and push the changes?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[githubnemo]

It seems that we have two ModelConv1D classes in test_custom_models. Could you fix that? :)
I think there was already a class called ModelConv1D with kernel size 2.

[grewalsk]

Is this good? @githubnemo

[githubnemo]

You checked in a merge conflict, if you could fix that and run make style once again, I think we're good to go

[githubnemo]

Suggested change

	<<<<<<< Updated upstream
	# method_comparison logs
	method_comparison/MetaMathQA/cancelled_results/
	method_comparison/MetaMathQA/temporary_results/
	=======
	**/.claude/settings.local.json
	>>>>>>> Stashed changes
	# method_comparison logs
	method_comparison/MetaMathQA/cancelled_results/
	method_comparison/MetaMathQA/temporary_results/

[grewalsk]

I think its good now. @githubnemo

[githubnemo]

@grewalsk It seems that there are some failing tests regarding LoKr/LoHa. Could you investigate?

You can reproduce those runs locally by running pytest -k loha -k lokr tests/.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[githubnemo]

not stale