Add KaSA implementation to layer.py

[nsbg]

cc @BenjaminBossan

I've added KaSA implementation part to layer.py by referring to the existing KaSA code. This is a very early draft of the PR, and there's a chance I misunderstood something, so a lot might need to be revised/added.

I'm not entirely sure what the sentence "For the PEFT integration, the new LoRA variant pattern introduced in #2443 should be leveraged (i.e. use a similar approach to extend vanilla LoRA as is used for DoRA)" - from #2516 - means exactly. If you could provide some guidance on this part, it would be really helpful for me.

I'm open to any kind of feedback!

[BenjaminBossan]

Thanks @nsbg for working on KaSA. Sorry for the late reply, I was out of office.

Regarding the implementation, I don't think this works correctly yet, as your code only updates the base weight and it also does it unconditionally, when we want users to be able to enable/disable KaSA.

The proper way to implement this is:

1. Add a new argument to LoraConfig called use_kasa (False by default).
2. Extend the LoRA variant resolution to include use_kasa.
3. Implement a KasaVariant similar to the existing DoraVariant. To see what changes are needed, search for the string diag in the existing KaSA code and port it over to the variant.

Later, we'll also need docs and tests, but this can wait until the implementation is done.

[nsbg]

Thank you for your review. I'll include your feedback in the next step.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

@nsbg No hurry, just asking if you still intend on working on this.