(possible) Misc. bug: `convert_lora_to_gguf.py` ignores `--outtype`

[jukofyork]

Name and Version

Tested on latest pull.

Problem description & steps to reproduce

I'm not sure if this is a bug or intended:

• Is --outtype supposed to convert to the type, eg: --outtype f16 on a f32 LoRA adapter in PEFT format is supposed to create an f16 LoRA in GGUF format (ie: the same command semantics as convert_hf_to_gguf_update.py)?
• or is it just a "type hint" incase it can't infer the type of the PEFT format LoRA?

I also tried running llama-quant on the resulting LoRA in f32 GGUF format, but it doesn't like being given these files...

Are non-f32 LoRAs even allowed?

If it is a bug then I'll do some more investigation and see if I can find out what is happening.

[CISC]

It's because the lora tensor names don't end in .weight:

llama.cpp/convert_hf_to_gguf.py

Lines 337 to 339 in 233d773

	or not new_name.endswith(".weight")
	):
	data_qtype = gguf.GGMLQuantizationType.F32

Arguably this can be said to be intentional as you generally do not want to deteriorate LoRA weights (and they are usually fairly small).

[jukofyork]

It's because the lora tensor names don't end in .weight:

llama.cpp/convert_hf_to_gguf.py

Lines 337 to 339 in 233d773
or not new_name.endswith(".weight")
):
data_qtype = gguf.GGMLQuantizationType.F32

Arguably this can be said to be intentional as you generally do not want to deteriorate LoRA weights (and they are usually fairly small).

Thanks.

Yeah, I'm not sure if only F32 makes sense, but for very large rank LoRAs (256+) the files can be quite substantial.

I'll close this now as agree it's not clearly a bug and just a little confusing why it ignores the setting.

[CISC]

I suppose it would make sense to use F16 if the original weights were that, but I doubt there are too many of those? Most are BF16 nowadays, and that's not a good idea to use as it will generally perform badly on a lot of hardware.

Allowing Q8 is probably perfectly viable and can be added as a special case for lora weights in prepare_tensors if you wish to make a PR?

[jukofyork]

I suppose it would make sense to use F16 if the original weights were that, but I doubt there are too many of those? Most are BF16 nowadays, and that's not a good idea to use as it will generally perform badly on a lot of hardware.

Allowing Q8 is probably perfectly viable and can be added as a special case for lora weights in prepare_tensors if you wish to make a PR?

I'll do some experiments over the next couple of weeks to see how useful it is first.