Does this work with Synthia-7B-v1.3

[stjepangolemac]

As mentioned, I'm trying to run this model. More info on it here.

I've managed to convert and quantize it without errors but when I run it I get this:

error loading model: create_tensor: tensor 'blk.0.attn_k.weight' has wrong shape; expected  4096,  4096, got  4096,  1024,     1,     1

I converted it with convert.py model_dir --ctx 4096 and then quantized to q4_0.

[slaren]

It works for me. Are you using the latest version?

[stjepangolemac]

Yes, built it from source just before I posted. Can you run me through exact steps you're doing on your machine?

[slaren]

git clone --depth 1 https://huggingface.co/migtissera/Synthia-7B-v1.3
./convert.py Synthia-7B-v1.3/

Note that until very recently, there was an issue converting some HF models that use GQA, such as this one. You need to update llama.cpp to the current master.

[Mihaiii]

It works for me too.

[prusnak]

There are also GGUF files available from here: https://huggingface.co/TheBloke/Synthia-7B-v1.3-GGUF/tree/main