LLaMa model no longer supported

[mMabeck]

I'm trying to ingest the state of the union text, without having modified anything other than downloading the files/requirements and the .bin files.

It's giving me this error:

/usr/local/bin/python3 /Users/magnusmabeck/privateGPT/ingest.py
%                                                                                                                                                          
magnus@MBA---Magnus privateGPT % /usr/local/bin/python3 /Users/magnus/privateGPT/ingest.py
Loading documents from source_documents
Loaded 1 documents from source_documents
Split into 90 chunks of text (max. 500 tokens each)
llama.cpp: loading model from /Users/magnusmabeck/privateGPT/models/ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format     = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 1024
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 7B
error loading model: this format is no longer supported (see https://github.com/ggerganov/llama.cpp/pull/1305)
llama_init_from_file: failed to load model

It seems the creator of llama.ccp (https://github.com/ggerganov/llama.cpp) has changed the bit-order which breaks former quantizations.

Is no one else having this "bug" or does anyone have some updated quantizationed models?

Environment (please complete the following information):

• OS / hardware: [macOS 13.3 / M2]
• Python version [3.11.3]

[andreakiro]

Indeed llama.cpp changed their quantization encoding. Options for a quick solve (from easiest to hardest):

• Avoid using the latest version of llama-cpp-python (0.1.50) and go for 0.1.49 or 0.1.48 (as in requirements.txt

$ pip install llama-cpp-python==0.1.48
$ python ingest.py # should work fine

• Download new model binaries (you should find them yourself though, as i don't have direct links)
• Requantize your current binary yourself on your machine, using the llama.cpp instructions

[dennydream]

Can confirm other comment. I ran into same issue then downgraded to 1.48.