Support Mimo-2.5

[ikawrakow]

Just the small change in 2.5 that adds a separate scale for the V tensors. Nothing related to vision or audio.

[ghost]

It gives me nonsense

的20 ** the** the *2. to on the1111 the the the the 49 49.11149

Also i'm not sure the KV cache is supposed to have different sizes for K and V, just one thing i noticed

llama_init_from_model: KV self size  = 7337.23 MiB, K (q8_0): 4402.34 MiB, V (q8_0): 2934.89 MiB

with:

      ./build/bin/llama-server
      -m pmodels/MiMo-V2.5-IQ4_XS.gguf
      -a mimo
      -ot "blk\.(?:[0-9]|[1-5][0-9]|[6][0-9])\.ffn.*_exps.*=CPU"
      -c 65000
      --parallel 1
      -b 4096 -ub 4096
      -ctk q8_0 -ctv q8_0
      --threads 7 --threads-batch 7 -ngl 95
      --host 127.0.0.1
      --port 8080
      --webui none
      --jinja

[ghost]

I might need further testing but I suspect that this breaks slot save for Qwen 3.5 too. This happens when restoring a slot saved with this PR.

INFO [      log_server_request] request | tid="139633827655680" timestamp=1777746132 remote_addr="127.0.0.1" remote_port=46462 status=200 method="POST" path="/slots/0" params={"action":"restore"}
/home/user/AI/ik/src/llama.cpp:7514: GGML_ASSERT(kv_self.cells[kv_self.head].pos == batch.pos[0]) failed
ptrace: Operation not permitted.
No stack.
The program is not being run.

[sayap]

I was testing a IQ3_XXS/IQ4_XS quant previously using the mainline PR branch from AesSedai. Using the same quant, this PR branch gives the same output, with better TG.

Speed with mainline PR:

prompt eval time =   21072.63 ms /  5183 tokens (    4.07 ms per token,   245.96 tokens per second)
       eval time =   47441.81 ms /   712 tokens (   66.63 ms per token,    15.01 tokens per second)

Speed with this PR:

prompt eval time =   20937.26 ms /  5183 tokens (    4.04 ms per token,   247.55 tokens per second)
       eval time =   42457.91 ms /   724 tokens (   58.64 ms per token,    17.05 tokens per second)

I made this quant without using imatrix, and the quality is so-so. Will make another one with IQ3_KS/IQ4_KSS and imatrix..

[ghost]

I was testing a IQ3_XXS/IQ4_XS quant previously using the mainline PR branch from AesSedai. Using the same quant, this PR branch gives the same output, with better TG.

Speed with mainline PR:

prompt eval time =   21072.63 ms /  5183 tokens (    4.07 ms per token,   245.96 tokens per second)
       eval time =   47441.81 ms /   712 tokens (   66.63 ms per token,    15.01 tokens per second)

Speed with this PR:

prompt eval time =   20937.26 ms /  5183 tokens (    4.04 ms per token,   247.55 tokens per second)
       eval time =   42457.91 ms /   724 tokens (   58.64 ms per token,    17.05 tokens per second)

I made this quant without using imatrix, and the quality is so-so. Will make another one with IQ3_KS/IQ4_KSS and imatrix..

Can you share the command and quant you used?

[ChicoPinto70]

I tested it with AesSedai non pro Q5_K_M and it worked fine (it just enter in an endless reasoning loop with a more complex prompt, but it also happens with the llama.cpp).

[Downtown-Case]

@sayap Can you share the ik imatrix file you made, if you made one?