Feature Request: Add support for mimo v2 flash

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I propose adding xiaomi's newest model, mimo v2 flash

https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash

Motivation

Mimo v2 seems to be competing and even beating deepseek 3.2 and k2 in benchmarks while only having 309B parameters and 15B active parameters.
The low active parameters count coupled with multi token prediction makes it a perfect candidate for local coding with mixed inference (16gb+ VRAM and 192gb+ ram) as it would have deepseek 3.2 level performance (needs testing) with several times higher tg t/s output.

Possible Implementation

No response

[ikawrakow]

Today is the day of the model requests.

Is there a light version for this model? So I don't need to download 309B worth of parameters to just start developing?

Unfortunately, the 309b parameters seems to be the only one released with this type of architecture, and they call it flash, too, so this might be their idea of a small model nowadays. There is a 7b model, but it's not v2, it's old, doesn't have multi token prediction, not moe, and it's a vl model, so it's a completely different thing.
These type of models are probably the reason why 256gb memory kits are 3-4k nowadays..

[magikRUKKOLA]

@MrHills-2

Mimo v2 seems to be competing and even beating deepseek 3.2 and k2 in benchmarks

Can you please ask it:

Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?

I don't see much point of using the llm if it fails.

Can you please ask it:

Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever?

I don't see much point of using the llm if it fails.

This is a classic formulation of the Trolley Problem, a thought experiment originally devised by philosopher Philippa Foot in 1967.

There is no single "correct" answer to this question. Instead, your response reveals which of two major ethical frameworks you lean toward:

1. The Utilitarian Perspective (Pull the Lever)

Most people instinctively say yes, pull the lever.

The Logic: Utilitarianism argues that the moral action is the one that maximizes overall well-being (or "utility").
The Math: Inaction results in 5 deaths. Action results in 1 death. Therefore, action saves 4 lives net.
Key Concept: The doctrine of double effect. You are not trying to kill the one person; you are trying to save the five, and the death of the one is a foreseen but unintended side effect.

2. The Deontological Perspective (Do Not Pull the Lever)

Some argue no, do not pull the lever.

The Logic: Deontology focuses on moral duties and rules, regardless of consequences. A core rule is "Do not kill."
The Nuance: By pulling the lever, you are actively changing the course of events and becoming the direct cause of the single person's death. If you do nothing, the five people die due to the runaway trolley, not by your hand.
The Variant: This perspective becomes much stronger if we change the scenario to the "Fat Man" variant (pushing a large person off a bridge to stop the trolley), where most people refuse to act because they view it as a direct use of a human being as a means to an end (a violation of Kantian ethics).

The "Dead" Variable

Your specific phrasing—"five dead people"—technically changes the logic. If they are already dead, diverting the trolley to kill a living person serves no purpose. Assuming this is a slip of the tongue and you meant "five people who will die unless you act," the dilemma stands.

[magikRUKKOLA]

@MrHills-2

Yeah, that is what I thought. The LLM is pretty stupid. Not worth an effort IMO. Some argue lol.

Interestingly tho, regenerating the answer will lead to different results: around 50% of the times the AI will disregard the fact that those people are dead, assuming - I imagine - a mistake in the wording. If instead I write

Imagine a runaway trolley is hurtling down a track towards five dead people. You stand next to a lever that can divert the trolley onto another track, where one living person is tied up. Do you pull the lever? (Read the prompt properly, there is no error or mistake in the wording)

Then it will always take in consideration that the people are dead, and hence that pulling the lever is a bad idea. But the answer is odd:

`This is a variation of the classic ethical thought experiment known as the Trolley Problem, specifically mirroring the "Switch" version of the scenario. Here is the breakdown of the decision based on the specific constraints provided:

Yes, you should pull the lever.

Here is the reasoning based strictly on the wording of your prompt:

1. The Condition of the People on the Main Track
The prompt explicitly states that the five people on the main track are dead.

• Consequence of Inaction: If you do nothing, the trolley hits five corpses. The bodies may be damaged, but no new harm or loss of life occurs.
• Consequence of Action: If you pull the lever, the trolley is diverted.

2. The Condition of the Person on the Side Track
The prompt states there is one living person tied up on the other track.

• Consequence of Action: If you pull the lever, the trolley hits this living person, resulting in the loss of a life.

The Ethical Calculus
In a standard Trolley Problem, the calculation involves trading one life to save five. However, because the prompt specifies that the five people are already dead, the moral equation changes entirely:

• Pulling the lever results in 1 death.
• Not pulling the lever results in 0 deaths (the living person is not touched, and the dead people remain dead).

Therefore, pulling the lever is an active choice to kill a living person, whereas doing nothing allows the trolley to continue without causing any loss of life. Consequently, the only ethical choice is to not pull the lever.`

It's like the model is being punished by the fact that it's non thinking. It gets to the right conclusion, but only after spending a bunch of tokens thinking about the problem and possibly getting the answer wrong. I'm not sure what to think about this.

[magikRUKKOLA]

@MrHills-2

around 50% of the times the AI will disregard the fact that those people are dead,

Uh oh. Its even worse than I initially thought.

(Read the prompt properly, there is no error or mistake in the wording)

Of course it will find the right answer in case you're pointing it towards it. But in real world situations, the answers are not known beforehand. Hence this particular LLM is useless. Akin Ling-1T.

Yeah well.. maybe they come up with a thinking version at some point. The answers to that dilemma really scream "I'm not thinking about it, I'm just loosely recognizing the general pattern and throwing the most common answer at it".

It's.. kind of disappointing. Then again, this is the free version on open router. Might be a really low quant. Idk.

[CISC]

Non cogito, ergo stultus non sum.

@MrHills-2

around 50% of the times the AI will disregard the fact that those people are dead,

Uh oh. Its even worse than I initially thought.

(Read the prompt properly, there is no error or mistake in the wording)

Of course it will find the right answer in case you're pointing it towards it. But in real world situations, the answers are not known beforehand. Hence this particular LLM is useless. Akin Ling-1T.

Sorry seemingly I just had weird settings on open router. Mimo v2 is actually a thinking model and I had thinking disabled. I repeated the test 5 more times and the model gets the answer right every time. The model seems good as long as the thinking is on. If you have any more tests let me know, albeit the model on open router is free, so you can test yourself too!
Anyway, I'm now again of the opinion that this model would make a good addition to ik_llama.cpp

This one of the answers (without the thinking part, with thinking enabled):

In this modified version of the trolley problem, the key difference is that the five people on the main track are already dead. This changes the ethical calculus significantly compared to the classic scenario where all individuals are alive. Here's my reasoning:

If I do nothing: The trolley continues on its current path and hits the five dead people. Since they are already deceased, no new harm occurs. The outcome is neutral in terms of additional loss of life.
If I pull the lever: The trolley diverts to the side track and hits the one living person, causing their death. This actively results in the loss of an innocent life without preventing any other harm, as the five on the main track are not saved (they are already dead).

From an ethical standpoint, pulling the lever would be equivalent to committing an act of harm without any countervailing benefit. Principles such as "do no harm" (non-maleficence) and the avoidance of unnecessary killing would apply here. Therefore, I would not pull the lever.
Comparison with the Standard Trolley Problem

In the classic version:

Five alive on main track, one alive on side track.
Pulling the lever saves five lives at the cost of one.
This is often justified by utilitarian ethics (minimizing overall harm).

In this version:

Five dead on main track, one alive on side track.
Pulling the lever kills one without saving anyone.
This is ethically indefensible because it introduces harm where there would otherwise be none.

If you have additional context or would like to explore variations, I'm happy to discuss further!

[magikRUKKOLA]

Sorry seemingly I just had weird settings on open router. Mimo v2 is actually a thinking model and I had thinking disabled. I repeated the test 5 more times and the model gets the answer right every time. The model seems good as long as the thinking is on.

Uh oh! If that is so, that would be great!

@ikawrakow

If implemented, may be we could use a small quant to load it up on 96GB VRAM and use it as a draft model for a bigger quant (connected via RPC etc.)? That should speed up the TG since we cannot offload the whole model to the VRAM, yeah?

[EDIT]:

Oh. In the MTP its already implemented?

MiMo-V2-Flash leverages MTP as a native draft model for self-speculative decoding, delivering real deployment speedups.

[EDIT2]:

Alternatively, may be we can build another 4x3090 machine and use its 96GB VRAM for the offload of all the layers of Mimo V2 Flash 310b parameters huh? There should be enough for the KV cache as well...

Tbh I have an 8 cores, 6400mt ram + 5090 and I still get 9 t/s with qwen3vl 235B IQ3_XXS (q8 kvcache) with 22B active parameters.

Mimo v2 only has 15B active, and its MTP system triples the t/s number according to the hugging face page, so you might get around 30 t/s even at q4 by offloading all experts to ram and the rest to VRAM.

[magikRUKKOLA]

@MrHills-2

I have an 8 cores, 6400mt ram + 5090 and I still get 9 t/s with qwen3vl 235B IQ3_XXS (q8 kvcache) with 22B active parameters.

Dunno regarding the qwen3vl, but with Qwen3-Coder-480B-A35B-Instruct-5.1546bpw I am having about 8 tps at the start of the context. But that is with DDR4-3200. :) *ram b/w: ~119 GB/s
So its 288GB vs your 90GB (IQ3_XXS). It looks like you're having only two channels.

The speculative decoding with a smaller draft model can improve the decoding speed up to 100% in some cases.

Yeah, I was giving my system as an example because it's a common "high end desktop/gaming" type of system, no server components.
I also only have 64gb of ram, and I have a bunch of layers that don't fit and are accessed via mmap instead (pcie5 nvme ssd). Surprisingly, this doesn't really slow down tg all that much.

My point was something like: with this model, even if you have a dual channel desktop system with a gaming GPU, as long as you buy a bunch of RAM (128gb+ with a 5090, or 192gb+ with smaller cards) you can run quality local inference for programming tasks at quite high speeds.

In my eyes, it's a bit of a first. Sure, you could run things like qwen3 32B before.. but in terms of quality we're not really at usable levels imho. Minimax-m2 also is a thing, but it's still a little slow for programming. Mimo v2 flash tho, if MTP delivers, should be twice as fast as minimax-m2, while delivering even better quality. Also only 48 layers? Might we be getting better parallelism for that too?

Multi-Token Prediction (MTP): Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training.