✨ feat: Add support for Qwen3 model

[johnmai-dev]

Add support for Qwen3 model
Port of https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/models/qwen3.py

[awni]

Looks great!!

Can you add a model configuration for Qwen3 1.7B:

Assume it will be here: mlx-community/Qwen3-1.7B-4bit

And can you update the default model in the LLM app to use that config (as apposed to the dated older qwen of a similar size?

[johnmai-dev]

Looks great!!

Can you add a model configuration for Qwen3 1.7B:

Assume it will be here: mlx-community/Qwen3-1.7B-4bit

And can you update the default model in the LLM app to use that config (as apposed to the dated older qwen of a similar size?

Received, sir.🫡🫡🫡

[awni]

Thanks for the changes @johnmai-dev. The model runs and produces ok output.. but it looks like there may be something off with the chat template 🤔 .

[johnmai-dev]

Thanks for the changes @johnmai-dev. The model runs and produces ok output.. but it looks like there may be something off with the chat template 🤔 .

Yes, it is indeed an issue with Jinja. I will only have time to fix it after work in the evening. The error message is: Runtime: Cannot call something that is not a function: got UndefinedValue

[DePasqualeOrg]

I added some missing string methods here: johnmai-dev/Jinja#15

This seems to fix the error.

[awni]

Awesome thanks @DePasqualeOrg. Let us know when it lands. We may need to make a PR on swift transformers to bump the jinja package version as well.

[DePasqualeOrg]

The update is available now. Because swift-transformers uses upToNextMinor for the dependencies, and this was a patch update, mlx-swift-examples should pick up the latest version when you update to the latest package versions in Xcode.

[johnmai-dev]

Awesome thanks @DePasqualeOrg. Let us know when it lands. We may need to make a PR on swift transformers to bump the jinja package version as well.

Jinja version 1.1.2 has been released. Only Update Package 😉

[johnmai-dev]

Thinking Mode

Non-thinking Mode

[awni]

Awesome! Tested locally and it works well now.

Regarding the thinking toggle.. I think we should find a way to keep it cause it's fun to play with.

But there are a couple issues to think through there:

1. It will always be there even for non-thinking models. Which doesn't really make sense
2. In the past I've seen other models use different tokenizer flags for that. So how does one generalize that other models?

[DePasqualeOrg]

I don't know if it will be possible to reliably determine this based on the tokenizer config files. My app Local Chat adds a lot of metadata about the models, such as whether they use thinking tags, whether the response starts with the start thinking tag, etc. To work reliably, this should probably be part of the model preset configurations, and more options may need to be added over time as models become more diverse.

[johnmai-dev]

Awesome! Tested locally and it works well now.

Regarding the thinking toggle.. I think we should find a way to keep it cause it's fun to play with.

But there are a couple issues to think through there:

1. It will always be there even for non-thinking models. Which doesn't really make sense
2. In the past I've seen other models use different tokenizer flags for that. So how does one generalize that other models?

Yes, I'm also debating whether to add a non-thinking mode.😂 But if we don't add it, we'll have to default to thinking mode.

[awni]

To work reliably, this should probably be part of the model preset configurations

Indeed.. I was thinking something similar. Models may need optional additional config info including which toggles they have (tools, thinking). And we can dynamically set that based on the config..

I'm ok keeping it the way it is for now. But with the intention of keeping an eye on it and refactoring / redesigning if it becomes more important.

[johnmai-dev]

To work reliably, this should probably be part of the model preset configurations

Indeed.. I was thinking something similar. Models may need optional additional config info including which toggles they have (tools, thinking). And we can dynamically set that based on the config..

I'm ok keeping it the way it is for now. But with the intention of keeping an eye on it and refactoring / redesigning if it becomes more important.

I had planned to expose modelType and only display the Thinking Toggle when modelType is Qwen3.

[johnmai-dev]

I'm ok keeping it the way it is for now. But with the intention of keeping an eye on it and refactoring / redesigning if it becomes more important.

If we keeping it, this PR is ready for review & merge. 🍻
@awni @davidkoski

[awni]

I had planned to expose modelType and only display the Thinking Toggle when modelType is Qwen3.

Yes, something like that could also work. Though I think it may be better to read it as an optinoal field in the config rather than hard code the model types that should have different features.

But, let's do this type of stuff in a follow on PR so we can prioritize merging this without the extra complication.

[awni]

LGTM!

[davidkoski]

Yeah, I think we can keep the toggle there in the example (just like we have the tool piece showing how to integrate that). If there is some metadata in the tokenizer config we can hook that up -- maybe there is something similar for tools?

[davidkoski]

Looks great, thank you for the contribution! @johnmai-dev and @DePasqualeOrg

[mzbac]

Thank you all for the awesome work, can't wait to integrate it in my mlx swift projects.

[DePasqualeOrg]

I had planned to expose modelType and only display the Thinking Toggle when modelType is Qwen3.

It might be easier to maintain if you use a field related to the functionality you want to control. For this type of additional functionality that is specific to some models, my app has a field that is a list of features, one of which could be an enum case toggleThinking.