Feature Request: Support for HY MT2 30b A3B

[Bearsaerker]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

This would be a very nice addition, as this would fly even on cpu and provide SOTA Translation capabilites. https://huggingface.co/tencent/Hy-MT2-30B-A3B

Motivation

Nice to have and I'd use it much

Possible Implementation

No response

[huanyge]

Not only is a3b not supported, it seems that none of the hy-mt2 models are currently supported
Error message: llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'hunyuan-dense'

[ikawrakow]

The Hunyuan dense models are not supported. But the MoE varians should be. Can you post the log when you try to run ik_llama.cpp with the model of the feature request and it fails? Thanks.

[Bearsaerker]

This is the log attempting to load this gguf

❯ ./llama-server -m /home/luis/Downloads/Hy-MT2-30B-A3B-Q2_K.gguf
INFO [                    main] build info | tid="140631413907456" timestamp=1779862676 build=0 commit="unknown"
INFO [                    main] system info | tid="140631413907456" timestamp=1779862676 n_threads=6 n_threads_batch=-1 total_threads=12 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes, VRAM: 7779 MiB
CUDA0: using device CUDA0 - 7234 MiB free
llama_model_loader: loaded meta data with 36 key-value pairs and 766 tensors from /home/luis/Downloads/Hy-MT2-30B-A3B-Q2_K.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = hy_v3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Hy MT2 30B A3B
llama_model_loader: - kv   3:                           general.basename str              = Hy-MT2
llama_model_loader: - kv   4:                         general.size_label str              = 30B-A3B
llama_model_loader: - kv   5:                               general.tags arr[str,1]       = ["translation"]
llama_model_loader: - kv   6:                          general.languages arr[str,36]      = ["zh", "en", "fr", "pt", "es", "ja", ...
llama_model_loader: - kv   7:                          hy_v3.block_count u32              = 48
llama_model_loader: - kv   8:                       hy_v3.context_length u32              = 262144
llama_model_loader: - kv   9:                     hy_v3.embedding_length u32              = 2048
llama_model_loader: - kv  10:                  hy_v3.feed_forward_length u32              = 6912
llama_model_loader: - kv  11:                 hy_v3.attention.head_count u32              = 32
llama_model_loader: - kv  12:              hy_v3.attention.head_count_kv u32              = 4
llama_model_loader: - kv  13:                       hy_v3.rope.freq_base f32              = 11158840.000000
llama_model_loader: - kv  14:     hy_v3.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                         hy_v3.expert_count u32              = 128
llama_model_loader: - kv  16:                    hy_v3.expert_used_count u32              = 8
llama_model_loader: - kv  17:                 hy_v3.attention.key_length u32              = 128
llama_model_loader: - kv  18:               hy_v3.attention.value_length u32              = 128
llama_model_loader: - kv  19:           hy_v3.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  20:    hy_v3.expert_shared_feed_forward_length u32              = 768
llama_model_loader: - kv  21:                  hy_v3.expert_weights_norm bool             = true
llama_model_loader: - kv  22:                 hy_v3.expert_weights_scale f32              = 2.826000
llama_model_loader: - kv  23:                   hy_v3.expert_gating_func u32              = 2
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = hunyuan-dense
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,120832]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,120832]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,119758]  = ["Ġ Ġ", "Ġ t", "Ġ a", "i n", "h e...
llama_model_loader: - kv  29:                tokenizer.ggml.bos_token_id u32              = 120000
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 120025
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 120002
llama_model_loader: - kv  32:          tokenizer.ggml.seperator_token_id u32              = 120007
llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {# ----------‑‑‑ special token ...
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                          general.file_type u32              = 10
llama_model_loader: - type  f32:  287 tensors
llama_model_loader: - type q2_K:  287 tensors
llama_model_loader: - type q3_K:  143 tensors
llama_model_loader: - type q4_K:   48 tensors
llama_model_loader: - type q6_K:    1 tensors
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'hy_v3'
llama_model_load_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/home/luis/Downloads/Hy-MT2-30B-A3B-Q2_K.gguf'
 ERR [              load_model] unable to load model | tid="140631413907456" timestamp=1779862676 model="/home/luis/Downloads/Hy-MT2-30B-A3B-Q2_K.gguf"

[ikawrakow]

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'hy_v3'

Is this architecture supported anywhere? It isn't supported in llama.cpp for instance.

[saood06]

Is this architecture supported anywhere? It isn't supported in llama.cpp for instance.

Transformers and vLLM have support of it. I did briefly look into it, to see if it is something I could do once my homelab situation gets better and from my brief research it didn't look too bad, nothing exotic from what I could tell but also it isn't popular enough for people to make a visual diagram or talk about the architecture and so I'd have to base my implementation on either vLLM or transformers which would be a first for me.

[Bearsaerker]

The dense models are supported by llama.cpp, only the MoE isn't.

[ikawrakow]

The dense models are supported by llama.cpp, only the MoE isn't.

Then how did you create the GGUF for the MoE model?

[Bearsaerker]

I did not make the gguf. It is some shady patched version of llama cpp as you can see on the huggingface site:

Clone this model repository first, then prepare a patched llama.cpp checkout:

git lfs install
git clone https://huggingface.co/GrahLnn/Hy-MT2-30B-A3B-4bit-GGUF
cd Hy-MT2-30B-A3B-4bit-GGUF
bash scripts/prepare_llama_cpp.sh ./llama.cpp

[ikawrakow]

I did not make the gguf. It is some shady patched version of llama cpp as you can see on the huggingface site:

And now you expect that we will all hurry to add support for this shady patched llama.cpp version?

[Bearsaerker]

Why so passive aggressive? I do not demand anything, I just think that this would be a nice addition if possible. If you or other people set priorities differently this totally fine. Unfortunately I'm not intelligent enough to make this PR myself.

[jilek-josef]

feels like a wrong place for this request, this repo objective are fancy optimizations on top of upstream llama, we need the upstream support first.

[ikawrakow]

@jilek-josef

feels like a wrong place for this request, this repo objective are fancy optimizations on top of upstream llama, we need the upstream support first.

This project was forked in June 2024 and has last synced with llama.cpp in early August 2024, so "fancy optimizations on top of upstream llama" is very far from the truth. There have been a few models that were supported here well before llama.cpp, most notably DeepSeek-V3 with MLA (when that model was the rage of the day last year). MTP was implemented here first, and so was tensor parallel, fused delta-net (they call it "gated delta-net" in llama.cpp), etc.

The requested model has 6,581 downloads as of today, so not exactly popular. Given the limited resources of this project, it is hard to justify spending the time to implement it here. If someone else does it, I would accept the PR (see for instance #1911, which added support for Laguna XS.2, a model missing in llama.cpp but not exactly popular), but don't see working on support for this model myself.

[jilek-josef]

@ikawrakow
I see, that's fair, didn't know you add model support as well.

Given the limited resources of this project, it is hard to justify spending the time to implement it here.

this was mostly my point, at upstream there are more resources, when it get's implemented there, it will be easier to make a PR here as well, or so I would assume at least.
The model has like 50k downloads btw, it has 2 separate repos which are the same except one is FP8 and much more people download it. And it's not exactly single model support, Hy3 MoE is sort of family of models.
I am trying if I can get it working on upstream, if I succeed and I will be satisfied with the implementation I can try to make a PR, no guarantees tho.

[ikawrakow]

The model has like 50k downloads btw,

Then the link provided in the issue should have been to the 50k download repository rather than the 6581 downloads Tencent repository.