model: add Cohere2-MoE North Mini Code support

[joelfarthing]

Date: 2026-06-10

Summary

This adds initial support for CohereLabs North-Mini-Code-1.0 GGUFs using the
cohere2_moe architecture. It has been validated on Q8_0, Q6_K, and Q4_K_M
GGUFs generated from the official North-Mini-Code-1.0 weights. Other
Cohere2-MoE-derived GGUFs may work if they preserve the same metadata and tensor
layout, but are not separately validated here. This compatibility PR includes:

• cohere2_moe architecture registration and model metadata loading
• Cohere2-MoE tensor mapping/loading
• graph support for North-Mini-Code's mixed global/sliding-window attention
  pattern
• RoPE handling for sliding-window layers and the leading dense prefix layer
• leading dense FFN plus routed MoE layer handling
• sigmoid expert gating metadata
• tokenizer/GGUF metadata support, including tokenizer.ggml.pre = cohere2_moe
• converter support for official CohereLabs/North-Mini-Code-1.0 weights
• source chat_template.jinja embedding during conversion

This PR does not include DFlash, MTP, speculation, performance tuning, custom
quantization work, broad chat/server UX changes, or benchmark-quality claims.

Architecture Notes

North-Mini-Code-1.0 is a Cohere2-MoE model released 9 June 2026, not a pure alias of existing Cohere2 or Command-R support.

The correctness-sensitive parts are:

• the model uses Cohere2MoeForCausalLM and GGUF architecture cohere2_moe
• Q/O attention projections are not square in the same way as standard
  Cohere2/Command-R attention tensors
• the attention layer pattern alternates one global layer followed by three
  sliding-window layers
• sliding-window attention uses RoPE, while later global attention layers do not
  use positional embeddings
• the first dense prefix layer also needs RoPE despite using full attention
• the model has one leading dense FFN block, then routed MoE layers
• expert routing uses sigmoid-gated top-k selection
• HF Q/K tensors are already in the interleaved RoPE layout expected by this
  runtime path, so the converter does not apply the Llama Q/K permutation

The converter emits the required Cohere2-MoE GGUF metadata and stacks routed
expert tensors into GGUF expert tensors. The runtime then dispatches to a
dedicated Cohere2-MoE tensor loader and graph builder instead of changing the
existing Cohere2 or Command-R paths.

Validation

Official CohereLabs source revision:

effaeda477c041c107d5a3d8c599cb5d6c5878ef

Reviewer/community convenience GGUFs generated from official CohereLabs weights:

https://huggingface.co/ji-farthing/North-Mini-Code-1.0-ik-llama-validation-GGUF

Files:

North-Mini-Code-1.0-ik_llama-Q8_0.gguf
North-Mini-Code-1.0-ik_llama-Q6_K.gguf
North-Mini-Code-1.0-ik_llama-Q4_K_M.gguf

These GGUFs require an ik_llama.cpp build with this Cohere2-MoE /
North-Mini-Code support, or another runtime with equivalent cohere2_moe
architecture handling. They are not expected to load or decode correctly with
current ikawrakow/main without this PR.

Validation highlights:

• fresh BF16 conversion completed from the CohereLabs checkpoint
• Q8_0, Q6_K, and Q4_K_M were quantized from the converted BF16 GGUF with
  current llama-quantize
• Q8_0, Q6_K, and Q4_K_M metadata was inspected after quantization
• Q8_0, Q6_K, and Q4_K_M embed tokenizer.chat_template
• the embedded template in each quant exactly matches the source
  chat_template.jinja
• Q8_0, Q6_K, and Q4_K_M load in llama-server
• Q8_0, Q6_K, and Q4_K_M pass a short OpenAI-compatible server chat sanity check
• Q8_0, Q6_K, and Q4_K_M were also exercised through a local OpenCode
  configuration and each produced saved code files in that harness
• local Pi artifact evals showed that the GGUFs load, respond, and run in that
  environment, but the Pi harness results are not used here as quality evidence
• CUDA build targets pass locally
• converter/GGUF Python files pass py_compile
• git diff --check passes locally

Representative local server shape used for the fresh artifacts:

./build-cuda/bin/llama-server \
  -m /path/to/North-Mini-Code-1.0-ik_llama-Q6_K.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  -c 32768 \
  -b 512 \
  -ub 512 \
  --cpu-moe \
  -ctk q8_0 \
  -ctv q4_0 \
  --jinja \
  --no-warmup

Notes / Caveats

• Q8_0, Q6_K, and Q4_K_M are provided as reviewer-facing compatibility
  artifacts.

• No BF16 GGUF is published as a user artifact.

• No performance claims are made.

• No benchmark-quality or agentic-coding quality claims are made.

• The validation claim is limited to conversion, loadability, metadata/template
  correctness, and basic local response/code-generation smoke tests.

• The implementation is intentionally scoped to Cohere2-MoE / North-Mini-Code
  compatibility rather than speculative decoding, DFlash, MTP, server UX, or
  broad runtime behavior changes.

• I have read the contributing guidelines

• Self-reported review complexity:

  • Low
  • Medium
  • High

[ikawrakow]

Move this within the if (model.layers[il].ffn_gate_inp == nullptr) branch.

Alternatively, remove COHERE2_MOE from the list of architectures supporting split mode graph and I will fix it after merging the PR.

Setting op_params[3] to 1 results in the AllReduce operation not being performed when using split mode graph (and you have added COHERE2_MOE to the list of architectures supporting split mode graph). This works for the dense layers because each partial result of attn_out is added to the corresponding partial result for the FFN network before running AllReduce. But there is no corresponding implementation in the MoE FFN, so what happens then is that we only add the partial attn_out result for the GPU that runs the AllReduce op.

[joelfarthing]

Thanks! I moved attn_out->op_params[3] = 1 into the dense FFN branch so the MoE layers don't suppress the attention all-reduce in split-mode graph.

I’m really enjoying the chance to work on ik_llama! New models keep adding surprise opportunities to learn.

Commit: 51ea994d9 Fix Cohere2-MoE graph split reduce handling