Repository navigation

Add DeepSeek-v4 (Flash/Pro) - #1192

Closed

Blaizzy wants to merge 62 commits into

ml-explore:mainfrom

Blaizzy:pc/add-deepseekv4flash-model

Closed

Add DeepSeek-v4 (Flash/Pro)#1192
Blaizzy wants to merge 62 commits into
ml-explore:mainfrom
Blaizzy:pc/add-deepseekv4flash-model

Commits on Nov 12, 2025
- fix input_embeddings prefill bug in generate_step
  Blaizzy
  committed
  2bfc361
- format
  Blaizzy
  committed
  5cf134c
Commits on Dec 2, 2025
- Merge branch 'ml-explore:main' into main
  Blaizzy
  authored
  2e09bd2
Commits on Mar 4, 2026
- Merge branch 'ml-explore:main' into main
  Blaizzy
  authored
  d59907c
Commits on Apr 2, 2026
- Merge branch 'ml-explore:main' into main
  Blaizzy
  authored
  e06ca01
Commits on Apr 24, 2026
- Merge branch 'ml-explore:main' into main
  Blaizzy
  authored
  5c965a5
- add DS4
  Blaizzy
  committed
  18c12cb
- Fix DeepSeek V4 Flash generation math
  Blaizzy
  committed
  28d991f
- Generalize safetensors dtype fallback
  Blaizzy
  committed
  be714fd
- Remove deprecated DeepSeek V4 tests for sanitization and model loading
  Blaizzy
  committed
  a833f26
- Refactor DeepSeek V4 tests to consolidate and clarify test cases
  Blaizzy
  committed
  28a83ea
- format
  Blaizzy
  committed
  bc1c285
- keep experts quantized
  Blaizzy
  committed
  f4f7b4d
- fix fp32 promotion (3 tps -> 11 tps)
  Blaizzy
  committed
  e3221c4
- Fix DeepSeek V4 batched routing
  Blaizzy
  committed
  16b268a
- compile _limited_swiglu, hc_split_sinkhorn and make robust predicate
  Blaizzy
  committed
  201267d
- Enhance quant_predicate to support new projection modes and add @mx.compile decorator to _limited_swiglu function
  Blaizzy
  committed
  f26e519
- keep experts in mxfp4
  Blaizzy
  committed
  61e006e
- Refactor quantized linear layer
  Blaizzy
  committed
  f2b4e9d
- Implement optimized Sinkhorn operations in DeepSeek V4 and add corresponding unit tests for HyperConnection
  Blaizzy
  committed
  c0d9222
- Optimize matrix multiplication in HyperConnection by replacing einsum with matmul for improved performance
  Blaizzy
  committed
  ef8c95d
- HC sinkhorn normalization
  Blaizzy
  committed
  f7ff216
- fix(deepseek_v4): numerical stability and compressed attention mask padding
  
  eauchs
  and
  machiabeli
  committed
  c6a7828
- Merge pull request #14 from eauchs/fix-blaizzy-pr1192
  Blaizzy
  authored
  f8ebcaf
Commits on Apr 25, 2026
- Refactor DeepSeek V4 scoring and compile rope full
  Blaizzy
  committed
  2a73e76
- Add DeepseekV4SwitchGLU and BatchRotatingKVCache; refactor position handling in DeepseekV4RoPE and DeepseekV4Cache
  Blaizzy
  committed
  c2c9801
- Refactor DeepseekV4Cache to improve state management and add length handling for batching
  Blaizzy
  committed
  8ac37c0
- format
  Blaizzy
  committed
  369fd37
- Revert "keep experts in mxfp4"
  Blaizzy
  committed
  166bba6
- Revert quant predicate (lower precision experts)
  Blaizzy
  committed
  cbb0b72
- Remove quantization_config from ModelArgs in deepseek_v4.py
  Blaizzy
  committed
  e3313d4
- Update deepseek_v4.py
  
  Blaizzy
  and
  kernelpool
  authored
  16de966
Commits on Apr 26, 2026
- Fix RoPE init
  Blaizzy
  committed
  01bcd7d
- Refactor RoPE initialization in V4Attention to streamline handling of compressed layers and update associated tests for consistency.
  Blaizzy
  committed
  8bd10c6
- Improve DeepseekV4Cache and V4Attention with improved trimming logic and mask handling for compressed layers
  Blaizzy
  committed
  014a10b
- Add scores parameter and updatie projection logic
  Blaizzy
  committed
  5d1a67a
- Optimize DeepSeek V4 performance
  
  0xClandestine
  and
  Blaizzy
  committed
  26f49f5
- Refactor dtype handling in _ensure_cached method and improve pooled tensor processing logic (fix cache blowing up)
  Blaizzy
  committed
  2ba4215
- format
  Blaizzy
  committed
  16f7205
- Fix matrix multiplication in _hc_expand_op to use original comb tensor instead of swapped axes
  Blaizzy
  committed
  193aa77
- Refactor output projection
  Blaizzy
  committed
  5a4aaa4
- Refactor kv tensor reshaping in V4Attention for improved dimensionality handling
  Blaizzy
  committed
  2591b51
- Remove redundant cache type check in V4Attention to streamline processing logic
  Blaizzy
  committed
  22da01a
- Fix DeepSeek V4 sparse pooled prefill memory
  
  Blaizzy
  and
  adurham
  committed
  8e8571a
- Remove sparse pooled attention test block
  Blaizzy
  committed
  ddeffe3
Commits on Apr 27, 2026
Commits on Apr 28, 2026
- Enable loading the original checkpoint
  angeloskath
  committed
  f83460f
- Simplify GLU and gate remove intermediate castings
  angeloskath
  committed
  81a8c57
- Fix RoPE to use the kernel by scaling freqs
  angeloskath
  committed
  4951496
Commits on Apr 29, 2026
- Start simplifying and speeding up the attention
  angeloskath
  committed
  3cf5282
- Attempt to fix IndexError on batching.
  pcuenca
  committed
  ad9e329
- Refactor compressor and compressed non-sparse attention
  angeloskath
  committed
  83d7e74
- Ensure dtype going into mx.fast.rope is int32
  pcuenca
  committed
  d8a47ae
- Merge pull request #21 from pcuenca/rope-dtype
  Blaizzy
  authored
  42a416b
- Merge pull request #20 from pcuenca/index-error
  Blaizzy
  authored
  85cb244
Commits on May 1, 2026
- Major cache refactor and attention simplification
  angeloskath
  committed
  2120a1b
- Simplify HyperConnection
  angeloskath
  committed
  ca8b299
- Add hyper_connection.py
  angeloskath
  committed
  c58834b
- Fix batch cache edge case
  angeloskath
  committed
  7efb57f
- Fix tensor parallel distributed
  angeloskath
  committed
  5c10538

Commits

Commits on Nov 12, 2025

Commits on Dec 2, 2025

Commits on Mar 4, 2026

Commits on Apr 2, 2026

Commits on Apr 24, 2026

Commits on Apr 25, 2026

Commits on Apr 26, 2026

Commits on Apr 27, 2026

Commits on Apr 28, 2026

Commits on Apr 29, 2026

Commits on May 1, 2026