Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
46eebc9
test(deepseek-v41): establish pinned checkpoint and numerical oracles…
valarLip Sep 12, 2026
f80f4ee
feat(deepseek-v41): add configuration and CSA2 topology (P01)
valarLip Sep 12, 2026
e6cc5bb
feat(deepseek-v41): load native weights and add quantized linear cont…
valarLip Sep 12, 2026
c2b7944
feat(deepseek-v41): implement mHC MoE and Engram math contracts (P03)
valarLip Sep 12, 2026
be63b4f
feat(deepseek-v41): make CSA2 index tie breaking configurable
valarLip Sep 12, 2026
afa8519
fix(attention): expose BF16 sparse softmax tile selection
valarLip Sep 12, 2026
446d5a0
feat(deepseek-v41): assemble eager text backbone and CSA2 state
valarLip Sep 12, 2026
a019d73
fix(deepseek-v41): reuse existing V4 BF16 attention
valarLip Sep 12, 2026
ce26bb1
feat(deepseek-v41): reuse V4 inverse RoPE
valarLip Sep 13, 2026
f9f8d41
fix(deepseek-v41): preserve empty inverse RoPE batches
valarLip Sep 13, 2026
70fbd2f
fix(blockscale): preserve group32 accumulation and E8M0 extremes
valarLip Sep 13, 2026
ff95c8e
fix(deepseek-v41): preserve normalization precision boundaries
valarLip Sep 13, 2026
6e9062f
fix(deepseek-v41): select reliable offline RCCL collectives
valarLip Sep 13, 2026
b3eb0c4
perf(deepseek-v41): project only the requested logit suffix
valarLip Sep 13, 2026
a4ff0fd
test(deepseek-v41): add independent checkpoint quality validation
valarLip Sep 13, 2026
75d0c59
docs(deepseek-v41): record norm regression and WKV precision diagnosis
valarLip Sep 13, 2026
83c8520
feat(deepseek-v41): integrate paged text runtime and request state
valarLip Sep 13, 2026
f66f0e6
perf(deepseek-v41): reuse A8W4 MoE and add packed cache graphs
valarLip Sep 13, 2026
6ba271f
docs(deepseek-v41): record AITER GSM8K regression
valarLip Sep 13, 2026
8b04076
feat(deepseek-v41): enable chat tools and numeric reasoning effort
valarLip Sep 13, 2026
49d195f
feat(deepseek-v41): enable native image requests and vision scope
valarLip Sep 13, 2026
b1e2743
feat(mesh): register DeepSeek V4.1 DSML dialect
valarLip Sep 13, 2026
9e0bdf2
feat(deepseek-v41): enable chunked image prefill and vision leases
valarLip Sep 14, 2026
41011da
feat(deepseek-v41): native DSpark, and the shared V4 paths it exposed
valarLip Sep 15, 2026
c7a33d8
fix(deepseek-v41): preserve model-owned online MoE quantization
valarLip Sep 15, 2026
33f954f
perf(deepseek-v41): reuse shared V4 normalization and projection kernels
valarLip Sep 16, 2026
52dea8b
refactor(deepseek-v41): reuse V4 request metadata staging
valarLip Sep 16, 2026
96e4fb7
feat(deepseek-v41): share V4's draft graph and LM head, and the asser…
valarLip Sep 16, 2026
19d0e96
refactor(deepseek-v41): compress the whole batch with V4's kernel, no…
valarLip Sep 16, 2026
6bdba28
feat(deepseek-v41): page the index plane, and score decode out of it
valarLip Sep 16, 2026
006fa47
feat(deepseek-v41): make a decode step two graph replays, not a hundred
valarLip Sep 17, 2026
cfdf915
perf(deepseek-v41): build a verify step's candidate cursors on the host
valarLip Sep 17, 2026
71df0e0
deepseek-v41: keep the text config working on transformers >= 5.13
ZhangLirong-amd Sep 18, 2026
668b60f
blockscale: choose split_k by grid occupancy, not by a width threshold
ZhangLirong-amd Sep 18, 2026
204a9c1
engram: read the n-gram tables with a device kernel over UVA
ZhangLirong-amd Sep 18, 2026
3094e70
refactor(deepseek-v41): one index scorer, one plane format, one cache
valarLip Sep 18, 2026
01a44a2
fix(deepseek-v41): score a long context in bands, and write the index…
valarLip Sep 18, 2026
28248e8
perf(deepseek-v41): write the draft's context KV in two kernels, not …
ZhangLirong-amd Sep 18, 2026
e934cd8
refactor(deepseek-v41): one GEMM per shared input, and ATOM's layers …
valarLip Sep 19, 2026
d1f7d26
perf(deepseek-v41): take the per-forward pointwise work off the criti…
valarLip Sep 19, 2026
24e2a55
Merge origin/main into feat/deepseek-v41-flash
valarLip Sep 19, 2026
7c5ee88
test: stop the suite from importing atom twice, and repair what noticed
valarLip Sep 19, 2026
c80516c
feat(deepseek-v41): hash Engram's n-gram rows on the device
valarLip Sep 20, 2026
94b3c8d
perf(deepseek-v41): take Engram's per-step host arithmetic off the cr…
valarLip Sep 20, 2026
8d5ad12
perf(deepseek-v41): build the draft block's prologue in three kernels…
ZhangLirong-amd Sep 20, 2026
9b79e7c
perf(deepseek-v41): gather the draft's windows as rows, not through t…
ZhangLirong-amd Sep 20, 2026
13145d2
perf(deepseek-v41): put the FP8 GEMM on the microscaling matrix core
valarLip Sep 20, 2026
40b5bfa
perf(deepseek-v41): two launches where the attention prologue had four
valarLip Sep 20, 2026
4b44404
refactor(deepseek-v41): delete what never ran, and let the guides des…
valarLip Sep 20, 2026
fc1584d
fix(tests): collect the whole suite without AITER, and admit TP-only …
valarLip Sep 20, 2026
78b6f3d
fix(engram): agree on the UVA fallback across ranks, and stop leaking…
valarLip Sep 20, 2026
22a5c43
Merge origin/main into feat/deepseek-v41-flash
valarLip Sep 20, 2026
01855ed
perf(engram): fuse post-WKV gating and residual update
ZhangLirong-amd Sep 21, 2026
3fa486f
perf(engram): overlap lookup and TP gather with early layers
ZhangLirong-amd Sep 21, 2026
f2bb0ff
fix(tests): guard the GPU-only modules before they import Triton, not…
valarLip Sep 21, 2026
b3cb26c
refactor(v41): unify runtime operations under torch compile guards
ZhangLirong-amd Sep 21, 2026
cf7e9ea
refactor(engram): put the package's import boundary in its directory
valarLip Sep 21, 2026
72d6b4a
style(v41): spell a test's kwargs as a literal, which is what ruff as…
valarLip Sep 21, 2026
f5cf70c
perf(v41): reuse block tables and batch indptr scans
ZhangLirong-amd Sep 21, 2026
4b0ca2e
style(v41): separate local standard-library imports for Ruff 0.16
ZhangLirong-amd Sep 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,4 @@ Key entry points:
| Debugging | `/debug-guide` |
| Adding a model | `/add-model` |
| MiniMax-H3 (video+audio diffusion) | `recipes/MiniMax-H3.md` |
| DeepSeek-V4.1-Flash | `recipes/DeepSeek-V4.1-Flash.md` |
130 changes: 90 additions & 40 deletions atom/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
from transformers import AutoConfig, GenerationConfig, PretrainedConfig

# plugin-related utilities
from atom.model_ops.attentions.pool_layout.v4_pool_fields import (
MQA_LOGITS_PRESHUFFLE_ROWS,
)
from atom.plugin import is_plugin_mode, is_vllm
from atom.plugin.config import PluginConfig
from atom.quant_spec import (
Expand Down Expand Up @@ -548,10 +551,11 @@ def remap_layer_name(
self,
hf_config: PretrainedConfig,
packed_modules_mapping: dict | None = None,
weights_mapper={},
weights_mapper: dict | None = None,
quant_exclude_name_mapping: dict[str, str] | None = None,
):
model_type = hf_config.model_type
weights_mapper = weights_mapper or {}
self.packed_modules_mapping = (
packed_modules_mapping if packed_modules_mapping is not None else {}
)
Expand All @@ -569,9 +573,10 @@ def remap_layer_name(
"gate_proj": ("gate_up_proj", 0),
"up_proj": ("gate_up_proj", 1),
}
elif model_type == "qwen3_moe" or model_type == "qwen3_next":
if getattr(hf_config, "mlp_only_layers", []):
self.packed_modules_mapping["gate_up_proj"] = ["gate_proj", "up_proj"]
elif model_type in ("qwen3_moe", "qwen3_next") and getattr(
hf_config, "mlp_only_layers", []
):
self.packed_modules_mapping["gate_up_proj"] = ["gate_proj", "up_proj"]

if weights_mapper:
self.exclude_layers = [
Expand Down Expand Up @@ -627,27 +632,17 @@ def _remap_layer_name(name: str) -> list[str]:
self.apply_exclude_name_mapping(quant_exclude_name_mapping)


# Rows per block that `deepgemm_fp8_paged_mqa_logits` requires to stay in its
# preshuffled layout, which is the only layout it computes correctly -- with
# `Preshuffle=False` it disagrees with the flat `fp8_mqa_logits` kernel by ~100%
# at every block size, and aiter's assert guards only the preshuffle side. Any
# cache that kernel pages over must therefore hold a multiple of this many rows
# per block. Sizing constants belong to whoever enforces them, and the block
# size is set here.
_MQA_LOGITS_PRESHUFFLE_ROWS = 16


def glm5_kpool_block_size(index_kpool: int) -> int:
"""Tokens per KV block that lets GLM-5.3's pooled index cache be exact.

A block of B tokens needs ``B // index_kpool`` index rows, and that count
must be a multiple of `_MQA_LOGITS_PRESHUFFLE_ROWS`. The smallest B that
must be a multiple of `MQA_LOGITS_PRESHUFFLE_ROWS`. The smallest B that
satisfies it is the product, and the smallest is what we want: a larger
block only adds paging waste, while a smaller one forces the cache to be
padded back up to one row per token -- which is the whole cost being
removed here.
"""
return index_kpool * _MQA_LOGITS_PRESHUFFLE_ROWS
return index_kpool * MQA_LOGITS_PRESHUFFLE_ROWS


def _glm5_next_unsupported_features(config: "Config") -> list[str]:
Expand Down Expand Up @@ -710,6 +705,10 @@ def get_hf_config(model: str, trust_remote_code: bool = False) -> PretrainedConf
model,
)
model_type = config_dict.get("model_type")
if model_type == "deepseek_v41":
from atom.models.deepseek_v41.config import normalize_hf_config

return normalize_hf_config(config_dict)

def _get_hf_token() -> str | None:
token = os.getenv("HF_TOKEN")
Expand Down Expand Up @@ -803,7 +802,7 @@ def _get_hf_token() -> str | None:
hf_config = AutoConfig.from_pretrained(
model, trust_remote_code=trust_remote_code
)
except ValueError as e:
except ValueError:
# For the unsupported model in current transformers, try vllm if in plugin mode
if is_vllm():
from vllm.transformers_utils.config import get_config
Expand All @@ -814,7 +813,7 @@ def _get_hf_token() -> str | None:
hf_config = get_config(model, trust_remote_code=trust_remote_code)
hf_config = maybe_patch_hf_config_from_gguf(model, hf_config)
else:
raise e
raise
return hf_config


Expand Down Expand Up @@ -851,9 +850,11 @@ def _normalize_minimax_m3_text_config(hf_config: PretrainedConfig) -> None:
if text_config is None or text_config is hf_config:
return

if getattr(text_config, "hidden_act", None) == "swigluoai":
if getattr(text_config, "swiglu_beta", None) is None:
text_config.swiglu_beta = 1.0
if (
getattr(text_config, "hidden_act", None) == "swigluoai"
and getattr(text_config, "swiglu_beta", None) is None
):
text_config.swiglu_beta = 1.0

for attr_name in (
"use_index_cache",
Expand Down Expand Up @@ -1340,6 +1341,10 @@ def hf_config_override(
# config fields. Route it to the DSpark draft model and skip the MTP
# n_predict=1 rewrite (DSpark uses dspark_block_size, not n_predict).
if getattr(hf_config, "dspark_block_size", None):
if hf_config.model_type in ("deepseek_v41", "deepseek_v41_text"):
hf_config.model_type = "deepseek_v41_dspark"
hf_config.architectures = ["DeepseekV41DSparkModel"]
return
hf_config.model_type = "deepseek_v4_dspark"
hf_config.architectures = ["DeepseekV4DSparkModel"]
logger.info(
Expand Down Expand Up @@ -1457,15 +1462,29 @@ class DSparkConfig:
buckets to capture for the ragged path (e.g. "1,3,6" or "8").
- q_buckets: CUDA-graph query-length buckets for the (older) batch-uniform
q-bucket verify path (independent of the ragged path).
- disable_sps_calib: skip SPS calibration (replays captured graphs at
warmup); fall back to the synthetic SPS stub.
- disable_sps_calib: skip automatic SPS calibration (replays captured
graphs at warmup). Without an explicit profile, use the synthetic stub.
- calibration_profile: JSON file containing model/runtime-bound SPS and
STS measurements. Takes precedence over automatic calibration.
"""

confidence_schedule: bool = False
ragged: bool = False
ragged_graph_sizes: str = ""
q_buckets: str = ""
disable_sps_calib: bool = False
# Explicit offline measurements; compatibility is checked before use.
calibration_profile: str | None = None

def __post_init__(self):
if self.calibration_profile is not None and (
not isinstance(self.calibration_profile, str)
or not self.calibration_profile
or not self.confidence_schedule
):
raise ValueError(
"DSpark calibration_profile requires a path and confidence_schedule=True"
)

@classmethod
def from_dict(cls, cfg: dict | None) -> "DSparkConfig":
Expand Down Expand Up @@ -2087,11 +2106,11 @@ def __post_init__(self):
self.hf_config.rope_parameters = rope_params

self.generation_config = get_generation_config(self.model)
if self.generation_config is not None:
if (
eos_ids := getattr(self.generation_config, "eos_token_id", None)
) is not None:
self.stop_token_ids = [eos_ids] if isinstance(eos_ids, int) else eos_ids
if self.generation_config is not None and (
(eos_ids := getattr(self.generation_config, "eos_token_id", None))
is not None
):
self.stop_token_ids = [eos_ids] if isinstance(eos_ids, int) else eos_ids
self.quant_config = QuantizationConfig(
self.hf_config,
self.online_quant_config,
Expand Down Expand Up @@ -2145,11 +2164,15 @@ def __post_init__(self):
or (self.plugin_config is not None and self.plugin_config.is_vllm)
) and self.compilation_config.level == CompilationLevel.PIECEWISE:
self.compilation_config.set_splitting_ops_for_v1()
# Keep an explicit cudagraph_mode (e.g. FULL); default to
# PIECEWISE only when unset. splitting_ops/sizes are set either
# way so the model is still piece-split-compiled at level 3.
# Default to PIECEWISE, except V4.1 whose Engram stream fork/join
# currently requires a FULL graph. Explicit choices are preserved.
# splitting_ops/sizes still configure the level-3 backend.
if self.compilation_config.cudagraph_mode is None:
self.compilation_config.cudagraph_mode = CUDAGraphMode.PIECEWISE
self.compilation_config.cudagraph_mode = (
CUDAGraphMode.FULL
if self.hf_config.model_type == "deepseek_v41_text"
else CUDAGraphMode.PIECEWISE
)
self.compilation_config.init_with_cudagraph_sizes()

self.torch_dtype = (
Expand Down Expand Up @@ -2231,7 +2254,10 @@ def __post_init__(self):
# Use the preserved `architectures` field (re-injected by get_hf_config,
# line 567) which keeps the original "DeepseekV4ForCausalLM[NextN]" name.
arches = getattr(self.hf_config, "architectures", None) or []
is_deepseek_v4 = any("DeepseekV4" in str(a) for a in arches)
is_deepseek_v4 = any(
str(a).startswith(("DeepseekV4For", "DeepseekV4MTP", "DeepseekV4DSpark"))
for a in arches
)
if is_deepseek_v4:
v4_block_size = 256
if self.kv_cache_block_size != v4_block_size:
Expand Down Expand Up @@ -2331,15 +2357,39 @@ def __post_init__(self):
# not yet describe the separate FP4 scale pool, so those integrations
# retain FP8 until their layouts support it. Every other model keeps
# the historical KV-cache-dtype default.
if self.index_cache_dtype is None and is_deepseek_v4:
if self.plugin_config is None and not self.kv_transfer_config:
from aiter.jit.utils.chip_info import get_gfx
if self.index_cache_dtype is None:
if is_deepseek_v4:
if self.plugin_config is None and not self.kv_transfer_config:
from aiter.jit.utils.chip_info import get_gfx

self.index_cache_dtype = "fp8" if get_gfx() == "gfx942" else "fp4"
else:
self.index_cache_dtype = "fp8" if get_gfx() == "gfx942" else "fp4"
else:
self.index_cache_dtype = "fp8"
elif self.hf_config.model_type == "deepseek_v41_text":
# CSA2 reads its index plane with the paged scorer and has no
# other, so the plane's format does not follow the main pool's.
# Only the default: an explicit value is carried through to the
# V4.1 runtime gate, which refuses anything else before weights
# load. Overriding it here would accept the flag and ignore it.
self.index_cache_dtype = "fp8"
elif self.index_cache_dtype is None:
self.index_cache_dtype = self.kv_cache_dtype
else:
self.index_cache_dtype = self.kv_cache_dtype

if self.hf_config.model_type == "deepseek_v41_text":
# CSA2's index plane is paged by the same scorer GLM's is, so a
# PAGE again has to hold a whole number of `MQA_LOGITS_PRESHUFFLE_ROWS`
# tiles -- and its ratio-2 owners halve the PAGE before that count
# is taken, which makes the floor twice the tile. 256 rather than
# that floor, and for the reason DeepSeek-V4 takes it: the block
# table is `[max_num_seqs, max_model_len / block_size]` int32 held
# twice and copied to the device every forward, 8 MB here against
# 134 MB at the 16 this field defaults to. What it costs is prefix
# reuse shorter than a PAGE, which stops matching at all.
self.kv_cache_block_size = 256

from atom.models.deepseek_v41.config import validate_runtime_config

validate_runtime_config(self)

def compute_hash(self) -> str:
"""
Expand Down
40 changes: 29 additions & 11 deletions atom/entrypoints/openai/api_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from atom.model_engine.request import RequestOutput
from atom.model_engine.sequence import new_token_ids
from atom.multimodal.processing import prepare_multimodal_inputs
from atom.multimodal.registry import get_native_multimodal_processor
from atom.utils import envs
from atom.utils.arg_parser import FlexibleArgumentParser
from atom.utils.gc_utils import (
Expand Down Expand Up @@ -650,7 +651,13 @@ def _get_multimodal_processor():
global processor
if processor is None:
logger.info(f"Loading multimodal processor from {model_name}...")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
processor = get_native_multimodal_processor(
_get_engine_config(), tokenizer, custom_message_encoder
)
if processor is None:
processor = AutoProcessor.from_pretrained(
model_name, trust_remote_code=True
)
return processor


Expand Down Expand Up @@ -1070,7 +1077,7 @@ def do_preprocess():
try:
engine.core_mgr.abort_request(_seq.id)
except Exception:
pass
logger.debug("abort_request failed on teardown", exc_info=True)
engine.io_processor.requests.pop(_seq.id, None)

finished_at = time.time()
Expand Down Expand Up @@ -1209,7 +1216,7 @@ def cleanup_stream(seq_id: int, aborted: bool = False) -> None:
try:
engine.core_mgr.abort_request(seq_id)
except Exception:
pass
logger.debug("abort_request failed on teardown", exc_info=True)
engine.io_processor.requests.pop(seq_id, None)


Expand Down Expand Up @@ -1519,7 +1526,7 @@ async def value_error_handler(request: Request, exc: ValueError):

@app.exception_handler(Exception)
async def general_error_handler(request: Request, exc: Exception):
logger.error(f"Unhandled error: {exc}", exc_info=True)
logger.error("Unhandled error: %s", exc, exc_info=exc)
return JSONResponse(
status_code=500,
content={
Expand Down Expand Up @@ -1614,7 +1621,7 @@ async def chat_completions(request: ChatCompletionRequest, raw_request: Request)
prompt = apply_chat_template(
tokenizer,
custom_message_encoder,
[msg.to_template_dict() for msg in messages],
messages,
tools=request.tools,
**merged_kwargs,
)
Expand Down Expand Up @@ -1794,8 +1801,8 @@ async def chat_completions(request: ChatCompletionRequest, raw_request: Request)
logger.error(f"Validation error in chat_completions: {e}")
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Error in chat_completions: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
logger.exception("Error in chat_completions")
raise HTTPException(status_code=500, detail=str(e)) from e


@app.post("/v1/completions")
Expand Down Expand Up @@ -1914,8 +1921,8 @@ async def completions(request: CompletionRequest, raw_request: Request):
logger.error(f"Validation error in completions: {e}")
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Error in completions: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
logger.exception("Error in completions")
raise HTTPException(status_code=500, detail=str(e)) from e


@app.post("/v1/messages")
Expand Down Expand Up @@ -1972,7 +1979,7 @@ async def anthropic_messages(request: AnthropicMessagesRequest, raw_request: Req
prompt = apply_chat_template(
tokenizer,
custom_message_encoder,
[msg.to_template_dict() for msg in messages],
messages,
tools=anthropic_to_openai_tools(request.tools),
**merged_kwargs,
)
Expand Down Expand Up @@ -2017,6 +2024,7 @@ async def anthropic_messages(request: AnthropicMessagesRequest, raw_request: Req
max_ctx = int(_v)
break
except Exception:
logger.debug("max_ctx candidate rejected", exc_info=True)
continue
if not max_ctx:
max_ctx = 30720
Expand Down Expand Up @@ -2256,6 +2264,14 @@ async def generate_anthropic_stream():
except _ClientDisconnected:
# Client hung up; seq already aborted + popped. Nothing to return.
return JSONResponse(status_code=499, content={"detail": "client disconnected"})
except ValueError as exc:
return JSONResponse(
status_code=400,
content={
"type": "error",
"error": {"type": "invalid_request_error", "message": str(exc)},
},
)
except Exception as e:
logger.exception("Error in anthropic_messages")
return JSONResponse(
Expand Down Expand Up @@ -2349,7 +2365,7 @@ async def responses_create(raw_request: Request):
prompt = apply_chat_template(
tokenizer,
custom_message_encoder,
[msg.to_template_dict() for msg in messages],
messages,
tools=openai_tools,
**merged_kwargs,
)
Expand Down Expand Up @@ -2537,6 +2553,8 @@ async def generate_responses_stream():

except _ClientDisconnected:
return JSONResponse(status_code=499, content={"detail": "client disconnected"})
except ValueError as exc:
return await value_error_handler(raw_request, exc)
except Exception as e:
logger.exception("Error in responses_create")
return JSONResponse(
Expand Down
Loading
Loading