Skip to content

[ROCm][Perf][DSv4.1] Score adaptive-verification decode steps as varlen sequences in the MXFP4 indexer - #66128

Open
maslan-coop26 wants to merge 7 commits into
vllm-project:mainfrom
maslan-coop26:rocm-dsv41-indexer-varlen-decode
Open

maslan-coop26 wants to merge 7 commits into
vllm-project:mainfrom
maslan-coop26:rocm-dsv41-indexer-varlen-decode

Conversation

@maslan-coop26

@maslan-coop26 maslan-coop26 commented Nov 6, 2026 •

Copy link
Copy Markdown

Overview

Under adaptive verification, the paged MXFP4 indexer (#58671) scores every decode row of DeepSeek-V4.1 as its own sequence, because its requests carry 1 to next_n query rows and the native path needs next_n each. AITER's paged MQA-logits kernel already takes varlen sequences (query_start_loc, as the prefill chunks use), so this plans those decode steps as their requests: a workgroup walks a request's KV tiles once for all its rows.

Claims

  • On MI355X, the dense decode indexer launches of DeepSeek-V4.1-Flash under adaptive verification take 1.4-2.4x less time: 22.1 -> 15.8 us to 58.3 -> 24.5 us for a ratio-2 layer and 56.9 -> 31.2 us to 118.3 -> 68.6 us for the candidate source, at 77-183 rows.
  • The selections are the same as the per-row launch's (the order of a row's top-k is not fixed).
  • About 58 us less per decode step on DP8 + EP8 at an agentic concurrency of 256, which a single end-to-end run cannot resolve (see Validation).

Validation

Kernel times in nightly 60bd6dd2 (vLLM main of 2026-11-16, AITER v0.1.29) with this branch merged with main, MI355X, HIP-graph replay, ratio-2 layer and the ratio-1 candidate source (with its candidate blocks), at the per-rank decode shapes of DeepSeek-V4.1-Flash DP8 + EP8 with DSpark K6 (mean ~6 rows a request):

Rows Ratio-2 layer, per row -> varlen (us) Candidate source, per row -> varlen (us)
77 22.1 -> 15.8 56.9 -> 31.2
134 48.5 -> 23.2 104.5 -> 64.7
183 58.3 -> 24.5 118.3 -> 68.6

Replaying the CUDA graph's own launch shape (48 request slots, mostly empty, the token count rounded up to the graph size) keeps the gain: 22 -> 16, 48 -> 32 and 59 -> 30 us for the ratio-2 layer. Building the varlen schedule costs 2-7 us of GPU time a step. A per-step timing run of the server put the saving at about 58 us per decode step, 0.3% of the step.

End to end (DeepSeek-V4.1-Flash, DP8 + EP8 with MoRI, DSpark K6 with adaptive verification, an agentic workload at concurrency 256, 8 x MI355X, on an earlier nightly before #67377, which changes the dense MXFP8 GEMMs and not the indexer), one A/B moved median TPOT from 5.336 to 5.357 ms (+0.4%), inside the ±0.5% spread of single runs on that setup, so it neither shows nor rules out the 0.3%.

In that A/B the two arms' per-position draft acceptance (each server's spec-decode stats over the whole run, about 28M drafted tokens each) and agentic accuracy match, so the varlen launches select the same entries on real adaptive steps:

Launch 1 2 3 4 5 6 Mean acceptance length Accuracy
Per row 0.837 0.659 0.490 0.349 0.242 0.159 3.735 0.534
Varlen 0.837 0.660 0.491 0.350 0.244 0.160 3.743 0.538

Tests, on MI355X, in nightly 60bd6dd2 with this branch merged with main:

pytest tests/kernels/attention/test_rocm_paged_mxfp4_indexer.py tests/v1/attention/test_rocm_paged_mxfp4_indexer_plan.py
75 passed

test_varlen_decode_layers_match_reference runs ragged steps ([2, 1, 2], [6, 1, 3, 6], and [6, 3, 1, 0] with an empty request slot) through the varlen launches against the torch reference, for a dense layer, the candidate source and the consumers both ways, and checks that they select the same entries as the per-row launches they replace. test_decode_row_shard_keeps_per_row_launch_on_varlen_steps builds TP4 decode steps under adaptive verification: 128 rows are sharded as on main (rank 1 gets rows 32-64, no varlen), and 32 rows stay whole and launch as varlen.

Checks vs main

Nightly 60bd6dd2 (vLLM main of 2026-11-16, AITER v0.1.29) on MI355X; main is the stock nightly, this PR is the branch's two code files over it.

One rank's indexer work for a decode step (the 8 index layers, CUDA graph, 1-7 rows a request), main -> this PR, us:

32 rows 64 112 224 448
TP1, 32K 203 -> 217 297 -> 269 410 -> 371 702 -> 615 1291 -> 1088
TP1, 128K 387 -> 323 619 -> 433 898 -> 587 1686 -> 980 3243 -> 1813

At TP >= 2, steps of 64+ rows keep #66047's shard and run main's code; smaller steps take varlen as at TP1.

Serving, 8K in / 1K out at concurrency 64, DSpark K6 with adaptive verification; two 4-GPU servers at once, one per arm, GPU halves swapped between launches; 4 paired rounds:

main this PR PR vs main, per pair
TP4 + EP4 12,367 tok/s, AL 5.63 12,443 tok/s, AL 5.61 -1.5% to +2.7%
DP4 + EP4 (TP1) 7,387 tok/s, AL 5.73 7,626 tok/s, AL 5.72 -5.9% to +21%

The step saving (about 0.3%) is below this noise; the runs show no regression.

RULER NIAH (lm-eval 0.4.13, the 8 niah tasks, greedy, 64 concurrent, TP4 + EP4), mean over tasks; every miss is niah_multivalue:

32K (n=400) 32K warm repeat 128K (n=200) 128K warm repeat
main 0.9988 0.9981 0.9925 0.9900
this PR 0.9994 0.9988 0.9938 0.9925
Commands
# TP4 + EP4 server (#66047's flags). DP4 + EP4: --tensor-parallel-size 1 --data-parallel-size 4
# --engram-config '{"cpu_offload":true,"embedding_across_dp":true}' and VLLM_ENGINE_READY_TIMEOUT_S=3600.
# The RULER servers ran without VLLM_NO_SWA_BOUNDED_REPLAY.
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=True \
VLLM_USE_BREAKABLE_CUDAGRAPH=1 VLLM_USE_RUST_FRONTEND=1 VLLM_NO_SWA_BOUNDED_REPLAY=1 \
AITER_TRITON_LOG_LEVEL=ERROR OMP_NUM_THREADS=1 \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 --enable-expert-parallel --moe-backend aiter \
  --max-model-len 1048576 --block-size 128 --max-num-batched-tokens 8192 --max-num-seqs 256 \
  --max-cudagraph-capture-size 1024 --gpu-memory-utilization 0.9 --language-model-only \
  --tokenizer-mode deepseek_v41 --tool-call-parser deepseek_v41 --reasoning-parser deepseek_v41 --enable-auto-tool-choice \
  --attention-config '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' --engram-config '{"cpu_offload":false}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'

# Serving: a 256-prompt warmup and one more round not counted, then two rounds per launch of
vllm bench serve --model deepseek-ai/DeepSeek-V4.1-Flash --dataset-name random --random-input-len 8192 \
  --random-output-len 1024 --ignore-eos --num-prompts 512 --max-concurrency 64 --temperature 0 --seed 0

# RULER NIAH at 32K; 128K: "max_seq_lengths":[131072] and --limit 25. The warm repeat sends the same prompts again.
lm_eval --model local-completions \
  --model_args model=deepseek-ai/DeepSeek-V4.1-Flash,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=64,tokenizer=<model dir>,tokenized_requests=False \
  --tasks niah_single_1,niah_single_2,niah_single_3,niah_multikey_1,niah_multikey_2,niah_multikey_3,niah_multiquery,niah_multivalue \
  --limit 50 --gen_kwargs temperature=0 --metadata '{"max_seq_lengths":[32768]}'

Details

  • RocmMxfp4IndexerMetadataBuilder adds decode_varlen (RocmMxfp4VarlenDecode: query_start_loc, per-request compressed context and block table) on every decode step under adaptive verification that [ROCm][Perf][DSv4.1] Shard the sparse-indexer decode rows across TP #66047's row shard leaves whole, uniform or not. Request slots past the step's requests have no rows (query_start_loc repeats the token count there).
  • build_rocm_mxfp4_decode_schedule and the dense decode launch pass those sequences to AITER with next_n set to the most rows a request can have, so a full request is one row block. The candidate consumers' gather is unchanged.
  • It is on whenever adaptive verification is, with no new flag: it only changes how AITER's kernel walks the same rows. Without adaptive verification, decode steps keep the native next_n-row path.
  • Everything stays on the device (slices of the step's query_start_loc, context lengths and block table), so there is no added host sync.
  • With [ROCm][Perf][DSv4.1] Shard the sparse-indexer decode rows across TP #66047's TP row sharding, a step the shard cuts (64+ decode rows at TP >= 2, no prefill) keeps main's per-row launch on the rank's rows, since a shard would cut a varlen step's requests apart. Varlen runs on the steps the shard leaves whole: TP1, fewer than 64 rows, and steps with prefill.
  • CI: like the rest of test_rocm_paged_mxfp4_indexer.py, the new kernel tests need CDNA4, so the MI300 attention-kernels step skips them. The optional :amd: (MI355 DPX) Attention Kernels step runs this file on gfx950 and triggers on this diff, but has to be unblocked on the PR; the tests were run on MI355X as above.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

…en sequences in the MXFP4 indexer

Under adaptive verification a request carries 1 to next_n query rows, so the
paged MXFP4 indexer's dense decode launches leave the next_n-row native path
and score every row as its own sequence, walking its context once per row.
AITER's paged MQA-logits kernel takes varlen sequences (query_start_loc), so
the builder now plans such steps as their requests and the dense decode
launches pass them, and a workgroup walks a request's KV tiles once for all
its rows. A request slot past the step's requests has no rows. The selections
are the same as the per-row launch's.

Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@mergify mergify Bot added deepseek Related to DeepSeek models rocm Related to AMD ROCm labels Nov 6, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 6, 2026
Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
@maslan-coop26
maslan-coop26 marked this pull request as ready for review November 10, 2026 01:39

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@whx-sjtu whx-sjtu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall lgtm. A few minor issues.

)
elif varlen is not None:
# The same for requests of 1 to next_n rows, as varlen sequences.
logger.info_once(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger.info_once here runs on every dense indexer layer of every eager/piecewise step (a scope check plus an lru_cache lookup each call), and the message says DeepSeek V4.1 although RocmMxfp4IndexerMetadataBuilder also serves DeepSeek-V4. Whether a step goes varlen is fixed by config (varlen_next_n), so could this move to the builder's __init__, where adaptive is decided? That also drops the new init_logger import and module logger here.

if self.varlen_next_n > 1:
    logger.info_once(
        "ROCm MXFP4 indexer: adaptive-verification decode steps launch as "
        "varlen sequences of up to %d rows a request",
        self.varlen_next_n,
    )

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b41f352: it is logged once from the builder's __init__ when varlen_next_n > 1, worded for any model, and the ops file no longer has a logger.

if metadata.decode_native is None and self.varlen_next_n > 1:
metadata.decode_varlen = RocmMxfp4VarlenDecode(
self.varlen_next_n,
cm.query_start_loc[: base.num_decodes + 1],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under adaptive verification the device query_start_loc read here can differ from the host copy (drafts are trimmed on device), which is the case this path exists for. test_varlen_decode_layers_match_reference builds RocmMxfp4VarlenDecode by hand, so this wiring is only exercised end to end, and the A/B in the description reports TPOT only. Could you add the per-position draft acceptance rate (or a gsm8k score) for both arms of that run, to show the selections match on real adaptive steps?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-position draft acceptance for the two runs of that A/B, from each server's spec-decode stats over the whole run (about 28M drafted tokens each), with the agentic accuracy score:

Launch 1 2 3 4 5 6 Mean acceptance length Accuracy
Per row 0.837 0.659 0.490 0.349 0.242 0.159 3.735 0.534
Varlen 0.837 0.660 0.491 0.350 0.244 0.160 3.743 0.538

They match to within 0.002 at every position, and accuracy is within the spread of single runs. I added this to the description.

self.native_next_n,
context_lens,
)
if metadata.decode_native is None and self.varlen_next_n > 1:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: metadata.decode_native is None is always true here. With adaptive verification native_next_n is 1, so native_decode returns None; without it varlen_next_n is 1. The condition is just self.varlen_next_n > 1, and every adaptive decode step takes varlen, uniform or not, so "Steps with uniform next_n rows keep the native path" in the description holds only without adaptive verification. Likewise max(self.native_next_n, self.varlen_next_n) in __init__ is always self.num_speculative_tokens + 1.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, thanks. The condition is now self.varlen_next_n > 1, the schedule buffer uses num_speculative_tokens + 1, and the description now says every decode step takes the varlen path under adaptive verification, and the native next_n-row path without it.

…ata builder

The dense decode launch logged it on every call. The builder decides it once from the config, so it logs there, worded for any model on this builder. Its condition reduces to varlen_next_n > 1, and the schedule buffer's row count to num_speculative_tokens + 1.

Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
)
from vllm.v1.attention.backends.mla.rocm_paged_mxfp4_indexer import (
RocmMxfp4IndexerMetadata,
RocmMxfp4VarlenDecode,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR collides with #66047. If both land, DecodeRowShard.narrow slices the rows, block table and native, but it never touches the new decode_varlen. That breaks at TP≥2 with adaptive verification and ≥64 decode rows: _dense_decode would get a rank's slice of q and logits along with the whole step's query_start_loc. The result is silent out-of-bounds logits writes and scoring of the wrong rows, and no AITER assert catches it. The two PRs also conflict textually in the test file and the backend. The fix belongs in whichever lands second. The simplest option is to set decode_varlen = None when a shard is planned; the other is to shard on request boundaries and rebase the query_start_loc slice.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dc4a110, on top of a merge of main. Varlen steps (every decode step under adaptive verification) now keep every row on every rank, as they did before #66047: a shard is a range of whole next_n-row requests, which would cut a varlen step's requests apart. Steps without adaptive verification still shard. test_decode_row_shard_skips_varlen_steps builds a TP4 step of 128 rows and checks that it isn't sharded and that its varlen launch covers every row; it fails without the guard. Varlen-aware sharding can follow if TP≥2 with adaptive verification needs it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: 0bb625d changes this. Steps the row shard cuts now keep main's per-row launch on the rank's rows, and varlen runs only on steps the shard leaves whole (TP1, under 64 rows, steps with prefill). Details in tuukkjs's thread.

[[2, 1, 2], [6, 1, 3, 6], [6, 3, 1, 0]],
ids=["ragged2", "ragged6", "empty_slot"],
)
def test_varlen_decode_layers_match_reference(monkeypatch, block, query_lens):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says this test only runs off-CI because the MI300 attention-kernels step skips it. But the optional ":amd: (MI355 DPX) Attention Kernels Shard %N" step (.buildkite/test-amd.yaml:4562) triggers on this diff and runs kernels/attention on gfx950, so it would run the new varlen cases. Could you get that step unblocked on this PR so the test has a CI record, and mention it in the description?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted in the description: the optional MI355 Attention Kernels step triggers on this diff but has to be unblocked on the PR. On MI355X this branch passes test_rocm_paged_mxfp4_indexer.py (57, including the varlen cases) and test_rocm_paged_mxfp4_indexer_plan.py (18).

pool."""
decode_native: RocmMxfp4NativeDecode | None = None
"""The same rows as next_n-row sequences, for the dense launches."""
decode_varlen: RocmMxfp4VarlenDecode | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

decode_varlen is set on every adaptive-verification decode step, uniform or not (line 470), so "where the requests' row counts differ" describes a condition the code doesn't check. The same goes for the unchanged comment at line 377 ("Uniform steps also launch the dense layers on next_n-row sequences"), which now holds only without adaptive verification.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dc4a110: the decode_varlen docstring and the builder's comment now say every decode step takes the varlen path under adaptive verification, and that the native next_n-row path serves only steps without it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: 0bb625d changes this again. decode_varlen is now set only on adaptive-verification decode steps that the row shard leaves whole, and the docstring says so.

@mergify

mergify Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maslan-coop26.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Nov 12, 2026
Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
With vllm-project#66047 in main, a TP>=2 decode step of 64+ rows is split into
contiguous ranges of whole next_n-row requests, one per rank. Under adaptive
verification every decode step launches its requests as varlen sequences of
1 to next_n rows, which such a range can cut apart, and DecodeRowShard.narrow
does not touch decode_varlen, so a rank would score its slice of q and logits
against the whole step's query_start_loc. Varlen steps now keep every row on
every rank, as they did before vllm-project#66047; steps without adaptive verification
still shard.

test_decode_row_shard_skips_varlen_steps builds such a step at TP4 on 128
rows and checks that it is not sharded and that its varlen launch covers
every row; the kernel test helper follows the same rule. The decode_varlen
docstring and the builder's comment now say the varlen path covers every
decode step under adaptive verification, and that the native next_n-row
path serves only steps without it.

Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
@mergify mergify Bot removed the needs-rebase label Nov 13, 2026
Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>

@tuukkjs tuukkjs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

  • The PR checklist asks performance PRs for an end-to-end serving result with exact commands and run-to-run spread; do you have repeated A/B runs vs main (TP1 and TP4) to add, since the single one is within noise? Or do you think we don't need e2e validation for this PR?
  • Every adaptive decode step now goes through the kernel's varlen path, and the new tests stop at 1,200-token contexts; could you add RULER NIAH at 32K and 128K vs main, as #66047 did?

Comment on lines +516 to +521
# A shard is a contiguous range of whole next_n-row requests, which
# would cut apart a varlen step's requests of 1 to next_n rows, so
# those steps keep every row on every rank.
if self.varlen_next_n <= 1 and not step_has_prefill(
cm, self.decode_threshold
):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me with adaptive verification at TP≥2, this skips #66047's row shard on steps of 64+ decode rows, and the FULL-graph consumers don't use varlen. Have you checked that this doesn't regress TP4 against main?

@maslan-coop26 maslan-coop26 Nov 18, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, it did regress TP4: on the steps #66047 shards, every rank scored every row, 30-805 us a step slower than main's slice plus its all-gathers (64-448 rows, 32K-128K). Fixed in 0bb625d: those steps keep main's per-row path, and varlen only runs on steps the shard leaves whole.

…cation decode steps

A step the TP row shard cuts keeps the per-row launch on the rank's rows, as on main; varlen runs only on decode steps the shard leaves whole (TP1, under 64 rows, steps with prefill). With every row on every rank, a TP4 rank's indexer work for those steps took 30-805 us a step more than main's slice plus its all-gathers.

Signed-off-by: Mehmet Aslan <286132399+maslan-coop26@users.noreply.github.com>
@maslan-coop26

Copy link
Copy Markdown
Author

@tuukkjs thanks. Fixed the TP4 shard issue (inline). Repeated A/Bs and RULER vs main are in the description under "Checks vs main":

  • TP4 + EP4 serving: +0.6% (pairs -1.5% to +2.7%), AL unchanged
  • DP4 + EP4 serving: within noise, AL unchanged
  • RULER NIAH 32K / 128K: unchanged (0.9994 vs 0.9988, 0.9938 vs 0.9925)

The step saving is about 0.3%, too small for a serving run to show, so these runs check for regressions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

4 participants