Skip to content

[Bugfix] Remove Coincidentally-Correct Tests - #54107

Open
morrison-turnansky wants to merge 14 commits into
vllm-project:mainfrom
morrison-turnansky:issue-48038-correctness-audit
Open

[Bugfix] Remove Coincidentally-Correct Tests #54107
morrison-turnansky wants to merge 14 commits into
vllm-project:mainfrom
morrison-turnansky:issue-48038-correctness-audit

Conversation

@morrison-turnansky

@morrison-turnansky morrison-turnansky commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes #48038

See gist for the audit. It breaks down each effected test and the high level change for that test.

Test Plan

These are test side only changes.

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

@mergify mergify Bot added multi-modality Related to multi-modality (#4194) mistral Related to Mistral models qwen Related to Qwen models kimi nvidia structured-output labels Aug 27, 2026
@mergify

mergify Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @morrison-turnansky.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Aug 27, 2026
@morrison-turnansky morrison-turnansky changed the title Issue 48038 correctness audit Remove Coincidentally-Correct Tests Aug 27, 2026
@arpera arpera moved this from Backlog to Not related in Structured Output (arpera) Aug 31, 2026
@morrison-turnansky
morrison-turnansky force-pushed the issue-48038-correctness-audit branch from 1233238 to c26d4c5 Compare August 31, 2026 14:52
@morrison-turnansky morrison-turnansky changed the title Remove Coincidentally-Correct Tests [Bugfix] Remove Coincidentally-Correct Tests Aug 31, 2026
@mergify mergify Bot added bug Something isn't working and removed needs-rebase labels Aug 31, 2026
@morrison-turnansky
morrison-turnansky marked this pull request as ready for review August 31, 2026 15:08
@mergify

mergify Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @morrison-turnansky.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 26, 2026
@morrison-turnansky
morrison-turnansky force-pushed the issue-48038-correctness-audit branch from 45f6c5b to 1cd3ab9 Compare October 5, 2026 15:52
@mergify mergify Bot removed the needs-rebase label Oct 5, 2026
@morrison-turnansky
updated tests/basic_correctness
2cc9a5d 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/compile
9fcf4f4 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/distrubuted
9fdf260 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/entrypoints
59df2f3 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated lora
eef30b0 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated model_executor
a872f2f 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/models
8451a20 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/plugin_tests
63503e5 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/samplers
74fd80e 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/vl
346d729 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
linting
e2c8507 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
update cuda graph mode differences
29ec0b3 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
updated tests/basic_correctness
035f447 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@morrison-turnansky
morrison-turnansky force-pushed the issue-48038-correctness-audit branch from 1cd3ab9 to 79caca3 Compare October 5, 2026 16:22
@morrison-turnansky
morrison-turnansky force-pushed the issue-48038-correctness-audit branch from 79caca3 to 035f447 Compare October 5, 2026 16:29
@morrison-turnansky
linting
0abc526 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
@mergify

mergify Bot commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @morrison-turnansky.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@jeejeelee jeejeelee Awaiting requested review from jeejeelee jeejeelee is a code owner
@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner
@njhill njhill Awaiting requested review from njhill njhill is a code owner
@heheda12345 heheda12345 Awaiting requested review from heheda12345 heheda12345 is a code owner
@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC is a code owner
@orozery orozery Awaiting requested review from orozery orozery is a code owner
@ivanium ivanium Awaiting requested review from ivanium ivanium is a code owner
@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner
@russellb russellb Awaiting requested review from russellb
@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner
@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner
@AndreasKaratzas AndreasKaratzas Awaiting requested review from AndreasKaratzas AndreasKaratzas is a code owner
@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche is a code owner
@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner
@zou3519 zou3519 Awaiting requested review from zou3519

At least 1 approving review is required to merge this pull request.

Assignees

@sangstar sangstar

Labels

bug Something isn't working kimi mistral Related to Mistral models multi-modality Related to multi-modality (#4194) needs-rebase nvidia qwen Related to Qwen models speculative-decoding structured-output torch.compile

Projects

NVIDIA
Status: No status
Structured Output
Status: No status
torch.compile integration
Status: To triage

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[RFC]: PyTorch vLLM correctness audit

4 participants

@morrison-turnansky @zou3519 @sangstar @arpera