[BugFix] Fix test breakages from transformers 4.45 upgrade

[njhill]

No description provided.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[njhill]

_get_logits_warper was rolled into _get_logits_processor

[njhill]

"mrope" gets renamed to "default" in the Qwen2-VL config class

[DarkLight1337]

Qwen2-VL cannot be run in transformers>=4.45 even with this change.

[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/cyrusleung/vllm/examples/offline_inference_vision_language_multi_image.py", line 286, in <module>
[rank0]:     main(args)
[rank0]:   File "/home/cyrusleung/vllm/examples/offline_inference_vision_language_multi_image.py", line 262, in main
[rank0]:     run_generate(model, QUESTION, IMAGE_URLS)
[rank0]:   File "/home/cyrusleung/vllm/examples/offline_inference_vision_language_multi_image.py", line 205, in run_generate
[rank0]:     req_data = model_example_map[model](question, image_urls)
[rank0]:   File "/home/cyrusleung/vllm/examples/offline_inference_vision_language_multi_image.py", line 151, in load_qwen2_vl
[rank0]:     llm = LLM(
[rank0]:   File "/home/cyrusleung/vllm/vllm/entrypoints/llm.py", line 214, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/home/cyrusleung/vllm/vllm/engine/llm_engine.py", line 564, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/cyrusleung/vllm/vllm/engine/llm_engine.py", line 325, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/home/cyrusleung/vllm/vllm/executor/executor_base.py", line 47, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/home/cyrusleung/vllm/vllm/executor/gpu_executor.py", line 40, in _init_executor
[rank0]:     self.driver_worker.load_model()
[rank0]:   File "/home/cyrusleung/vllm/vllm/worker/worker.py", line 183, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/home/cyrusleung/vllm/vllm/worker/model_runner.py", line 1016, in load_model
[rank0]:     self.model = get_model(model_config=self.model_config,
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/model_loader/__init__.py", line 19, in get_model
[rank0]:     return loader.load_model(model_config=model_config,
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/model_loader/loader.py", line 399, in load_model
[rank0]:     model = _initialize_model(model_config, self.load_config,
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/model_loader/loader.py", line 176, in _initialize_model
[rank0]:     return build_model(
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/model_loader/loader.py", line 161, in build_model
[rank0]:     return model_class(config=hf_config,
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/qwen2_vl.py", line 876, in __init__
[rank0]:     self.model = Qwen2Model(config, cache_config, quant_config)
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/qwen2.py", line 248, in __init__
[rank0]:     self.start_layer, self.end_layer, self.layers = make_layers(
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/utils.py", line 282, in make_layers
[rank0]:     [PPMissingLayer() for _ in range(start_layer)] + [
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/utils.py", line 283, in <listcomp>
[rank0]:     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/qwen2.py", line 250, in <lambda>
[rank0]:     lambda prefix: Qwen2DecoderLayer(config=config,
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/qwen2.py", line 175, in __init__
[rank0]:     self.self_attn = Qwen2Attention(
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/models/qwen2.py", line 133, in __init__
[rank0]:     self.rotary_emb = get_rope(
[rank0]:   File "/home/cyrusleung/vllm/vllm/model_executor/layers/rotary_embedding.py", line 1003, in get_rope
[rank0]:     raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
[rank0]: ValueError: Unknown RoPE scaling type default

But if we install the older version of transformers mentioned in the docs (git+https://github.com/huggingface/transformers.git@21fac7abba2a37fae86106f87fcf9974fd1e3830), then vLLM cannot be run because it imports MLlamaConfig from the top level. We'll open a separate PR to patch in Qwen2-VL support for transformers v4.45.

[njhill]

Thanks @DarkLight1337 yeah I was just making a change to update the scaling type in the config back to "mrope" if it's "default". @ywang96 found this open issue huggingface/transformers#33401

[njhill]

Change made in ce1d477

[njhill]

ValueError rather than OSError is now thrown when there's no config.json present

[ywang96]

Thanks for fixing! Didn't know we have an except check for a particular error for loading lora tokenizer.

Let's see if this can make CI go all green!

[mgoin]

These seem like reasonable changes, LGTM if it makes the build green thanks!

[njhill]

Pushed another commit to hopefully fix some additional failures.

[jeejeelee]

It seems that this failure is caused by transformers modifying the related function args. This has been reported to the official chatglm3 team.

[DarkLight1337]

I can confirm that the following tests pass locally:

• tests/distributed/test_pipeline_parallel.py
• tests/entrypoints/openai/test_serving_chat.py
• tests/models/decoder_only/language/test_granite.py
• tests/models/decoder_only/vision_language/test_llava_next_video.py
• tests/models/decoder_only/vision_language/test_llava_onevision.py
• tests/models/test_registry.py

The following tests still fail locally (but pass in CI):

• tests/samplers/test_sampler.py

_____________________________________________________________________________________ test_sampler_top_k_top_p[cuda:1-71] _____________________________________________________________________________________

seed = 71, device = 'cuda:1'

        hf_probs = processors(torch.zeros_like(fake_logits), fake_logits.clone())
        hf_probs = torch.softmax(hf_probs, dim=-1, dtype=torch.float)
>       torch.testing.assert_close(hf_probs, sample_probs, rtol=0.0, atol=1e-5)
E       AssertionError: Tensor-likes are not close!
E       
E       Mismatched elements: 2 / 5312000 (0.0%)
E       Greatest absolute difference: 0.4882833957672119 at index (61, 9638) (up to 1e-05 allowed)
E       Greatest relative difference: 1.0 at index (61, 9638) (up to 0.0 allowed)

FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:0-11] - AssertionError: Tensor-likes are not close!
FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:0-36] - AssertionError: Tensor-likes are not close!
FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:0-71] - AssertionError: Tensor-likes are not close!
FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:1-11] - AssertionError: Tensor-likes are not close!
FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:1-36] - AssertionError: Tensor-likes are not close!
FAILED tests/samplers/test_sampler.py::test_sampler_top_k_top_p[cuda:1-71] - AssertionError: Tensor-likes are not close!

[DarkLight1337]

@hongxiayang can you look into the failing AMD tests and fix them in a separate PR?

Update: I suspect it has something to do with entrypoints/openai/test_oot_registration.py. Let's see whether running that separately can fix the problem.

[DarkLight1337]

I have unblocked additional tests to check for regressions.

[DarkLight1337]

Re: LoRA 1 and LoRA 3 failures, I think we need to regenerate https://huggingface.co/yard1/llama-2-7b-sql-lora-test to have model_type in the config? seems that the error still fails to get caught somehow...

Update: We need (?) to fix the error message matching in

vllm/vllm/transformers_utils/tokenizer.py

Line 132 in 4bb98f2

or "requires you to execute the tokenizer file" in str(e)):

Update 2: This is odd. The test passes locally without having to update the error message matching...

Update 3: Never mind, I was looking at the wrong logs.

[njhill]

@DarkLight1337 looks like test_basic_distributed_correctness.py is still called in the tests but was removed in #7820

[DarkLight1337]

@DarkLight1337 looks like test_basic_distributed_correctness.py is still called in the tests but was removed in #7820

I'll update this PR once the other tests finish.

[njhill]

Re: LoRA 1 and LoRA 3 failures, I think we need to regenerate https://huggingface.co/yard1/llama-2-7b-sql-lora-test to have model_type in the config? seems that the error still fails to get caught somehow...

Update: We need (?) to fix the error message matching in

vllm/vllm/transformers_utils/tokenizer.py

Line 132 in 4bb98f2

or "requires you to execute the tokenizer file" in str(e)):

Update 2: This is odd. The test passes locally without having to update the error message matching...

The CI failure looks potentially like an incompatibility of the custom chatglm3-6b tokenizer and the latest transformers version.

[DarkLight1337]

Re: LoRA 1 and LoRA 3 failures, I think we need to regenerate https://huggingface.co/yard1/llama-2-7b-sql-lora-test to have model_type in the config? seems that the error still fails to get caught somehow...

Update: We need (?) to fix the error message matching in

vllm/vllm/transformers_utils/tokenizer.py

Line 132 in 4bb98f2

or "requires you to execute the tokenizer file" in str(e)):

Update 2: This is odd. The test passes locally without having to update the error message matching...

Oops, I just realized I was looking at the old logs before your fix. Only LoRA 4 fails now.

[ywang96]

The CI failure looks potentially like an incompatibility of the custom chatglm3-6b tokenizer and the latest transformers version.

IMO we should just skip this test since @jeejeelee mentioned it's also reported to the chatglm team

[DarkLight1337]

The PR should be good to go now, hope that the tests pass this time. (LoRA Long Context has been failing for a while, so it is probably unrelated to transformers upgrade. We can fix that later)

[njhill]

Everything has passed apart from llava OOM still :(

[ywang96]

Everything has passed apart from llava OOM still :(

Let's see if this final try works - if not I'll just remove this test because it is pretty memory intensive (7B model with 4 images on a L4 gpu)

[vrdn-23]

Re: LoRA 1 and LoRA 3 failures, I think we need to regenerate https://huggingface.co/yard1/llama-2-7b-sql-lora-test to have model_type in the config? seems that the error still fails to get caught somehow...

Update: We need (?) to fix the error message matching in

vllm/vllm/transformers_utils/tokenizer.py

Line 132 in 4bb98f2

or "requires you to execute the tokenizer file" in str(e)):

Update 2: This is odd. The test passes locally without having to update the error message matching...

Update 3: Never mind, I was looking at the wrong logs.

@DarkLight1337 @njhill I'm seeing the same error while loading LoRA adapters on the latest version of 0.6.2. It seems that the changes Nick made have not been reflected in the latest pip version release? Is there a post patch fix incoming with these changes soon?

[DarkLight1337]

Re: LoRA 1 and LoRA 3 failures, I think we need to regenerate https://huggingface.co/yard1/llama-2-7b-sql-lora-test to have model_type in the config? seems that the error still fails to get caught somehow...
Update: We need (?) to fix the error message matching in

vllm/vllm/transformers_utils/tokenizer.py

Line 132 in 4bb98f2

or "requires you to execute the tokenizer file" in str(e)):

Update 2: This is odd. The test passes locally without having to update the error message matching...
Update 3: Never mind, I was looking at the wrong logs.

@DarkLight1337 @njhill I'm seeing the same error while loading LoRA adapters on the latest version of 0.6.2. It seems that the changes Nick made have not been reflected in the latest pip version release? Is there a post patch fix incoming with these changes soon?

Yes, this PR was merged after v0.6.2. I'm not sure when the next patch will be dropped.