VLLM 0.6.6.post1: There is no module or parameter named 'multi_modal_projector.cross_attn.layer_norm_kv' in AriaForConditionalGeneration

[dcrockwell]

Using vllm version 0.6.6.post1, there appears to be an issue with a missing parameter multi_modal_projector.cross_attn.layer_norm_kv in AriaForConditionalGeneration

Below are the logs from the attempt:

INFO 01-23 16:15:25 config.py:2272] Downcasting torch.float32 to torch.bfloat16.
INFO 01-23 16:15:31 config.py:510] This model supports multiple tasks: {'score', 'classify', 'embed', 'generate', 'reward'}. Defaulting to 'generate'.
INFO 01-23 16:15:31 config.py:1310] Defaulting to use mp for distributed inference
WARNING 01-23 16:15:31 arg_utils.py:1113] The model has a long context length (65536). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 01-23 16:15:31 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post1) with config: model='rhymes-ai/Aria', speculative_config=None, tokenizer='rhymes-ai/Aria', skip_tokenizer_init=False, tokenizer_mode=slow, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65536, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=rhymes-ai/Aria, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False, 
WARNING 01-23 16:15:31 tokenizer.py:174] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
WARNING 01-23 16:15:31 multiproc_worker_utils.py:312] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 01-23 16:15:31 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 01-23 16:15:32 selector.py:120] Using Flash Attention backend.
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:36 selector.py:120] Using Flash Attention backend.
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:36 multiproc_worker_utils.py:222] Worker ready; awaiting tasks
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:36 selector.py:120] Using Flash Attention backend.
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:36 multiproc_worker_utils.py:222] Worker ready; awaiting tasks
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:36 selector.py:120] Using Flash Attention backend.
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:36 multiproc_worker_utils.py:222] Worker ready; awaiting tasks
INFO 01-23 16:15:37 utils.py:918] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:37 utils.py:918] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:37 utils.py:918] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:37 utils.py:918] Found nccl from library libnccl.so.2
INFO 01-23 16:15:37 pynccl.py:69] vLLM is using nccl==2.22.3
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:37 pynccl.py:69] vLLM is using nccl==2.22.3
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:37 pynccl.py:69] vLLM is using nccl==2.22.3
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:37 pynccl.py:69] vLLM is using nccl==2.22.3
(VllmWorkerProcess pid=9457) WARNING 01-23 16:15:37 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=9458) WARNING 01-23 16:15:37 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=9459) WARNING 01-23 16:15:37 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 01-23 16:15:37 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
INFO 01-23 16:15:37 shm_broadcast.py:255] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_85a2fc8d'), local_subscribe_port=46095, remote_subscribe_port=None)
INFO 01-23 16:15:37 model_runner.py:1094] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:37 model_runner.py:1094] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:37 model_runner.py:1094] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:37 model_runner.py:1094] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:37 selector.py:249] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:37 selector.py:249] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:37 selector.py:249] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:37 selector.py:129] Using XFormers backend.
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:37 selector.py:129] Using XFormers backend.
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:37 selector.py:129] Using XFormers backend.
INFO 01-23 16:15:37 selector.py:249] Cannot use FlashAttention-2 backend for head size 72.
INFO 01-23 16:15:37 selector.py:129] Using XFormers backend.
(VllmWorkerProcess pid=9458) INFO 01-23 16:15:38 weight_utils.py:251] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=9459) INFO 01-23 16:15:38 weight_utils.py:251] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=9457) INFO 01-23 16:15:38 weight_utils.py:251] Using model weights format ['*.safetensors']
INFO 01-23 16:15:38 weight_utils.py:251] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/12 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   8% Completed | 1/12 [00:01<00:19,  1.74s/it]
Loading safetensors checkpoint shards:  17% Completed | 2/12 [00:03<00:16,  1.64s/it]
Loading safetensors checkpoint shards:  25% Completed | 3/12 [00:05<00:15,  1.73s/it]
Loading safetensors checkpoint shards:  33% Completed | 4/12 [00:05<00:10,  1.35s/it]
Loading safetensors checkpoint shards:  42% Completed | 5/12 [00:07<00:10,  1.51s/it]
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] Exception in worker VllmWorkerProcess while processing method load_model.
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] Traceback (most recent call last):
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 230, in _run_worker_process
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 155, in load_model
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     self.model_runner.load_model()
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1096, in load_model
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     return loader.load_model(vllm_config=vllm_config)
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 366, in load_model
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     loaded_weights = model.load_weights(
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 676, in load_weights
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 237, in load_weights
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 226, in _load_module
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     raise ValueError(msg)
(VllmWorkerProcess pid=9458) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] ValueError: There is no module or parameter named 'multi_modal_projector.cross_attn.layer_norm_kv' in AriaForConditionalGeneration
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] Exception in worker VllmWorkerProcess while processing method load_model.
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] Traceback (most recent call last):
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 230, in _run_worker_process
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 155, in load_model
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     self.model_runner.load_model()
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1096, in load_model
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     return loader.load_model(vllm_config=vllm_config)
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 366, in load_model
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     loaded_weights = model.load_weights(
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 676, in load_weights
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 237, in load_weights
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 226, in _load_module
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236]     raise ValueError(msg)
(VllmWorkerProcess pid=9457) ERROR 01-23 16:15:47 multiproc_worker_utils.py:236] ValueError: There is no module or parameter named 'multi_modal_projector.cross_attn.layer_norm_kv' in AriaForConditionalGeneration
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236] Exception in worker VllmWorkerProcess while processing method load_model.
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236] Traceback (most recent call last):
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 230, in _run_worker_process
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]              ^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 155, in load_model
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     self.model_runner.load_model()
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1096, in load_model
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     self.model = get_model(vllm_config=self.vllm_config)
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     return loader.load_model(vllm_config=vllm_config)
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 366, in load_model
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     loaded_weights = model.load_weights(
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]                      ^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 676, in load_weights
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 237, in load_weights
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 226, in _load_module
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236]     raise ValueError(msg)
(VllmWorkerProcess pid=9459) ERROR 01-23 16:15:48 multiproc_worker_utils.py:236] ValueError: There is no module or parameter named 'multi_modal_projector.cross_attn.layer_norm_kv' in AriaForConditionalGeneration
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/server.py", line 58, in <module>
[rank0]:     main()
[rank0]:   File "/home/ubuntu/server.py", line 7, in main
[rank0]:     llm = LLM(
[rank0]:           ^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/utils.py", line 986, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 230, in __init__
[rank0]:     self.llm_engine = self.engine_class.from_engine_args(
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 517, in from_engine_args
[rank0]:     engine = cls(
[rank0]:              ^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 273, in __init__
[rank0]:     self.model_executor = executor_class(vllm_config=vllm_config, )
[rank0]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 36, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 83, in _init_executor
[rank0]:     self._run_workers("load_model",
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 157, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 155, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1096, in load_model
[rank0]:     self.model = get_model(vllm_config=self.vllm_config)
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
[rank0]:     return loader.load_model(vllm_config=vllm_config)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 366, in load_model
[rank0]:     loaded_weights = model.load_weights(
[rank0]:                      ^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 676, in load_weights
[rank0]:     loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 237, in load_weights
[rank0]:     autoloaded_weights = set(self._load_module("", self.module, weights))
[rank0]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
[rank0]:     yield from self._load_module(prefix,
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 198, in _load_module
[rank0]:     yield from self._load_module(prefix,
[rank0]:   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 226, in _load_module
[rank0]:     raise ValueError(msg)
[rank0]: ValueError: There is no module or parameter named 'multi_modal_projector.cross_attn.layer_norm_kv' in AriaForConditionalGeneration
ERROR 01-23 16:15:48 multiproc_worker_utils.py:123] Worker VllmWorkerProcess pid 9459 died, exit code: -15
INFO 01-23 16:15:48 multiproc_worker_utils.py:127] Killing local vLLM worker processes
Loading safetensors checkpoint shards:  42% Completed | 5/12 [00:10<00:14,  2.01s/it]

[rank0]:[W123 16:15:49.575131960 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[xffxff]

Can you try to install the latest code of vLLM? This bug has been fixed by vllm-project/vllm#12203 and vllm-project/vllm#12309

[dcrockwell]

This seems to solve the previous problem, but now it's complaining The size of tensor a (2048) must match the size of tensor b (832) at non-singleton dimension 1:

$ vllm serve rhymes-ai/Aria --dtype auto --api-key token-abc123 --tensor-parallel-size 4
INFO 01-24 20:33:35 __init__.py:183] Automatically detected platform cuda.
INFO 01-24 20:33:36 api_server.py:786] vLLM API server version 0.6.6.post2.dev371+gab5bbf5a
INFO 01-24 20:33:36 api_server.py:787] args: Namespace(subparser='serve', model_tag='rhymes-ai/Aria', config='', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key='token-abc123', lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='rhymes-ai/Aria', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x75a4524a5c60>)
INFO 01-24 20:33:36 api_server.py:201] Started engine process with PID 2214
INFO 01-24 20:33:37 config.py:2318] Downcasting torch.float32 to torch.float16.
INFO 01-24 20:33:39 __init__.py:183] Automatically detected platform cuda.
INFO 01-24 20:33:41 config.py:2318] Downcasting torch.float32 to torch.float16.
INFO 01-24 20:33:45 config.py:520] This model supports multiple tasks: {'classify', 'score', 'embed', 'reward', 'generate'}. Defaulting to 'generate'.
INFO 01-24 20:33:45 config.py:1332] Defaulting to use mp for distributed inference
WARNING 01-24 20:33:45 arg_utils.py:1121] The model has a long context length (65536). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 01-24 20:33:48 config.py:520] This model supports multiple tasks: {'reward', 'generate', 'embed', 'classify', 'score'}. Defaulting to 'generate'.
INFO 01-24 20:33:48 config.py:1332] Defaulting to use mp for distributed inference
WARNING 01-24 20:33:48 arg_utils.py:1121] The model has a long context length (65536). This may cause OOM errors during the initial memory profiling phase, or result in low performance due to small KV cache space. Consider setting --max-model-len to a smaller value.
INFO 01-24 20:33:48 llm_engine.py:232] Initializing an LLM engine (v0.6.6.post2.dev371+gab5bbf5a) with config: model='rhymes-ai/Aria', speculative_config=None, tokenizer='rhymes-ai/Aria', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=65536, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=rhymes-ai/Aria, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
WARNING 01-24 20:33:50 multiproc_worker_utils.py:298] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 01-24 20:33:50 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 01-24 20:33:51 cuda.py:225] Using Flash Attention backend.
INFO 01-24 20:33:53 __init__.py:183] Automatically detected platform cuda.
INFO 01-24 20:33:53 __init__.py:183] Automatically detected platform cuda.
INFO 01-24 20:33:53 __init__.py:183] Automatically detected platform cuda.
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:54 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:54 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:54 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:55 cuda.py:225] Using Flash Attention backend.
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:55 cuda.py:225] Using Flash Attention backend.
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:55 cuda.py:225] Using Flash Attention backend.
INFO 01-24 20:33:57 utils.py:938] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:57 utils.py:938] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:57 utils.py:938] Found nccl from library libnccl.so.2
INFO 01-24 20:33:57 pynccl.py:67] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:57 utils.py:938] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:57 pynccl.py:67] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:57 pynccl.py:67] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:57 pynccl.py:67] vLLM is using nccl==2.21.5
(VllmWorkerProcess pid=2453) WARNING 01-24 20:33:57 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=2454) WARNING 01-24 20:33:57 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
WARNING 01-24 20:33:57 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorkerProcess pid=2452) WARNING 01-24 20:33:57 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
INFO 01-24 20:33:57 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_faa25b54'), local_subscribe_port=48849, remote_subscribe_port=None)
INFO 01-24 20:33:57 model_runner.py:1110] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:57 model_runner.py:1110] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:57 model_runner.py:1110] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:57 model_runner.py:1110] Starting to load model rhymes-ai/Aria...
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:57 cuda.py:209] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:57 cuda.py:222] Using XFormers backend.
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:57 cuda.py:209] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:57 cuda.py:222] Using XFormers backend.
INFO 01-24 20:33:57 cuda.py:209] Cannot use FlashAttention-2 backend for head size 72.
INFO 01-24 20:33:57 cuda.py:222] Using XFormers backend.
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:57 cuda.py:209] Cannot use FlashAttention-2 backend for head size 72.
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:57 cuda.py:222] Using XFormers backend.
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:58 config.py:2928] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:58 config.py:2928] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:58 config.py:2928] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 01-24 20:33:58 config.py:2928] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
(VllmWorkerProcess pid=2454) INFO 01-24 20:33:58 weight_utils.py:251] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2452) INFO 01-24 20:33:58 weight_utils.py:251] Using model weights format ['*.safetensors']
INFO 01-24 20:33:58 weight_utils.py:251] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2453) INFO 01-24 20:33:58 weight_utils.py:251] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/12 [00:00<?, ?it/s]
ERROR 01-24 20:34:06 engine.py:387] The size of tensor a (2048) must match the size of tensor b (832) at non-singleton dimension 1
ERROR 01-24 20:34:06 engine.py:387] Traceback (most recent call last):
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
ERROR 01-24 20:34:06 engine.py:387]     engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 01-24 20:34:06 engine.py:387]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
ERROR 01-24 20:34:06 engine.py:387]     return cls(ipc_path=ipc_path,
ERROR 01-24 20:34:06 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 73, in __init__
ERROR 01-24 20:34:06 engine.py:387]     self.engine = LLMEngine(*args, **kwargs)
ERROR 01-24 20:34:06 engine.py:387]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 271, in __init__
ERROR 01-24 20:34:06 engine.py:387]     self.model_executor = executor_class(vllm_config=vllm_config, )
ERROR 01-24 20:34:06 engine.py:387]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 260, in __init__
ERROR 01-24 20:34:06 engine.py:387]     super().__init__(*args, **kwargs)
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 49, in __init__
ERROR 01-24 20:34:06 engine.py:387]     self._init_executor()
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 123, in _init_executor
ERROR 01-24 20:34:06 engine.py:387]     self._run_workers("load_model",
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 183, in _run_workers
ERROR 01-24 20:34:06 engine.py:387]     driver_worker_output = run_method(self.driver_worker, sent_method,
ERROR 01-24 20:34:06 engine.py:387]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/utils.py", line 2208, in run_method
ERROR 01-24 20:34:06 engine.py:387]     return func(*args, **kwargs)
ERROR 01-24 20:34:06 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 182, in load_model
ERROR 01-24 20:34:06 engine.py:387]     self.model_runner.load_model()
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
ERROR 01-24 20:34:06 engine.py:387]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 01-24 20:34:06 engine.py:387]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
ERROR 01-24 20:34:06 engine.py:387]     return loader.load_model(vllm_config=vllm_config)
ERROR 01-24 20:34:06 engine.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 379, in load_model
ERROR 01-24 20:34:06 engine.py:387]     loaded_weights = model.load_weights(
ERROR 01-24 20:34:06 engine.py:387]                      ^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 657, in load_weights
ERROR 01-24 20:34:06 engine.py:387]     loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 233, in load_weights
ERROR 01-24 20:34:06 engine.py:387]     autoloaded_weights = set(self._load_module("", self.module, weights))
ERROR 01-24 20:34:06 engine.py:387]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 194, in _load_module
ERROR 01-24 20:34:06 engine.py:387]     yield from self._load_module(prefix,
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 171, in _load_module
ERROR 01-24 20:34:06 engine.py:387]     loaded_params = module_load_weights(weights)
ERROR 01-24 20:34:06 engine.py:387]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 364, in load_weights
ERROR 01-24 20:34:06 engine.py:387]     weight_loader(param, loaded_weight, shard_id)
ERROR 01-24 20:34:06 engine.py:387]   File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 213, in weight_loader
ERROR 01-24 20:34:06 engine.py:387]     param.data.copy_(up_and_gate)
ERROR 01-24 20:34:06 engine.py:387] RuntimeError: The size of tensor a (2048) must match the size of tensor b (832) at non-singleton dimension 1
ERROR 01-24 20:34:07 multiproc_worker_utils.py:122] Worker VllmWorkerProcess pid 2452 died, exit code: -15
INFO 01-24 20:34:07 multiproc_worker_utils.py:126] Killing local vLLM worker processes
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 389, in run_mp_engine
    raise e
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
    return cls(ipc_path=ipc_path,
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/multiprocessing/engine.py", line 73, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 271, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config, )
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 260, in __init__
    super().__init__(*args, **kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 49, in __init__
    self._init_executor()
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 123, in _init_executor
    self._run_workers("load_model",
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/executor/mp_distributed_executor.py", line 183, in _run_workers
    driver_worker_output = run_method(self.driver_worker, sent_method,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/utils.py", line 2208, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/worker.py", line 182, in load_model
    self.model_runner.load_model()
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
    self.model = get_model(vllm_config=self.vllm_config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model
    return loader.load_model(vllm_config=vllm_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/model_loader/loader.py", line 379, in load_model
    loaded_weights = model.load_weights(
                     ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 657, in load_weights
    loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 233, in load_weights
    autoloaded_weights = set(self._load_module("", self.module, weights))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 194, in _load_module
    yield from self._load_module(prefix,
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 171, in _load_module
    loaded_params = module_load_weights(weights)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 364, in load_weights
    weight_loader(param, loaded_weight, shard_id)
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/model_executor/models/aria.py", line 213, in weight_loader
    param.data.copy_(up_and_gate)
RuntimeError: The size of tensor a (2048) must match the size of tensor b (832) at non-singleton dimension 1
Loading safetensors checkpoint shards:   0% Completed | 0/12 [00:08<?, ?it/s]

[rank0]:[W124 20:34:08.048430892 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/uvloop/__init__.py", line 105, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 814, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/conda/envs/pytorch/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 131, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/conda/envs/pytorch/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/pytorch/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 225, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/conda/envs/pytorch/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[dcrockwell]

@xffxff Is this something I'm doing wrong, or is this a known issue in Aria?