[Frontend] Support chat_template_kwargs in LLM.chat

[DarkLight1337]

This PR enables disabling thinking mode per request in offline inference by adding chat_template_kwargs argument.

Note: The keyword argument for disabling thinking mode depends on the chat template. For example, Granite 3.2 uses thinking while Qwen 3 uses enable_thinking.

Example using Qwen 3:

from vllm import LLM, SamplingParams

llm = LLM("Qwen/Qwen3-8B")
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.8,
    top_k=20,
    max_tokens=8192,
    presence_penalty=1.5,
)

outputs = llm.chat(
    [{"role": "user", "content": "Give me a short introduction to large language models."}],
    sampling_params=sampling_params,
    chat_template_kwargs={"enable_thinking": False},  # or True
)

print("-" * 50)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)
    print("-" * 50)

Resolve this #17327 (comment)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

@fyabc would be great if you could get your team to add this to the Qwen docs!

[ad1192214879]

Can you support chat_template_kwargs also in LLM.generate?

[DarkLight1337]

Can you support chat_template_kwargs also in LLM.generate?

LLM.generate doesn't use chat template so it doesn't make sense to support it there

[ad1192214879]

Can you support chat_template_kwargs also in LLM.generate?

LLM.generate doesn't use chat template so it doesn't make sense to support it there
Because we need to use LLM.generate ，we can't use no-thinking mode when using offline batched inference？

[DarkLight1337]

LLM.chat can also be used in offline inference, so I'm not really sure why you couldn't just use that.

[rangehow]

Can you support chat_template_kwargs also in LLM.generate?

LLM.generate doesn't use chat template so it doesn't make sense to support it there
Because we need to use LLM.generate ，we can't use no-thinking mode when using offline batched inference？

在官方的示例上简单修改了一下，大概就是下面这样

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

if __name__ == '__main__':
# Sample prompts.
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]

    
    sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=32768)
    

    tokenizer = AutoTokenizer.from_pretrained(model_dir)
    messages_list=[]
    for prompt in prompts:
        messages = [
            {"role": "user", "content": prompt}
        ]
        messages_list.append(messages)
    et = False
    text = tokenizer.apply_chat_template(
        messages_list,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking= et# Switches between thinking and non-thinking modes. Default is True.
    )
    
    llm = LLM(model = ,tensor_parallel_size=4,enable_prefix_caching=True)
    # Generate texts from the prompts. The output is a list of RequestOutput objects
    # that contain the prompt, generated text, and other information.
    outputs = llm.generate(prompts, sampling_params)
    # Print the outputs.
    print("\nGenerated Outputs:\n" + "-" * 60)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt:    {prompt!r}")
        print(f"Output:    {generated_text!r}")
        print("-" * 60)

[fyabc]

@fyabc would be great if you could get your team to add this to the Qwen docs!

Hi @DarkLight1337, thank you for your contribution!
Since vllm-0.8.5 does not include this PR, I think we can update doc when the new version is released?

[DarkLight1337]

@fyabc would be great if you could get your team to add this to the Qwen docs!

Hi @DarkLight1337, thank you for your contribution! Since vllm-0.8.5 does not include this PR, I think we can update doc when the new version is released?

Sounds good. @simon-mo are we planning on another release soon once the bugfixes for Qwen3 are in?

[simon-mo]

Yes v0.9.0 release milestone is still open. We just merged the torch upgrade. I'm looking at next Monday as a good checkpoint to release maybe?

[RonanKMcGovern]

@DarkLight1337 so there's no straightforward approach then to disable thinking via the chat completions endpoint? Thanks

(I'm also confused why --enable-thinking needs to be passed for thinking, if omitting it doesn't disable thinking. Unless I've misunderstood. I did test without the flag and still saw thinking.)

[DarkLight1337]

You can already disable it when calling the endpoint, even without this PR. See https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes

[fyabc]

@fyabc would be great if you could get your team to add this to the Qwen docs!

Hi @DarkLight1337, thank you for your contribution! Since vllm-0.8.5 does not include this PR, I think we can update doc when the new version is released?

@DarkLight1337 Qwen doc already updated the latest llm.chat usage.
https://qwen.readthedocs.io/en/latest/deployment/vllm.html#python-library