Inference tutorial - Part 3 of e2e series [WIP] #2343
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2343
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit bd2600f with merge base 2898903 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jainapurva
added
the
topic: documentation
docs/source/inference.rst
Outdated
|
|
||
| vllm serve pytorch/Phi-4-mini-instruct-float8dq --tokenizer microsoft/Phi-4-mini-instruct -O3 | ||
|
|
||
| Inference with vLLM |
There was a problem hiding this comment.
should we move this after Inference with Transformers
There was a problem hiding this comment.
cc @jainapurva I think if vLLM is our recommended serving solution, this should go before transformers.
docs/source/inference.rst
Outdated
|
|
||
| vLLM automatically leverages torchao's optimized kernels when serving quantized models, providing significant throughput improvements. | ||
|
|
||
| Setting up vLLM with Quantized Models |
There was a problem hiding this comment.
nit: this doesn't have to be a new section I think
|
Hi @jainapurva, by the way I'm adding a 
|
jainapurva
force-pushed
the
inference_tutorial
branch
from
b93b892 to
ce675b8
Compare
docs/source/inference.rst
Outdated
| .. note:: | ||
| For more information on supported quantization and sparsity configurations, see `HF-Torchao Docs <https://huggingface.co/docs/transformers/main/en/quantization/torchao>`_. | ||
|
|
||
| Inference with vLLM |
There was a problem hiding this comment.
for this section, can you replace with https://huggingface.co/pytorch/Qwen3-8B-int4wo-hqq#inference-with-vllm
it might be easier to do command line compared to code
docs/source/serving.rst
Outdated
| print(f"Output: {generated_text!r}") | ||
| print("-" * 60) | ||
|
|
||
| [Optional] Inference with Transformers |
There was a problem hiding this comment.
We should have an Inference w/ SGlang section
There was a problem hiding this comment.
I tested the integration of TorchAO and SGLang, came across a lot of issues in running the server. As discussed with @jerryzh168 offline, we can add this later, after more thorough testing and updates.
No description provided.