Repository navigation
docs: add MiniCPM-V 4.7 cookbook #43829
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
zijiexia
merged 14 commits into
sgl-project:main
from
JustinTong0323:xinyuan/minicpmv47-cookbook
Oct 13, 2026
+882
−2
Merged
Changes from 6 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
1280930
docs: add MiniCPM-V 4.7 cookbook
JustinTong0323 c06f819
docs: verify MiniCPM-V 4.7 FP8/INT4 DSPARK recipes
JustinTong0323 60cd457
docs: update MiniCPM-V 4.7 DSPARK accuracy and accept lengths
JustinTong0323 50a2209
docs: add MiniCPM-V 4.7 parser flags, patch image, reasoning and tool…
JustinTong0323 a93f920
docs: single-GPU MiniCPM-V 4.7 recipes with H200/H100 cells and measu…
JustinTong0323 0981027
docs: point MiniCPM-V 4.7 cookbook at the public checkpoints
JustinTong0323 8b84f59
docs: refine MiniCPM-V 4.7 DSPARK and quantization guidance, scope --…
JustinTong0323 4e1964b
docs: load MiniCPM-V 4.7 FP8/INT4 tokenizer from the base repo
JustinTong0323 c312568
docs: drop --trust-remote-code from MiniCPM-V 4.7 FP8/INT4 recipes
JustinTong0323 d1f2ef9
docs: MiniCPM-V 4.7 cookbook probe inputs, matched DSPARK comparison,…
JustinTong0323 58e8689
docs: MiniCPM-V 4.7 DSPARK guidance from matched GB300 and H200 runs
JustinTong0323 148a2c2
docs: MiniCPM-V 4.7 DSPARK off/on TPOT table for GB300 and H200
JustinTong0323 af1d920
docs: MiniCPM-V 4.7 DSPARK table as JSX markup
JustinTong0323 7bef804
docs: MiniCPM-V 4.7 note on reasoning-parser output for content-only …
JustinTong0323 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,330 @@ | ||
| --- | ||
| title: MiniCPM-V 4.7 | ||
| description: "Deploy OpenBMB MiniCPM-V 4.7 with SGLang — 1B dense and 35B-A3B Mixture-of-Experts vision-language models on a hybrid Gated DeltaNet backbone, in BF16, FP8, or INT4, with DSpark speculative decoding on a single GB300 or H200." | ||
| tag: NEW | ||
| --- | ||
|
|
||
| ## Deployment | ||
|
|
||
| <a id="install" /> | ||
|
|
||
| <Accordion title="Install SGLang"> | ||
|
|
||
| MiniCPM-V 4.7 support comes from [sgl-project/sglang#41993](https://github.com/sgl-project/sglang/pull/41993); both paths below use that PR's code. For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. | ||
|
|
||
| <Tabs> | ||
|
|
||
| <Tab title="Python (from source)"> | ||
|
|
||
| ```bash Command | ||
| git clone https://github.com/sgl-project/sglang.git | ||
| cd sglang | ||
| git fetch origin pull/41993/head:minicpm-v-4.7 | ||
| git checkout minicpm-v-4.7 | ||
| pip install --upgrade pip | ||
| pip install -e "python" | ||
| ``` | ||
|
|
||
| Then run the **Python** output of the command panel below in that environment. | ||
|
|
||
| </Tab> | ||
|
|
||
| <Tab title="Docker"> | ||
|
|
||
| ```bash Command | ||
| docker pull lmsysorg/sglang:dev-minicpm-v-4.7 | ||
| ``` | ||
|
|
||
| For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. | ||
|
|
||
| </Tab> | ||
|
|
||
| </Tabs> | ||
|
|
||
| </Accordion> | ||
|
|
||
| Pick the hardware, variant, and quantization to generate the launch command. Every recipe runs on a single GPU and is verified on GB300 and H200; the H100 recipes (1B, FP8, INT4) are unverified. The **Speculative Decoding** row trades throughput for latency: turn **DSPARK** on for single-user or low-concurrency chat, and leave it off for multi-user and batch serving (see §2). | ||
|
|
||
| import { Deployment } from "/src/snippets/_deployment.jsx"; | ||
| import { config } from "/src/snippets/configs/openbmb/minicpm-v-4.7.jsx"; | ||
| import { benchmarks } from "/src/snippets/configs/openbmb/minicpm-v-4.7-benchmarks.jsx"; | ||
|
|
||
| <Deployment config={config} benchmarks={benchmarks} /> | ||
|
|
||
| ## Playground | ||
|
|
||
| The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's validated launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. | ||
|
|
||
| import { Playground } from "/src/snippets/_playground.jsx"; | ||
|
|
||
| <Playground config={config} /> | ||
|
|
||
| ## 1. Model Introduction | ||
|
|
||
| MiniCPM-V 4.7 is the latest model in OpenBMB's MiniCPM-V series, covering single-image, multi-image, and video understanding. It ships in two sizes: a 1B dense model and a 35B-A3B Mixture-of-Experts model, the series' first MoE. | ||
|
|
||
| Both sizes use a Qwen3.5-style hybrid language backbone in which every fourth layer is full attention and the rest are Gated DeltaNet linear attention: 24 layers for 1B, and 40 layers with 256 experts (8 active per token) for 35B-A3B. Native context length is 262,144 tokens. The vision encoder compresses visual tokens early inside the ViT, slices high-resolution images into up to 9 tiles, and supports mixed 4x/16x visual token compression (see §3.3). Thinking is on by default. | ||
|
|
||
| **License:** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0). | ||
|
|
||
| <table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}> | ||
| <colgroup> | ||
| <col style={{width: "42%"}} /> | ||
| <col style={{width: "22%"}} /> | ||
| <col style={{width: "36%"}} /> | ||
| </colgroup> | ||
| <thead> | ||
| <tr style={{borderBottom: "2px solid #d55816"}}> | ||
| <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Checkpoint</th> | ||
| <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Precision</th> | ||
| <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Role on this page</th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-1B">openbmb/MiniCPM-V-4.7-1B</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1B dense base (1.55B parameters including the vision encoder).</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B">openbmb/MiniCPM-V-4.7-35B-A3B</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>35B-A3B MoE base (35.2B total parameters, about 3B active).</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-FP8">openbmb/MiniCPM-V-4.7-35B-A3B-FP8</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8 W8A8 (compressed-tensors)</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>35B-A3B FP8.</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-int4">openbmb/MiniCPM-V-4.7-35B-A3B-int4</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>INT4 W4A16, group size 128 (compressed-tensors)</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>35B-A3B INT4.</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-DSpark">openbmb/MiniCPM-V-4.7-1B-DSpark</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16 draft</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Draft model for the 1B <strong>DSPARK</strong> row; not served on its own.</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-DSpark">openbmb/MiniCPM-V-4.7-35B-A3B-DSpark</a></td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16 draft</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Draft model for the 35B-A3B <strong>DSPARK</strong> row; not served on its own.</td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
|
|
||
| OpenBMB also publishes AWQ, GPTQ, and GGUF exports and a 1B int4 checkpoint. They are not part of the matrix above. | ||
|
|
||
| **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20`, from the 35B-A3B checkpoints' `generation_config.json`. SGLang applies these defaults, so do not hardcode them in client code. | ||
|
|
||
| **Resources:** [Hugging Face (1B)](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) · [Hugging Face (35B-A3B)](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B) · [OpenBMB documentation](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/minicpm_v4dot7_en.md) · [SGLang model support (#41993)](https://github.com/sgl-project/sglang/pull/41993). | ||
|
|
||
| ## 2. Configuration Tips | ||
|
|
||
| - Every generated command carries `--trust-remote-code` and `--mamba-radix-cache-strategy extra_buffer`, the radix-cache strategy for the Gated DeltaNet hybrid backbone. | ||
| - Every recipe runs on one GPU. On H100 the 35B-A3B BF16 checkpoint (about 66 GB of weights) does not fit the default static memory fraction of an 80 GB card; use FP8 or INT4 there. | ||
| - Pick the quantization by platform. On H200, quantization is a speed win: at concurrency 16, INT4 and FP8 reach 15,808 and 15,280 tokens/s per GPU against 12,103 for BF16, and INT4 has the fastest 35B-A3B decode (TPOT 3.47 ms at concurrency 1). On GB300, BF16 decodes fastest (TPOT 4.36 ms), FP8 is on par (4.44 ms), and INT4 is slower (8.56 ms), so FP8 and INT4 there save memory, not time. | ||
| - SGLang detects FP8 and INT4 from the checkpoint's `quantization_config`; no `--quantization` flag is needed. | ||
| - The **DSPARK** row adds `--speculative-algorithm DSPARK` and `--speculative-draft-model-path` pointing at the variant's DSpark checkpoint. Use it for low-concurrency serving of the 35B-A3B variants. On real text (sharegpt) at concurrency 1, it cuts TPOT from 4.36 to 2.80 ms for BF16 and from 4.44 to 2.76 ms for FP8 on GB300, and from 3.69 to 3.02 ms (BF16), 3.55 to 2.61 ms (FP8), and 3.47 to 2.38 ms (INT4) on H200. On 1B the draft overhead roughly cancels the gain, and at concurrency 16 DSPARK raises TPOT on every configuration. | ||
| - Benchmark speculative decoding on real text. Random-token prompts collapse draft acceptance, so they say nothing about DSPARK. For a real-workload reference, GSM8K accept length is about 2.6 on 1B, 4.1 on 35B-A3B BF16, 4.0 on FP8, and 3.8 on INT4. | ||
| - Every command carries `--reasoning-parser auto --tool-call-parser auto`. SGLang resolves both from the chat template, to `qwen3` for reasoning and `qwen3_coder` for tool calls, so the thinking trace lands in `message.reasoning_content` and tool calls in `message.tool_calls` (see §3.1 and §3.2). To turn either off, toggle it in the **Parsers** card of the [Playground](#playground). | ||
| - Thinking is on by default. To get a direct answer for one request, pass `"chat_template_kwargs": {"enable_thinking": false}` (in the OpenAI Python client, `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`); `reasoning_content` then comes back empty. | ||
| - Image and video requests need no extra server flags. | ||
|
|
||
| ## 3. Advanced Usage | ||
|
|
||
| ### 3.1 Reasoning | ||
|
|
||
| MiniCPM-V 4.7 thinks by default. The reasoning parser in every generated command returns the thinking trace in `message.reasoning_content` and the final answer in `message.content`. | ||
|
|
||
| <Accordion title="Reasoning (Python)"> | ||
|
|
||
| ```python Example | ||
| import requests | ||
|
|
||
| resp = requests.post( | ||
| "http://localhost:30000/v1/chat/completions", | ||
| json={ | ||
| "model": "openbmb/MiniCPM-V-4.7-35B-A3B", | ||
| "messages": [{"role": "user", "content": "What is 23*17?"}], | ||
| }, | ||
| ).json() | ||
| choice = resp["choices"][0] | ||
| print("finish_reason:", choice["finish_reason"]) | ||
| print("--- reasoning_content ---") | ||
| print(choice["message"]["reasoning_content"]) | ||
| print("--- content ---") | ||
| print(choice["message"]["content"]) | ||
| ``` | ||
|
|
||
| </Accordion> | ||
|
|
||
| <Accordion title="Example Output"> | ||
|
|
||
| ```text Output | ||
| finish_reason: stop | ||
| --- reasoning_content --- | ||
| The user is asking for the product of 23 and 17. I need to compute 23 * 17. | ||
|
|
||
| Let me calculate: | ||
| 23 * 17 | ||
| I can break it down: | ||
| 23 * 10 = 230 | ||
| 23 * 7 = 161 | ||
| Now add: 230 + 161 = 391 | ||
|
|
||
| Wait, let me double-check: | ||
| 23 * 17 | ||
| 23 * (10 + 7) = 23*10 + 23*7 | ||
| 23*10 = 230 | ||
| 23*7: 20*7=140, 3*7=21, so 140+21=161 | ||
| Now 230 + 161 = 391 | ||
|
|
||
| So the product is 391. | ||
|
|
||
| --- content --- | ||
|
|
||
|
|
||
| To find 23 × 17, I'll calculate the multiplication: | ||
|
|
||
| 23 × 17 = 391 | ||
|
|
||
| Answer: 391 | ||
| ``` | ||
|
|
||
| </Accordion> | ||
|
|
||
| ### 3.2 Tool calling | ||
|
|
||
| The tool-call parser in every generated command returns calls in `message.tool_calls`, with the arguments as a JSON string. Because thinking is on, print `reasoning_content` and `content` alongside the calls. | ||
|
|
||
| <Accordion title="Tool calling (Python)"> | ||
|
|
||
| ```python Example | ||
| import json | ||
|
|
||
| import requests | ||
|
|
||
| tools = [{ | ||
| "type": "function", | ||
| "function": { | ||
| "name": "get_weather", | ||
| "description": "Get the current weather for a city.", | ||
| "parameters": { | ||
| "type": "object", | ||
| "properties": {"city": {"type": "string", "description": "City name"}}, | ||
| "required": ["city"], | ||
| }, | ||
| }, | ||
| }] | ||
|
|
||
| resp = requests.post( | ||
| "http://localhost:30000/v1/chat/completions", | ||
| json={ | ||
| "model": "openbmb/MiniCPM-V-4.7-35B-A3B", | ||
| "messages": [{"role": "user", "content": "What is the weather in Beijing today?"}], | ||
| "tools": tools, | ||
| }, | ||
| ).json() | ||
| choice = resp["choices"][0] | ||
| msg = choice["message"] | ||
| print("finish_reason:", choice["finish_reason"]) | ||
| print("--- reasoning_content ---") | ||
| print(msg["reasoning_content"]) | ||
| print("--- content ---") | ||
| print(repr(msg["content"])) | ||
| print("--- tool_calls ---") | ||
| print(json.dumps(msg["tool_calls"], indent=1)) | ||
| ``` | ||
|
|
||
| </Accordion> | ||
|
|
||
| <Accordion title="Example Output"> | ||
|
|
||
| ```text Output | ||
| finish_reason: tool_calls | ||
| --- reasoning_content --- | ||
| The user is asking about the weather in Beijing. I need to call the get_weather function with the city parameter set to Beijing. | ||
|
|
||
| --- content --- | ||
| '\n\n' | ||
| --- tool_calls --- | ||
| [ | ||
| { | ||
| "id": "call_592d3b19eca4432eaf1a89f1", | ||
| "index": 0, | ||
| "type": "function", | ||
| "function": { | ||
| "name": "get_weather", | ||
| "arguments": "{\"city\": \"Beijing\"}" | ||
| } | ||
| } | ||
| ] | ||
| ``` | ||
|
|
||
| </Accordion> | ||
|
|
||
| To get the final answer, append the assistant message and a `tool` role message with the tool's result, then call the API again with the same `tools` list. | ||
|
|
||
| ### 3.3 Per-request image processing | ||
|
|
||
| OpenAI-compatible `/v1/chat/completions` and `/v1/completions` requests accept `mm_processor_kwargs`, which overrides the image processor for that request only: | ||
|
|
||
| - `downsample_mode`: `"16x"` (default) or `"4x"`. `"4x"` keeps more visual tokens per tile. | ||
| - `max_slice_nums`: the maximum number of high-resolution slices. `1` sends the image without slicing. | ||
|
|
||
| <Accordion title="Image token budget (Python)"> | ||
|
|
||
| ```python Example | ||
| from openai import OpenAI | ||
|
|
||
| client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") | ||
|
|
||
| IMAGE_URL = "https://www.ilankelman.org/stopsigns/australia.jpg" | ||
|
|
||
| for overrides in ({}, {"downsample_mode": "4x"}, {"max_slice_nums": 1}): | ||
| resp = client.chat.completions.create( | ||
| model="openbmb/MiniCPM-V-4.7-35B-A3B", | ||
| messages=[{ | ||
| "role": "user", | ||
| "content": [ | ||
| {"type": "image_url", "image_url": {"url": IMAGE_URL}}, | ||
| {"type": "text", "text": "Describe this image in one sentence."}, | ||
| ], | ||
| }], | ||
| extra_body={"mm_processor_kwargs": overrides} if overrides else None, | ||
| ) | ||
| print(overrides or "default", "prompt_tokens =", resp.usage.prompt_tokens) | ||
| ``` | ||
|
|
||
| </Accordion> | ||
|
|
||
| Prompt token counts measured on a 1568×448 image with the same text prompt: | ||
|
|
||
| <table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}> | ||
| <thead> | ||
| <tr style={{borderBottom: "2px solid #d55816"}}> | ||
| <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}><code>mm_processor_kwargs</code></th> | ||
| <th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Prompt tokens</th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none (<code>"16x"</code>)</td> | ||
| <td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>355</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>{`{"downsample_mode": "4x"}`}</code></td> | ||
| <td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>1291</td> | ||
| </tr> | ||
| <tr> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"max_slice_nums": 1}`}</code></td> | ||
| <td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>93</td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
|
|
||
| Overrides do not leak between requests: alternating `"16x"` and `"4x"` on the same image returns the matching token count and a correct answer each time. | ||
|
|
||
| For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On GB300, BF16 and FP8 decode at about the same speed at concurrency 1 (TPOT 4.36 vs 4.44 ms), and FP8 pulls ahead at concurrency 16 (16,059 vs 14,094 tokens/s per GPU). INT4 is slower on GB300 at every concurrency (TPOT 8.56 ms at concurrency 1), so use it there only to save memory.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Applied with your wording in 8b84f59.