Skip to content

feat: Add model version tracking with API endpoints and response metadata - #8795

Merged
zhyncs merged 20 commits into
sgl-project:mainfrom
yitianlian:model_version
Aug 14, 2025
Merged

zhyncs merged 20 commits into
sgl-project:mainfrom
yitianlian:model_version

Conversation

@yitianlian

Copy link
Copy Markdown
Contributor

Motivation

This PR introduces a model_version tracking capability for SGLang server deployments. With this feature, each deployed model is associated with a specific version that can be queried or updated.
It is essential to identify which version of a model is currently deployed, particularly in reinforcement learning (RL) systems, where knowing the model version for each rollout is crucial for data selection and the design of off-policy RL algorithms.

Modifications

Core Infrastructure Changes

Server Arguments & Configuration:

  • Added model_version: str = "version_0" to ServerArgs dataclass in server_args.py
  • Added --model-version CLI argument for deployment-time version specification
  • Used tokenizer_manager.server_args.model_version as single source of the model version

API Response Integration:

  • Enhanced /generate responses to include model_version in meta_info field
  • Added metadata: Optional[Dict[str, Any]] field to OpenAI ChatCompletionResponse and CompletionResponse
  • Populated metadata.model_version in both chat and completion OpenAI endpoints

New API Endpoints

Version Management:

  • GET /get_model_version: Retrieve current model version
  • POST /update_model_version: Update model version with optional request abortion

Enhanced Weight Update APIs:

  • Extended UpdateWeightFromDiskReqInput, UpdateWeightsFromTensorReqInput, and UpdateWeightsFromDistributedReqInput with optional model_version parameter
  • Updated /update_weights_from_disk, /update_weights_from_tensor, and /update_weights_from_distributed endpoints to accept and apply version updates
  • Version updates only occur on successful weight updates

Accuracy Test

None

Benchmark & Profiling

None

Checklist

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@mickqian mickqian changed the title Feature: Add model version tracking with API endpoints and response metadata feat: Add model version tracking with API endpoints and response metadata Aug 5, 2025
@mickqian

mickqian commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Hi @yitianlian, thanks for ur contribution! Could you provide a few cases of this feat, so that people with no RL background can easily understand this?

@ocss884

ocss884 commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Hi @yitianlian, thanks for ur contribution! Could you provide a few cases of this feat, so that people with no RL background can easily understand this?

In current RL practice, we may need to 1. interrupt the inference engine -> 2. fetch all ongoing requests -> 3. update weight -> 4. re-construct batch and do generation. Therefore each training batch can contain data itself from multiple prior policy versions and that could affect the final performance of model. We'd like to add-on one more property to monitor the data.

Recommended reading: https://arxiv.org/pdf/2505.24298

@ocss884

ocss884 commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Great work Chengxing!

@JustinTong0323

Copy link
Copy Markdown
Collaborator

check notebook CI:

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-nbconvert", line 7, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/jupyter_core/application.py", line 284, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 1075, in launch_instance
    app.start()
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/nbconvertapp.py", line 420, in start
    self.convert_notebooks()
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/nbconvertapp.py", line 597, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/nbconvertapp.py", line 563, in convert_single_notebook
    output, resources = self.export_single_notebook(
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/nbconvertapp.py", line 487, in export_single_notebook
    output, resources = self.exporter.from_filename(
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/exporters/exporter.py", line 201, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/exporters/exporter.py", line 220, in from_file
    return self.from_notebook_node(
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/exporters/notebook.py", line 36, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/exporters/exporter.py", line 154, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/exporters/exporter.py", line 353, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/preprocessors/base.py", line 48, in __call__
    return self.preprocess(nb, resources)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/preprocessors/execute.py", line 103, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python3.10/dist-packages/nbconvert/preprocessors/execute.py", line 124, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/usr/local/lib/python3.10/dist-packages/jupyter_core/utils/__init__.py", line 158, in wrapped
    return loop.run_until_complete(inner)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/dist-packages/nbclient/client.py", line 1062, in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
  File "/usr/local/lib/python3.10/dist-packages/nbclient/client.py", line 918, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
url = f"http://localhost:{port}/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["is_generation"] is True
assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["preferred_sampling_params"] is None
assert response_json.keys() == {
    "model_path",
    "is_generation",
    "tokenizer_path",
    "preferred_sampling_params",
}
------------------

----- stdout -----
[2025-08-05 06:31:28] INFO:     127.0.0.1:34638 - "GET /get_model_info HTTP/1.1" 200 OK
------------------

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[3], line 10
      8 assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
      9 assert response_json["preferred_sampling_params"] is None
---> 10 assert response_json.keys() == {
     11     "model_path",
     12     "is_generation",
     13     "tokenizer_path",
     14     "preferred_sampling_params",
     15 }

AssertionError: 

@mickqian

mickqian commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Hi @yitianlian, thanks for ur contribution! Could you provide a few cases of this feat, so that people with no RL background can easily understand this?

In current RL practice, we may need to 1. interrupt the inference engine -> 2. fetch all ongoing requests -> 3. update weight -> 4. re-construct batch and do generation. Therefore each training batch can contain data itself from multiple prior policy versions and that could affect the final performance of model. We'd like to add-on one more property to monitor the data.

Recommended reading: https://arxiv.org/pdf/2505.24298

Thanks, this makes sense. But we might want to emphasize that this version is fully customized, not to be confused with official huggingface or model version.

WDYT?

@ocss884

ocss884 commented Aug 5, 2025

Copy link
Copy Markdown
Collaborator

Hi @yitianlian, thanks for ur contribution! Could you provide a few cases of this feat, so that people with no RL background can easily understand this?

In current RL practice, we may need to 1. interrupt the inference engine -> 2. fetch all ongoing requests -> 3. update weight -> 4. re-construct batch and do generation. Therefore each training batch can contain data itself from multiple prior policy versions and that could affect the final performance of model. We'd like to add-on one more property to monitor the data.
Recommended reading: https://arxiv.org/pdf/2505.24298

Thanks, this makes sense. But we might want to emphasize that this version is fully customized, not to be confused with official huggingface or model version.

WDYT?

Yeah we may need a better naming for it. The scenario is that you need to update the instance's weight online. Just come up some: instance/model/weights_version/revision. And change the default value to string "default" or int 0. @yitianlian Any opinions on it

@yitianlian

Copy link
Copy Markdown
Contributor Author

Yeah we may need a better naming for it. The scenario is that you need to update the instance's weight online. Just come up some: instance/model/weights_version/revision. And change the default value to string "default" or int 0. @yitianlian Any opinions on it

Sure! I'm also concerned that users will be confused by the name. I thought model_iteration_id or model_update_id might be better?

As for the failed notebook CI, I’ll add an extra key in the test to ensure it passes.

@mickqian @ocss884

@yitianlian

Copy link
Copy Markdown
Contributor Author

I have changed model_version to weight_version and also fixed the notebook CI error.

@mickqian @ocss884

Comment thread test/srt/test_model_version.py Resolved
@zhuzilin zhuzilin mentioned this pull request Aug 7, 2025
6 tasks
@mickqian
mickqian enabled auto-merge (squash) August 7, 2025 08:05
@yitianlian

Copy link
Copy Markdown
Contributor Author

Hi, is there anything I need to do to help get this PR merged?
@mickqian

@zhyncs zhyncs self-assigned this Aug 9, 2025
@mickqian

Copy link
Copy Markdown
Collaborator

Hi, is there anything I need to do to help get this PR merged? @mickqian

Sounds good, I have approved already

@yitianlian

Copy link
Copy Markdown
Contributor Author

Hi, is there anything I need to do to help get this PR merged? QAQ

@hebiao064 hebiao064 self-assigned this Aug 13, 2025
@zhyncs
zhyncs disabled auto-merge August 14, 2025 19:13
@zhyncs
zhyncs merged commit c1c7dc4 into sgl-project:main Aug 14, 2025
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025
MahmoudAshraf97 pushed a commit to MahmoudAshraf97/sglang that referenced this pull request Sep 8, 2025

# Update weight version if provided and weights update was successful
if success and obj.weight_version is not None:
_update_weight_version_if_provided(obj.weight_version)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq: why do we put logic here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants