Repository navigation
[GRPO] Adds an option to sleep vllm when running in colocated mode #3968
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
c9e02bd
44f1533
654e97d
072f40c
ab22c8d
6763031
e55b88a
adb4736
415d187
78f9d60
8c67f8e
6e4acbc
0130281
1810b52
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -835,7 +835,10 @@ def __init__( | |
| # Latest vLLM v1 memory profiler is misled by the high default value (i.e., 32768) - thinking there's not enough memory | ||
| max_num_batched_tokens=4096, | ||
| model_impl=self.args.vllm_model_impl, | ||
| enable_sleep_mode=self.args.vllm_sleep_enabled, | ||
|
edbeeching marked this conversation as resolved.
Resolved
|
||
| ) | ||
| if self.args.vllm_sleep_enabled: | ||
| self.llm.sleep(level=1) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Every time vllm wakes up, it wakes up to an updated model, right? So why not use level = 2 to further improve efficiency?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah good point, let me rerun the benchmark with level=2.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For level = 2, you will also need to wake-up then sleep in BAsically anytime you touch vllm (in generation or loading/syncying the model), you wake up, do the work, and then go back to sleep. Additional note: the reason we did not move sleep to upstream was because of a vllm bug, which is just recently fixed by this PR. So that means, you need to use vllm version that incorporates the fix to be able to use sleep level 2 without segmentation fault.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I have already included a wake up call before the weight sync. I don't re-sleep as we have the gen step straight after. It looks like the level 2 fix has not made it into their most recent release. I will leave level=1 for now for better backward compatability. Unless there are other reasons to go with level 2? |
||
| else: | ||
| raise ValueError(f"vllm_mode must be either 'server' or 'colocate', got '{self.vllm_mode}'.") | ||
|
|
||
|
|
@@ -1426,6 +1429,11 @@ def _generate_and_score_completions( | |
|
|
||
| # Generate completions using either vLLM or regular generation | ||
| if self.use_vllm: | ||
| if self.vllm_mode == "colocate" and self.args.vllm_sleep_enabled: | ||
| # wake up colocated vLLM instances if needed | ||
| torch.cuda.empty_cache() # required to avoid OOM in some cases | ||
| self.llm.wake_up() | ||
|
|
||
| # First, update the vLLM weights if needed | ||
| if self.state.global_step != self._last_loaded_step: | ||
| self._move_model_to_vllm() | ||
|
|
@@ -1534,6 +1542,9 @@ def _generate_and_score_completions( | |
| tp_slice = slice(local_rank_in_group * orig_size, (local_rank_in_group + 1) * orig_size) | ||
| completion_ids = completion_ids[tp_slice] | ||
|
|
||
| if self.args.vllm_sleep_enabled: | ||
| self.llm.sleep(level=1) | ||
|
|
||
| # Pad the completions, and concatenate them with the prompts | ||
| completion_ids = [torch.tensor(ids, device=device) for ids in completion_ids] | ||
| completion_ids = pad(completion_ids, padding_value=self.pad_token_id) | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.