Repository navigation
Fix top_k default value to 0 for disabling top-k filtering #4695
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
21cf0a9
8ad0480
6f1192e
8645da5
71bf5b2
4376906
986f7c2
b02c444
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -506,7 +506,7 @@ def __init__( | |
| "repetition_penalty": self.repetition_penalty, | ||
| "temperature": self.temperature, | ||
| "top_p": self.top_p, | ||
| "top_k": -1 if self.top_k is None else self.top_k, | ||
| "top_k": -1 if not self.top_k else self.top_k, | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think can (should) just use
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, the vLLM support to consider all tokens by setting top_k=0 (besides -1) was introduced in vllm-0.9.0 and we require vllm>=0.10.2.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. On the other hand, what if the users still set
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we could indeed deprecate |
||
| "min_p": 0.0 if self.min_p is None else self.min_p, | ||
| "max_tokens": args.max_new_tokens, | ||
| "detokenize": False, # to avoid vllm to decode (we don't need it) | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -81,8 +81,8 @@ class GRPOConfig(TrainingArguments): | |||||
| top_p (`float`, *optional*, defaults to `1.0`): | ||||||
| Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to | ||||||
| `1.0` to consider all tokens. | ||||||
| top_k (`int`, *optional*): | ||||||
| Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is | ||||||
| top_k (`int`, defaults to `0`): | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is | ||||||
| disabled and all tokens are considered. | ||||||
| min_p (`float`, *optional*): | ||||||
| Minimum token probability, which will be scaled by the probability of the most likely token. It must be a | ||||||
|
|
@@ -433,10 +433,10 @@ class GRPOConfig(TrainingArguments): | |||||
| "Set to 1.0 to consider all tokens." | ||||||
| }, | ||||||
| ) | ||||||
| top_k: int | None = field( | ||||||
| default=None, | ||||||
| top_k: int = field( | ||||||
| default=0, | ||||||
| metadata={ | ||||||
| "help": "Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, " | ||||||
| "help": "Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, " | ||||||
| "top-k-filtering is disabled and all tokens are considered." | ||||||
| }, | ||||||
| ) | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -77,8 +77,8 @@ class RLOOConfig(TrainingArguments): | |||||
| top_p (`float`, *optional*, defaults to `1.0`): | ||||||
| Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to | ||||||
| `1.0` to consider all tokens. | ||||||
| top_k (`int`, *optional*): | ||||||
| Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is | ||||||
| top_k (`int`, defaults to `0`): | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is | ||||||
| disabled and all tokens are considered. | ||||||
| min_p (`float`, *optional*): | ||||||
| Minimum token probability, which will be scaled by the probability of the most likely token. It must be a | ||||||
|
|
@@ -328,10 +328,10 @@ class RLOOConfig(TrainingArguments): | |||||
| "Set to 1.0 to consider all tokens." | ||||||
| }, | ||||||
| ) | ||||||
| top_k: int | None = field( | ||||||
| default=None, | ||||||
| top_k: int = field( | ||||||
| default=0, | ||||||
| metadata={ | ||||||
| "help": "Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, " | ||||||
| "help": "Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, " | ||||||
| "top-k-filtering is disabled and all tokens are considered." | ||||||
| }, | ||||||
| ) | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.