Repository navigation
[Flex][RL] add custom mask support - #3633
liangel-02 wants to merge 1 commit into
Conversation
3aa1859 to
80dad2c
Compare
80dad2c to
b1d20ae
Compare
| class Config(Module.Config): | ||
| block_size: int | tuple[int, int] = _DEFAULT_SPARSE_BLOCK_SIZE | ||
| kernel_options: dict = field(default_factory=dict) | ||
| mask_mod: MaskMod.Config | None = None |
There was a problem hiding this comment.
We have another mask_type in BaseAttention. Having two configs for one thing is not a good idea. This is also implicit. For example, I'm developing a selected gather to improve CP + sliding window. How do I figure out which one should I look into to decide whether I should apply selected gather? mask_type or mask_mod?
There was a problem hiding this comment.
fwiw I've killed the mask_type in BaseAttention.
tianyu-l
left a comment
There was a problem hiding this comment.
Let's slow down a bit.
It seems vllm has native sliding window support. Do we need to support custom masking beyond that? gpt-oss is the only model which uses sliding window / full + causal document masking; all other models are using full causal document masking. Maybe we can support sliding window only, and revisit if we need to support new models with more complicated masking.
We need a robust way to error out if user change the model to use some exotic mask, which won't be captured by today's VLLMAttentionWrapper.
| class Config(Module.Config): | ||
| block_size: int | tuple[int, int] = _DEFAULT_SPARSE_BLOCK_SIZE | ||
| kernel_options: dict = field(default_factory=dict) | ||
| mask_mod: MaskMod.Config | None = None |
There was a problem hiding this comment.
fwiw I've killed the mask_type in BaseAttention.
| num_kv_heads=n_kv_heads, | ||
| head_dim=head_dim, | ||
| mask_mod=getattr(inner, "mask_mod", None), | ||
| sliding_window=sliding_window, |
There was a problem hiding this comment.
this makes every single layer using the same sliding window, which doesn't make sense in alternating full / sliding window setup? e.g. gpt-oss
| if mask_mod_cfg is not None and config.sliding_window is None: | ||
| self.vllm_attn.logical_mask_mod = mask_mod_cfg.build().get_mask_mod() |
There was a problem hiding this comment.
hmm I think we need some discussion here before we build the solution:
- it seems for vllm, we just need to specify the sliding window part, and vllm will handle continuous batching
- but for trainer, we ourselves need to specify "document mask" AND "sliding window mask"
- I don't want to keep two different solutions,
get_attention_masksfor training, andMaskMod.Configfor inference. I want to unify the solution as much as possible.
| num_kv_heads=num_kv_heads, | ||
| cache_config=cache_config, | ||
| quant_config=None, | ||
| per_layer_sliding_window=config.sliding_window, |
There was a problem hiding this comment.
It seems this is taking precedence over mask_mod. Maybe you could take it from https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L48.
https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L128 is layer-specific. To avoid using the same sliding_window size for all layers, we probably should:
- make gpt-oss
Attention.sliding_window_sizeoptional, when it's None, thisper_layer_sliding_windowshould be None. - remove
use_sliding_attentionand depend on sliding_window_size None or not to determine which mask to use in training. - still don't accept custom mask mod, because the model we support so far doesn't require beyond sliding window attention.
- (separate) support gpt-oss training with VarlenAttention backend. Basically this is not true https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L66-L68?
| inner.kernel_options["BLOCK_M"] = self._BLOCK_M | ||
| inner.kernel_options["BLOCK_N"] = self._BLOCK_N | ||
| if self.mask_mod is not None: | ||
| inner.mask_mod = self.mask_mod |
There was a problem hiding this comment.
this is potentially changing model definition, should not do this.
| mask_mod: MaskMod.Config | None = None | ||
| sliding_window: int | None = None |
There was a problem hiding this comment.
Is there a essential difference (e.g. perf), if I use my custom mask_mod to support sliding window attention, instead of using the vllm native sliding_window int option.
|
closing for #3687 |
adding custom mask support (ie sliding window attention) for Flex Attention RL loop.
self.vllm_attn.logical_mask_modso we just need to pass this throughbatch invariant grpo script

sliding window unit test

custom mask mod unit test
