Skip to content

[Flex][RL] add custom mask support - #3633

Closed
liangel-02 wants to merge 1 commit into
mainfrom
sliding_window_rl
Closed

liangel-02 wants to merge 1 commit into
mainfrom
sliding_window_rl

Conversation

@liangel-02

@liangel-02 liangel-02 commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

adding custom mask support (ie sliding window attention) for Flex Attention RL loop.

  • vLLM takes in custom mask mod via self.vllm_attn.logical_mask_mod so we just need to pass this through
  • in order to work with cuda graphs, we need to cache mask mod generation on generator side in titan so that vLLM recognizes it as the same object and doesn't rebuild
  • on vLLM side, we need to move custom mask handling out of the fwd [FlexAttention] make custom mask mods fully cudagraphable vllm-project/vllm#45232
  • unified flex configs to be more clean

batch invariant grpo script
Screenshot 2026-06-11 at 12 57 56 AM

sliding window unit test
Screenshot 2026-06-11 at 12 55 10 AM

custom mask mod unit test
Screenshot 2026-06-11 at 12 38 40 PM

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 11, 2026
@liangel-02
liangel-02 force-pushed the sliding_window_rl branch 2 times, most recently from 3aa1859 to 80dad2c Compare June 11, 2026 16:34
@liangel-02
liangel-02 marked this pull request as ready for review June 11, 2026 16:43
class Config(Module.Config):
block_size: int | tuple[int, int] = _DEFAULT_SPARSE_BLOCK_SIZE
kernel_options: dict = field(default_factory=dict)
mask_mod: MaskMod.Config | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have another mask_type in BaseAttention. Having two configs for one thing is not a good idea. This is also implicit. For example, I'm developing a selected gather to improve CP + sliding window. How do I figure out which one should I look into to decide whether I should apply selected gather? mask_type or mask_mod?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw I've killed the mask_type in BaseAttention.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, I miss that part.

@tianyu-l tianyu-l left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's slow down a bit.

It seems vllm has native sliding window support. Do we need to support custom masking beyond that? gpt-oss is the only model which uses sliding window / full + causal document masking; all other models are using full causal document masking. Maybe we can support sliding window only, and revisit if we need to support new models with more complicated masking.

We need a robust way to error out if user change the model to use some exotic mask, which won't be captured by today's VLLMAttentionWrapper.

class Config(Module.Config):
block_size: int | tuple[int, int] = _DEFAULT_SPARSE_BLOCK_SIZE
kernel_options: dict = field(default_factory=dict)
mask_mod: MaskMod.Config | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw I've killed the mask_type in BaseAttention.

num_kv_heads=n_kv_heads,
head_dim=head_dim,
mask_mod=getattr(inner, "mask_mod", None),
sliding_window=sliding_window,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes every single layer using the same sliding window, which doesn't make sense in alternating full / sliding window setup? e.g. gpt-oss

Comment on lines +290 to +291
if mask_mod_cfg is not None and config.sliding_window is None:
self.vllm_attn.logical_mask_mod = mask_mod_cfg.build().get_mask_mod()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm I think we need some discussion here before we build the solution:

  • it seems for vllm, we just need to specify the sliding window part, and vllm will handle continuous batching
  • but for trainer, we ourselves need to specify "document mask" AND "sliding window mask"
  • I don't want to keep two different solutions, get_attention_masks for training, and MaskMod.Config for inference. I want to unify the solution as much as possible.

num_kv_heads=num_kv_heads,
cache_config=cache_config,
quant_config=None,
per_layer_sliding_window=config.sliding_window,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this is taking precedence over mask_mod. Maybe you could take it from https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L48.

https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L128 is layer-specific. To avoid using the same sliding_window size for all layers, we probably should:

  • make gpt-oss Attention.sliding_window_size optional, when it's None, this per_layer_sliding_window should be None.
  • remove use_sliding_attention and depend on sliding_window_size None or not to determine which mask to use in training.
  • still don't accept custom mask mod, because the model we support so far doesn't require beyond sliding window attention.
  • (separate) support gpt-oss training with VarlenAttention backend. Basically this is not true https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/model.py#L66-L68?

inner.kernel_options["BLOCK_M"] = self._BLOCK_M
inner.kernel_options["BLOCK_N"] = self._BLOCK_N
if self.mask_mod is not None:
inner.mask_mod = self.mask_mod

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is potentially changing model definition, should not do this.

Comment on lines +229 to +230
mask_mod: MaskMod.Config | None = None
sliding_window: int | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a essential difference (e.g. perf), if I use my custom mask_mod to support sliding window attention, instead of using the vllm native sliding_window int option.

@liangel-02
liangel-02 marked this pull request as draft June 15, 2026 22:14
@liangel-02

Copy link
Copy Markdown
Contributor Author

closing for #3687

@liangel-02 liangel-02 closed this Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rl ciflow/8gpu CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants