Skip to content

[RL] enable GPT-OSS for titan RL loop - #3687

Merged
liangel-02 merged 3 commits into
mainfrom
torchtitan-flex
Jun 18, 2026
Merged

liangel-02 merged 3 commits into
mainfrom
torchtitan-flex

Conversation

@liangel-02

@liangel-02 liangel-02 commented Jun 16, 2026 •

Copy link
Copy Markdown
Contributor

as title, using varlen attention first before revisitng flex in a followup

loss stays the same for training between main flex (today), flex after sliding_window semantic changes, and newly added varlen
image

Screenshot 2026-06-17 at 5 34 51 PM Screenshot 2026-06-17 at 10 17 40 PM

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 16, 2026
@liangel-02
liangel-02 force-pushed the torchtitan-flex branch 2 times, most recently from ac3e39c to 88d09f7 Compare June 16, 2026 16:32
@liangel-02 liangel-02 changed the title Torchtitan flex [Flex][RL] enable GPT-OSS for titan RL loop Jun 16, 2026
@liangel-02
liangel-02 force-pushed the torchtitan-flex branch 3 times, most recently from aa558cb to af4ad50 Compare June 16, 2026 18:16
@liangel-02
liangel-02 marked this pull request as ready for review June 16, 2026 18:17
Comment thread torchtitan/models/gpt_oss/model.py Resolved
Comment thread torchtitan/models/gpt_oss/model.py Resolved
Comment thread torchtitan/models/gpt_oss/model.py Resolved
@liangel-02 liangel-02 changed the title [Flex][RL] enable GPT-OSS for titan RL loop [RL] enable GPT-OSS for titan RL loop Jun 17, 2026
@liangel-02
liangel-02 marked this pull request as draft June 17, 2026 03:18
@liangel-02
liangel-02 force-pushed the torchtitan-flex branch 3 times, most recently from f9f60fd to cd1be7b Compare June 17, 2026 03:28

@tianyu-l tianyu-l left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question 1: is Flex support removed for gpt-oss?
question 2: is the implementation mathematically correct for training with document masking?

Comment thread torchtitan/models/gpt_oss/model.py Resolved
Comment thread torchtitan/models/common/attention.py Resolved
Comment thread torchtitan/models/common/attention.py Resolved
Comment thread torchtitan/models/common/attention.py Resolved
Comment thread torchtitan/models/gpt_oss/model.py Resolved
@liangel-02
liangel-02 force-pushed the torchtitan-flex branch 4 times, most recently from fa0d345 to c33a008 Compare June 17, 2026 23:52
@liangel-02
liangel-02 requested a review from tianyu-l June 17, 2026 23:54
@liangel-02
liangel-02 marked this pull request as ready for review June 17, 2026 23:54
Comment on lines +125 to +126
if isinstance(sinks, DTensor):
sinks = sinks.to_local(grad_placements=sinks.placements)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why didn't we need this before?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the application now fires in the attention forward which on vllm path would be w/ local tensors

Comment thread torchtitan/models/gpt_oss/model.py Resolved
Comment thread torchtitan/models/common/attention.py Resolved
@liangel-02
liangel-02 force-pushed the torchtitan-flex branch 2 times, most recently from 379855f to 9bfe25a Compare June 18, 2026 15:43
@liangel-02
liangel-02 requested a review from tianyu-l June 18, 2026 15:46
@liangel-02
liangel-02 merged commit e403578 into main Jun 18, 2026
19 of 24 checks passed
aditvenk added a commit that referenced this pull request Jun 25, 2026
…aselines

Three independent CI failures in the graph_trainer test suite:

1. gpt_oss trace tests (test_trace_module.py). PR #3687 flipped the gpt_oss
   Attention.Config.inner_attention default from FlexAttention to
   VarlenAttention (attn_backend defaults to "varlen"). Request
   attn_backend="flex" explicitly in all three usages (test_gpt_oss,
   test_flex_attention_annotations, test_gpt_oss_fsdp).

2. test_eager_self_deterministic (test_bitwise_deterministic.py). A torch
   numerics update drifted eager results below loss-print precision, so the
   hardcoded golden model/grad hashes (and Qwen3 MoE loss values) no longer
   match. Regenerate the baselines via EXPECTTEST_ACCEPT=1 for all six
   model/attention variants. The determinism machinery is unchanged: the
   self-comparison tests still pass bitwise.

3. AutoParallel llama3 tests (integration_tests.py, test_numerics.py).
   PyTorch #186754 (2026-06-24) removed propagate_single_input_strategy in
   favor of propagate_single_input_single_dim_strategy, but AutoParallel's
   convert_element_type_rule still imports the old name, so the sharding
   optimizer fails with ImportError. Disable autoparallel_llama3_fsdp_tp and
   test_llama3_aot_fx_trace_autoparallel_vs_eager until AutoParallel migrates.
   See #3699.
aditvenk added a commit that referenced this pull request Jun 25, 2026
…aselines

Three independent CI failures in the graph_trainer test suite:

1. gpt_oss trace tests (test_trace_module.py). PR #3687 flipped the gpt_oss
   Attention.Config.inner_attention default from FlexAttention to
   VarlenAttention (attn_backend defaults to "varlen"). Request
   attn_backend="flex" explicitly in all three usages (test_gpt_oss,
   test_flex_attention_annotations, test_gpt_oss_fsdp).

2. test_eager_self_deterministic (test_bitwise_deterministic.py). A torch
   numerics update drifted eager results below loss-print precision, so the
   hardcoded golden model/grad hashes (and Qwen3 MoE loss values) no longer
   match. Regenerate the baselines via EXPECTTEST_ACCEPT=1 for all six
   model/attention variants. The determinism machinery is unchanged: the
   self-comparison tests still pass bitwise.

3. AutoParallel llama3 tests (integration_tests.py, test_numerics.py).
   PyTorch #186754 (2026-06-24) removed propagate_single_input_strategy in
   favor of propagate_single_input_single_dim_strategy, but AutoParallel's
   convert_element_type_rule still imports the old name, so the sharding
   optimizer fails with ImportError. Disable autoparallel_llama3_fsdp_tp and
   test_llama3_aot_fx_trace_autoparallel_vs_eager until AutoParallel migrates.
   See #3699.
aditvenk added a commit that referenced this pull request Jun 26, 2026
#3791)

…aselines

Three independent CI failures in the graph_trainer test suite:
    
1. gpt_oss trace tests (test_trace_module.py). PR #3687 flipped the
gpt_oss
       Attention.Config.inner_attention default from FlexAttention to
       VarlenAttention (attn_backend defaults to "varlen"). Request
       attn_backend="flex" explicitly in all three usages (test_gpt_oss,
       test_flex_attention_annotations, test_gpt_oss_fsdp).
    
2. test_eager_self_deterministic (test_bitwise_deterministic.py). A
torch
numerics update drifted eager results below loss-print precision, so the
hardcoded golden model/grad hashes (and Qwen3 MoE loss values) no longer
match. Regenerate the baselines via EXPECTTEST_ACCEPT=1 for all six
model/attention variants. The determinism machinery is unchanged: the
       self-comparison tests still pass bitwise.
    
3. AutoParallel llama3 tests (integration_tests.py, test_numerics.py).
PyTorch #186754 (2026-06-24) removed propagate_single_input_strategy in
favor of propagate_single_input_single_dim_strategy, but AutoParallel's
convert_element_type_rule still imports the old name, so the sharding
optimizer fails with ImportError. Disable autoparallel_llama3_fsdp_tp
and
test_llama3_aot_fx_trace_autoparallel_vs_eager until AutoParallel
migrates.
       See #3699.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rl ciflow/8gpu CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants