Skip to content

Reward Model Training on Megatron backend #720

Open

Description

@terrykong
No description provided.

Activity

  1. afennelly-mitre commented on Oct 14, 2025

    @afennelly-mitre

    @terrykong any idea of when this feature will see development progress?

  2. afennelly-mitre commented on Oct 29, 2025

    @afennelly-mitre

    @terrykong any idea of when this feature will see development progress?

    @terrykong re pinging to see if this is something that is planned for any time soon (ex: v0.4 release?)

  3. terrykong commented on Oct 29, 2025

    @terrykong
    CollaboratorAuthor

    Hi @afennelly-mitre . v0.4 will be going out in about a week. Reward models on the mcore backend hasn't been started yet.

    @odelalleau @jveronvialard to share if any plans

    The first iteration will likely be porting over the aligner implementation, and we'll need some care around checkpoint saving.

  4. afennelly-mitre commented on Oct 29, 2025

    @afennelly-mitre

    Hi @afennelly-mitre . v0.4 will be going out in about a week. Reward models on the mcore backend hasn't been started yet.

    @odelalleau @jveronvialard to share if any plans

    The first iteration will likely be porting over the aligner implementation, and we'll need some care around checkpoint saving.

    Hey @terrykong, thank you for the speedy reply! Our team is looking forward to the v0.4 release, and if we get a working reward model environment setup that uses the mcore backend, I will follow up here and try to submit a new PR to contribute to the repo.

    Thank you!

  5. afennelly-mitre commented on Oct 29, 2025

    @afennelly-mitre

    @terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?

  6. terrykong commented on Oct 29, 2025

    @terrykong
    CollaboratorAuthor

    @terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?

    This is a diff feature than what's tracked here since this is just to train a mcore RM, but yes, in theory it's possible, but you'd probably have to change https://github.com/NVIDIA-NeMo/RL/blob/main/examples/run_grpo_rm.py to generalize to N reward model environments.

    For context, there's also another integration we're pretty excited about to incorporate a richer environment suite. I can't give ETA yet, but that's coming soon.

  7. RayenTian commented on Oct 30, 2025

    @RayenTian
    Contributor

    @terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?

    The current implementation of the reward model environment supports configuring mcore as the training backend and dtensor as the reward model backend both on one node and multiple node. To achieve this setup, you only need to modify the policy backend parameter in the configuration file to specify your desired backend for each component.

  8. odelalleau commented on Oct 31, 2025

    @odelalleau
    Contributor

    @odelalleau @jveronvialard to share if any plans

    No concrete plans right now (to my knowledge)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions