You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The first iteration will likely be porting over the aligner implementation, and we'll need some care around checkpoint saving.
Hey @terrykong, thank you for the speedy reply! Our team is looking forward to the v0.4 release, and if we get a working reward model environment setup that uses the mcore backend, I will follow up here and try to submit a new PR to contribute to the repo.
@terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?
@terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?
For context, there's also another integration we're pretty excited about to incorporate a richer environment suite. I can't give ETA yet, but that's coming soon.
@terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?
The current implementation of the reward model environment supports configuring mcore as the training backend and dtensor as the reward model backend both on one node and multiple node. To achieve this setup, you only need to modify the policy backend parameter in the configuration file to specify your desired backend for each component.
Activity
afennelly-mitre commentedon Oct 14, 2025
@terrykong any idea of when this feature will see development progress?
afennelly-mitre commentedon Oct 29, 2025
@terrykong re pinging to see if this is something that is planned for any time soon (ex: v0.4 release?)
terrykong commentedon Oct 29, 2025
Hi @afennelly-mitre . v0.4 will be going out in about a week. Reward models on the mcore backend hasn't been started yet.
@odelalleau @jveronvialard to share if any plans
The first iteration will likely be porting over the aligner implementation, and we'll need some care around checkpoint saving.
afennelly-mitre commentedon Oct 29, 2025
Hey @terrykong, thank you for the speedy reply! Our team is looking forward to the v0.4 release, and if we get a working reward model environment setup that uses the mcore backend, I will follow up here and try to submit a new PR to contribute to the repo.
Thank you!
afennelly-mitre commentedon Oct 29, 2025
@terrykong do you think it would be feasible to have the reward model environment hosted on one node using the dtensor backend, and have all other nodes using the mcore backend? I'm wondering if node separation would allow for a "hybrid backend" usage, avoiding the need to implement the reward model env with the mcore backend?
terrykong commentedon Oct 29, 2025
This is a diff feature than what's tracked here since this is just to train a mcore RM, but yes, in theory it's possible, but you'd probably have to change https://github.com/NVIDIA-NeMo/RL/blob/main/examples/run_grpo_rm.py to generalize to N reward model environments.
For context, there's also another integration we're pretty excited about to incorporate a richer environment suite. I can't give ETA yet, but that's coming soon.
RayenTian commentedon Oct 30, 2025
The current implementation of the reward model environment supports configuring mcore as the training backend and dtensor as the reward model backend both on one node and multiple node. To achieve this setup, you only need to modify the policy backend parameter in the configuration file to specify your desired backend for each component.
odelalleau commentedon Oct 31, 2025
No concrete plans right now (to my knowledge)