Support for mutliturn online RL training

[UbeCc]

Currently, verl only support single-turn rl training. As agents turning is becoming urgent, will verl support multiturn rl in the next few days?
Maybe I can help. Thanks!

@PeterSH6 @zhaochenyang20

[zhaochenyang20]

Sure. Welcome to join. And, proposal should be professional, look at how I make proposal to SGLang:

Reaserch Project 开题

1. 问题是什么，如何清晰定义问题；
2. 这个问题的 scope 如何，有什么非常强的假设，这些假设合理么；
3. 谁在乎这个问题，不要说谷歌会在乎这个问题，要精确到谷歌的某个组甚至是某个人会在乎这个问题；
4. 现有的方案是什么，有什么不足；
5. 我们的方案可能是什么（开题的时候不一定能写完）；
6. 如何评估？达到了什么效果则说明我们的方法奏效了；
7. 这个过程有什么不确定性，风险如何；
8. 计划工作的 timeline 是如何；

Feature 开题

1. 目标框架现在的实现是什么样子的；有什么问题；
2. 修改方案是什么，预计会修改什么部分；
3. 将会达到的效果如何；
4. 有什么不确定性；
5. 计划工作的 timeline；

[PeterSH6]

@UbeCc Nice suggestion!
We can discuss the plan this week.
Could you connect with us through WeChat or Slack?

[zhaochenyang20]

@UbeCc @PeterSH6 I can connect you guys. Haoran is my senior. And, good night 😂

[UbeCc]

@UbeCc @PeterSH6 I can connect you guys. Haoran is my senior. And, good night 😂

Thanks Chenyang, enjoy your day!

[UbeCc]

@UbeCc Nice suggestion! We can discuss the plan this week. Could you connect with us through WeChat or Slack?

Yeah, let me send my WeChat id through email

[YSLIU627]

Great idea! I could also offer some help!

[AIBionics]

I'm also working on multiturn online RL training at the moment, and I'd be glad to assist if you need any help.
Maybe we can create a WeChat group and then add everyone to the group for discussion.

[UbeCc]

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

ubecwang@gmail.com

[YSLIU627]

Great! My wechat is liuzhihan0627 . See you then, Best, Zhihan

…

On Tue, Feb 25, 2025 at 6:45 PM Haoran Wang ***@***.***> wrote: Yeah could you plz send your WeChat id to me through email? Let me create a group and work together. ***@***.*** — Reply to this email directly, view it on GitHub <#385 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APK6TFYXJJBZODWCCJZJ3RL2RUFB7AVCNFSM6AAAAABX3D4UF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGYYTCOJZGM> . You are receiving this because you commented.Message ID: ***@***.***> [image: UbeCc]*UbeCc* left a comment (volcengine/verl#385) <#385 (comment)> Yeah could you plz send your WeChat id to me through email? Let me create a group and work together. ***@***.*** — Reply to this email directly, view it on GitHub <#385 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APK6TFYXJJBZODWCCJZJ3RL2RUFB7AVCNFSM6AAAAABX3D4UF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTGYYTCOJZGM> . You are receiving this because you commented.Message ID: ***@***.***>

[sbl1996]

Yeah could you plz send your WeChat id to me through email?

Let me create a group and work together.

ubecwang@gmail.com

I've already wrote a multi-turn implementation and it works, but the training is not stable. Could you add me to the wechat group? My wechat id is Zukala-Koth. Great thanks! @UbeCc

[zhaochenyang20]

@sbl1996 Sure. I will tell him tommorow.

[UbeCc]

Yeah could you plz send your WeChat id to me through email?
Let me create a group and work together.
ubecwang@gmail.com

I've already wrote a multi-turn implementation and it works, but the training is not stable. Could you add me to the wechat group? My wechat id is Zukala-Koth. Great thanks! @UbeCc

Done. Thank you!

[oliverz20]

Hi, I’m interested in multiturn RL as well. Could you please add me to the group? My WeChat ID is sfoliver. Thanks a lot! @UbeCc

[Tshiyao]

Hi @UbeCc , I'm also really into multiturn RL and would love to join the group! My WeChat ID is Liu_Qihuang . Looking forward to connecting and learning more. Thanks!