[RFC] Profiling system in async mode

[tardis-key]

Feature request

a profiling system designed for asynchronous mode,this system needs to support the following key scenarios.

1. different async backends (vllm/sglang/fsdp/megatron).
2. AgentLoop (default)
3. engine worker

Motivation

The adoption of agentloop, full async, and similar paradigms has shifted Verl's workflow towards an asynchronous mode. And in #4106, rollout is changed to server mode by default. However, the current profiling system seems incompatible with asynchronous frameworks. Therefore, there is a need to redesign a profiling framework to support profiling data collection across different hardware architectures.

Profiling Capability for Rollout Independent Processes

• Vllm
• Sglang
• [perf] feat: verl profiler system support Agent Loop scenario and integrate torch.profiler #4320

Continuous Adaptation During Reconstruction

• agentLoop adaptation(legacy_worker)
• [perf] feat: verl profiler system support Agent Loop scenario and integrate torch.profiler #4320
• megatron worker adaptation(legacy_worker)
• discrete profiling with mindspeed doesn't work #4206
• [megatron] feat: support discrete profiling for mindspeed #4271
• support engine worker
• [worker] fix: Add profiler initialization for ActorRolloutRefWorker in engine_worker #4586

Configuration and Documentation Optimization

• modification of profiler configuration to reduce data volume
• [perf] fix: modify the NPU profiler default configuration #4475
• supplement the guidance for MindStudio Insight's JupyterLab launch method
• [doc] feat: add JupyterLab plugin instructions #4536
• fine-grained profiling tutorial for FSDP and Megatron
• [doc] feat: Add fine-grained profiling tutorial for FSDP and Megatron on Ascend #4610

[tardis-key]

After #4280, vLLM colocated training-inference rollout with process separation is enabled

• we should try to prepare necessary ENV for vllm profiling in code

Although vLLM 0.13 will be released soon and will support enabling profiling via API paras , we must ensure that the current version remains functional. Meanwhile, ENV enabling will continue to be compatible even after the upgrade to vLLM 0.13

[tardis-key]

In #5025, some issues that were not identified during the previous review process have been discovered and are documented as follows.

• step info is already defined in global_profiler, so step_start and step_end should not exist. And the logic related to auto_stop also requires further consideration of the issue of discontinuous collection steps, such as [1, 2, 5]. https://github.com/verl-project/verl/blob/main/docs/ascend_tutorial/ascend_profiling_zh.rst @bithighrr

  global_profiler:
     steps: [1, 2, 5]
     save_path: ./outputs/profile
  

• These parameters are most likely incompatible with the NPU version of SGLang. Test it ASAP

• NPU profiling CI is required [ci] feat: add profiling tests to vLLM ci #5215

• unify the parameter(api or ENV) preparation interface for httpServer.

@mengchengTang @wuxibin89

[bithighrr]

Global timesteps are the iterative steps for RL training, and rollout timesteps are the profiling steps of rollouts in the current global timestep. auto_stop only applies to the rollout profiling process, these functionalities should be decoupled.

For NPU-related issues, add CI test cases to validate regression tests.

[tardis-key]

Global timesteps are the iterative steps for RL training, and rollout timesteps are the profiling steps of rollouts in the current global timestep. auto_stop only applies to the rollout profiling process, these functionalities should be decoupled.

For NPU-related issues, add CI test cases to validate regression tests.

steps in global_profiler controls profiling, not RL training.

The rollout profiling steps can be realized by global_profiler and rollout.profiler, instead of adding extra parameters

[bithighrr]

The variable 'global_steps' would not limit the profiling number of forward steps (inference process) during rollout stage

[tardis-key]

#5121
Almost all issues have been fixed, except for CI.

[tardis-key]

#5215 ci check is merged!