FP8 doesn't result in speedup over BF16

[nightlessbaron]

1. Verified Flash-RL is installed using the Tutorial.
2. Currently using Fix FlashRL compatibility with vllm 0.10.2 and torch 2.8 #36 for experiments.
3. Setup: 70B LLM, 32k sequence length, 512 training bsz, 32 mini bsz, 32 H200 nodes

Performance (in seconds) per step between FP8 (blue) and BF16 (yellow) doesn't vary a lot

The FP8 overhead is even more apparent in smaller models (8B) with 8k response length.

[nightlessbaron]

@LiyuanLucasLiu @yaof20 can you have a look at it?

[LiyuanLucasLiu]

thanks for reporting! can you share a bit more for debugging? particularly:

1. the reshard time as in timing_s/reshard;
2. the full training log;

[nightlessbaron]

Thanks @LiyuanLucasLiu , here are the details:

[Full FP8 Training Log]

I am not logging reshard time, however attaching a few relevant plots.

[LiyuanLucasLiu]

my suspension now is that the environment patch is not properly picked up (since the flashrl init and update logs are absent in your log). Can you try yaof20/verl#1 to see whether it helps? Or, can you try running through provided int8 example and check whether you can reproduce in your envs?

[nightlessbaron]

Ok, I believe I have imported flashrl, but I will re-run it and also test int8

[nightlessbaron]

@LiyuanLucasLiu adding yaof20/verl#1 doesn't help.

[nightlessbaron]

I also ended up downgrading to torch 2.7 and vllm 0.10.0 as these are the versions that FlashRL is tested for from your end but the FP8 still seems to be having some overhead over BF16 🤔

[LiyuanLucasLiu]

I see, in this case, another reason could be the rollout is not compute bound. Can you test the rollout setup with standalone vllm to see whether it can speed things up? Fp8 overhead is expected but usually offset by the rollout generation time

[nightlessbaron]

Thanks for the quick response, @LiyuanLucasLiu
So if I understand this correctly, if we are not in a compute bound regime, we might not see gains on Fp8 quantization over Bf16

[LiyuanLucasLiu]

yes. however, I think 70b should be compute bound. So maybe a simple way to test this is to directly run some controlled experiments to test the speed of fp8 70b vs bf16 70b in your setting, in order to debug the issue.

[nightlessbaron]

@LiyuanLucasLiu, looks like the culprit is indeed due to the overhead of conversion of quantized weights. As we can see, the generation speed is improving, but the overhead from weight updates is negating the speed up from using fp8.

Do you have any suggestions on how to reduce this overhead?