[QST] Inference total time fluctuates too much

[ShaobinChen-AH]

What is your question?

****I ran command
python3 inference/inference_gr_ranking_async.py --gin_config_file ./inference/configs/kuairand_1k_inference_ranking.gin --checkpoint_dir /workspace/data/iter500 --mode simulate --max_bs 8

and then I found the Total time it outputs was nearly 996 s. Then, I ran this command again without any modification and that this time the Total time it outputs was nearly 363 s. why? why does the result fluctuate so much. I can not understand this**

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.

[geoffreyQiu]

Hi,
by running inference/inference_gr_ranking_async.py, are you testing the kvcache branch from here?
If so, there are some errors in onload/offload synchronization, and we just fixed them last week. You may update and try again. Our tests on A100/L40 give quite similar results from several runs. Let me know if you still have variant results and what configs / changes you have.
Also, this branch is under development and we'll formally merge it in 26'Jan as shown in the latest roadmap

[shijieliu]

this should be resolved and close this issue now. If it still exists, pls reopen this issue