Will MLPerf intend to include the performance evaluation of LLMs

[tohneecao]

GPT3 and so on

[psyhtest]

Yes, this is planned for MLPerf Inference v3.1 in early August (and MLPerf Training v3.0 in late May).

[tohneecao]

Glad to hear this，may i ask which LLM and which Accelerator exactly.

[psyhtest]

It's (a clone of) GPT3 (175B parameters). It is up to MLCommons members whether to optimise it for their accelerator products.

[psyhtest]

Would you be interested to see results from NVIDIA? Qualcomm? Anyone else?

[tohneecao]

I prefer NVIDIA, are there any results can show？

[psyhtest]

All official v3.1 results will be announced in late August - early September.