In the MLPerf Inference v2.1 round, Qualcomm and their partners submitted a number of RetinaNet results, which the Review Committee eventually accepted to the Open division under the Preview category. Also, ResNet50 and BERT results from the same systems were accepted to the Closed division under the Available category.
Given the extenuating circumstances surrounding these RetinaNet submissions, I believe it would be reasonable NOT to require ALL of such submissions to be replayed in the v3.0 round under the Available category, for example, if a system is no longer available for experimentation.
In the MLPerf Inference v2.1 round, Qualcomm and their partners submitted a number of RetinaNet results, which the Review Committee eventually accepted to the Open division under the Preview category. Also, ResNet50 and BERT results from the same systems were accepted to the Closed division under the Available category.
Given the extenuating circumstances surrounding these RetinaNet submissions, I believe it would be reasonable NOT to require ALL of such submissions to be replayed in the v3.0 round under the Available category, for example, if a system is no longer available for experimentation.