You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/submission/submission-cli.md
+48-29Lines changed: 48 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,35 +9,54 @@ Please refer to the [installation page](site:inference/install/) to install MLCF
9
9
10
10
=== "Custom automation based MLPerf results"
11
11
If you have not followed the `mlcr` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way. You can see the real examples for the expected folder structure [here](https://github.com/mlcommons/inference/tree/submission-generation-examples).
12
-
```
13
-
└── System description ID(SUT Name)
14
-
├── system_meta.json
15
-
└── Benchmark
16
-
└── Scenario
17
-
├── Performance
18
-
| └── run_1 run for all scenarios
19
-
| ├── mlperf_log_summary.txt
20
-
| └── mlperf_log_detail.txt
21
-
├── Accuracy
22
-
| ├── mlperf_log_summary.txt
23
-
| ├── mlperf_log_detail.txt
24
-
| ├── mlperf_log_accuracy.json
25
-
| └── accuracy.txt
26
-
|── Compliance_Test_ID
27
-
| ├── Performance
28
-
| | └── run_x/#1 run for all scenarios
29
-
| | ├── mlperf_log_summary.txt
30
-
| | └── mlperf_log_detail.txt
31
-
| ├── Accuracy # for TEST01 only
32
-
| | ├── baseline_accuracy.txt (if test fails in deterministic mode)
33
-
| | ├── compliance_accuracy.txt (if test fails in deterministic mode)
34
-
| | ├── mlperf_log_accuracy.json
35
-
| | └── accuracy.txt
36
-
| ├── verify_performance.txt
37
-
| └── verify_accuracy.txt # for TEST01 only
38
-
|── user.conf
39
-
└── measurements.json
40
-
```
12
+
13
+
The submission generator supports two kinds of results, shown in the tabs below. **LoadGen based results** are produced by the reference/optimized implementations and contain the usual `mlperf_log_*` files. **Inference endpoint based results** are produced by the LLM/API endpoint harness; a scenario is treated as an endpoint run when a `config.yaml` (or `config.yml`) is present at the scenario root, in which case a single JSON summary replaces the LoadGen logs and there is no `run_1` subfolder. The mode folder names (`performance`, `accuracy`) must be lowercase.
14
+
15
+
=== "LoadGen based results"
16
+
```
17
+
└── System description ID(SUT Name)
18
+
├── system_meta.json
19
+
└── Benchmark
20
+
└── Scenario
21
+
├── performance
22
+
| └── run_1 run for all scenarios
23
+
| ├── mlperf_log_summary.txt
24
+
| └── mlperf_log_detail.txt
25
+
├── accuracy
26
+
| ├── mlperf_log_summary.txt
27
+
| ├── mlperf_log_detail.txt
28
+
| ├── mlperf_log_accuracy.json
29
+
| └── accuracy.txt
30
+
|── Compliance_Test_ID
31
+
| ├── performance
32
+
| | └── run_x/#1 run for all scenarios
33
+
| | ├── mlperf_log_summary.txt
34
+
| | └── mlperf_log_detail.txt
35
+
| ├── accuracy # for TEST01 only
36
+
| | ├── baseline_accuracy.txt (if test fails in deterministic mode)
37
+
| | ├── compliance_accuracy.txt (if test fails in deterministic mode)
38
+
| | ├── mlperf_log_accuracy.json
39
+
| | └── accuracy.txt
40
+
| ├── verify_performance.txt
41
+
| └── verify_accuracy.txt # for TEST01 only
42
+
|── user.conf
43
+
└── measurements.json
44
+
```
45
+
46
+
=== "Inference endpoint based results"
47
+
```
48
+
└── System description ID(SUT Name)
49
+
├── system_meta.json
50
+
└── Benchmark
51
+
└── Scenario
52
+
├── config.yaml # endpoint run marker (config.yml also accepted); copied into the submission
53
+
├── measurements.json
54
+
├── performance
55
+
| └── result_summary.json
56
+
└── accuracy
57
+
└── accuracy_results.json
58
+
```
59
+
Note: endpoint runs do not require a `user.conf` and do not use the `run_1` subfolder or any `mlperf_log_*` files.
41
60
42
61
<details>
43
62
<summary>Click here if you are submitting in open division</summary>
0 commit comments