Skip to content

Commit 5db1668

Browse files
committed
fix directory structure info
1 parent 6046356 commit 5db1668

1 file changed

Lines changed: 48 additions & 29 deletions

File tree

docs/submission/submission-cli.md

Lines changed: 48 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -9,35 +9,54 @@ Please refer to the [installation page](site:inference/install/) to install MLCF
99

1010
=== "Custom automation based MLPerf results"
1111
If you have not followed the `mlcr` commands under the individual model pages in the [benchmarks](../index.md) directory, please make sure that the result directory is structured in the following way. You can see the real examples for the expected folder structure [here](https://github.com/mlcommons/inference/tree/submission-generation-examples).
12-
```
13-
└── System description ID(SUT Name)
14-
├── system_meta.json
15-
└── Benchmark
16-
└── Scenario
17-
├── Performance
18-
| └── run_1 run for all scenarios
19-
| ├── mlperf_log_summary.txt
20-
| └── mlperf_log_detail.txt
21-
├── Accuracy
22-
| ├── mlperf_log_summary.txt
23-
| ├── mlperf_log_detail.txt
24-
| ├── mlperf_log_accuracy.json
25-
| └── accuracy.txt
26-
|── Compliance_Test_ID
27-
| ├── Performance
28-
| | └── run_x/#1 run for all scenarios
29-
| | ├── mlperf_log_summary.txt
30-
| | └── mlperf_log_detail.txt
31-
| ├── Accuracy # for TEST01 only
32-
| | ├── baseline_accuracy.txt (if test fails in deterministic mode)
33-
| | ├── compliance_accuracy.txt (if test fails in deterministic mode)
34-
| | ├── mlperf_log_accuracy.json
35-
| | └── accuracy.txt
36-
| ├── verify_performance.txt
37-
| └── verify_accuracy.txt # for TEST01 only
38-
|── user.conf
39-
└── measurements.json
40-
```
12+
13+
The submission generator supports two kinds of results, shown in the tabs below. **LoadGen based results** are produced by the reference/optimized implementations and contain the usual `mlperf_log_*` files. **Inference endpoint based results** are produced by the LLM/API endpoint harness; a scenario is treated as an endpoint run when a `config.yaml` (or `config.yml`) is present at the scenario root, in which case a single JSON summary replaces the LoadGen logs and there is no `run_1` subfolder. The mode folder names (`performance`, `accuracy`) must be lowercase.
14+
15+
=== "LoadGen based results"
16+
```
17+
└── System description ID(SUT Name)
18+
├── system_meta.json
19+
└── Benchmark
20+
└── Scenario
21+
├── performance
22+
| └── run_1 run for all scenarios
23+
| ├── mlperf_log_summary.txt
24+
| └── mlperf_log_detail.txt
25+
├── accuracy
26+
| ├── mlperf_log_summary.txt
27+
| ├── mlperf_log_detail.txt
28+
| ├── mlperf_log_accuracy.json
29+
| └── accuracy.txt
30+
|── Compliance_Test_ID
31+
| ├── performance
32+
| | └── run_x/#1 run for all scenarios
33+
| | ├── mlperf_log_summary.txt
34+
| | └── mlperf_log_detail.txt
35+
| ├── accuracy # for TEST01 only
36+
| | ├── baseline_accuracy.txt (if test fails in deterministic mode)
37+
| | ├── compliance_accuracy.txt (if test fails in deterministic mode)
38+
| | ├── mlperf_log_accuracy.json
39+
| | └── accuracy.txt
40+
| ├── verify_performance.txt
41+
| └── verify_accuracy.txt # for TEST01 only
42+
|── user.conf
43+
└── measurements.json
44+
```
45+
46+
=== "Inference endpoint based results"
47+
```
48+
└── System description ID(SUT Name)
49+
├── system_meta.json
50+
└── Benchmark
51+
└── Scenario
52+
├── config.yaml # endpoint run marker (config.yml also accepted); copied into the submission
53+
├── measurements.json
54+
├── performance
55+
| └── result_summary.json
56+
└── accuracy
57+
└── accuracy_results.json
58+
```
59+
Note: endpoint runs do not require a `user.conf` and do not use the `run_1` subfolder or any `mlperf_log_*` files.
4160

4261
<details>
4362
<summary>Click here if you are submitting in open division</summary>

0 commit comments

Comments
 (0)