Skip to content
This repository was archived by the owner on Oct 22, 2024. It is now read-only.

Commit eab2df5

Browse files
huningxinhuningxin
authored
Merge pull request #29 from BruceDai/converted_nnapi-cts
Add compatibility tests generated from Android Neural Networks API (NNAPI) Compatibility Test Suite.
2 parents b1d9b0e + be83e61 commit eab2df5

File tree

141 files changed

+13365
-16
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

141 files changed

+13365
-16
lines changed

‎README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,20 @@ A JavaScript implementation of the [Web Neural Network API](https://webmachinele
4444

4545
Open the web browser and navigate to http://localhost:8080/test
4646

47+
#### Run only CTS tests in node.js.
48+
49+
```sh
50+
> npm run cts
51+
```
52+
53+
#### Run only CTS tests in web browser.
54+
55+
```sh
56+
> npm start
57+
```
58+
59+
Open the web browser and navigate to http://localhost:8080/test/cts.html
60+
4761
## Other scripts
4862
### Build docs
4963

‎node_setup.js

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
global.navigator = {};
2+
require('./dist/webnn-polyfill.js');
3+
global.chai = require('chai');

‎package.json

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
"lintjs": "eslint . --no-eslintrc --config .eslintrcforjs.js --ext .js",
1919
"lint": "npm run lintts && npm run lintjs",
2020
"format": "clang-format --glob=\"{{src,test}/**/*.ts,{src,test}/**/*.js}\" --style=file -i",
21-
"test": "cross-env NODE_ENV=test mocha --require ./test/node_setup.js --exit --recursive"
21+
"test": "cross-env NODE_ENV=test mocha --require ./node_setup.js --exit test/*/*.js test/cts/from_nnapi/tests/cts.js",
22+
"cts": "cross-env NODE_ENV=test mocha --require ./node_setup.js --exit test/cts/from_nnapi/tests/cts.js"
2223
},
2324
"repository": {
2425
"type": "git",
@@ -45,9 +46,9 @@
4546
"@babel/polyfill": "^7.11.5",
4647
"@babel/preset-env": "^7.11.0",
4748
"@babel/register": "^7.11.5",
48-
"@tensorflow/tfjs-backend-cpu": "^2.7.0",
49-
"@tensorflow/tfjs-backend-webgl": "^2.7.0",
50-
"@tensorflow/tfjs-core": "^2.7.0",
49+
"@tensorflow/tfjs-backend-cpu": "^2.8.3",
50+
"@tensorflow/tfjs-backend-webgl": "^2.8.3",
51+
"@tensorflow/tfjs-core": "^2.8.3",
5152
"@typescript-eslint/eslint-plugin": "^4.0.1",
5253
"@typescript-eslint/parser": "^4.0.1",
5354
"babel-loader": "^8.1.0",

‎test/cts.html

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
<html>
2+
<head>
3+
<meta charset="utf-8">
4+
<title>WebNN API Polyfill | Mocha Tests</title>
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
6+
<link rel="stylesheet" href="https://unpkg.com/mocha/mocha.css" />
7+
</head>
8+
<body>
9+
<div id="mocha"></div>
10+
<script src="https://unpkg.com/chai/chai.js"></script>
11+
<script src="https://unpkg.com/mocha/mocha.js"></script>
12+
<script src="../dist/webnn-polyfill.js"></script>
13+
<script type="module" src="./utils.js"></script>
14+
<script class="mocha-init">
15+
mocha.setup('bdd');
16+
mocha.checkLeaks();
17+
</script>
18+
<script type="module" src="./cts/from_nnapi/tests/cts.js"></script>
19+
<script class="mocha-exec" type="module">
20+
mocha.run();
21+
</script>
22+
</body>
23+
</html>

‎test/cts/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Compatibility Test Suite (CTS)
2+
3+
This [cts](./) folder is for holding converted tools and tests from existed native tests of Machine Learning for WebNN API.
4+
5+
[./from_nnapi](./from_nnapi): holds converted tests from [Android Neural Networks API (NNAPI) CTS](https://android.googlesource.com/platform/frameworks/ml/+/refs/tags/android-cts-10.0_r5/nn/runtime/test/specs/).

‎test/cts/from_nnapi/COPYING

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
These two Python scripts cts_generator.py and test_generator.py in
2+
test/generator/src folder come from those scripts in nn/tools/test_generator of
3+
https://android.googlesource.com/platform/frameworks/ml/
4+
(tags: android-cts-10.0_r5) with some modification below:
5+
6+
1. Modify cts_generator.py to generate CTS for WebNN API.
7+
2. Modify test_generator.py to generate CTS for WebNN API.
8+
9+
Copyright 2017 The Android Open Source Project
10+
Licensed under the Apache License, Version 2.0 (the "License");
11+
you may not use this file except in compliance with the License.
12+
You may obtain a copy of the License at
13+
http://www.apache.org/licenses/LICENSE-2.0
14+
Unless required by applicable law or agreed to in writing, software
15+
distributed under the License is distributed on an "AS IS" BASIS,
16+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
See the License for the specific language governing permissions and
18+
limitations under the License.

‎test/cts/from_nnapi/README.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Converted CTS from NNAPI CTS
2+
3+
## Test Generator Tool
4+
This [test_generator](./test_generator) tool is for converting existed native
5+
[NNAPI CTS](https://android.googlesource.com/platform/frameworks/ml/+/refs/tags/android-cts-10.0_r5/nn/runtime/test/specs/) of V1_0, V1_1 and V1_2 versions to these tests for WebNN API.
6+
7+
### NNAPI Operations being map to WebNN API Operations Tables
8+
* Part I
9+
10+
The `test_generator` tool has supported to convert those tests of following
11+
NNAPI operations to the tests for such WebNN API operations of first wave.
12+
13+
| NNAPI | WebNN API (first wave ops) |
14+
|:------------------------------|:------------------------------|
15+
| RELU1 | clamp |
16+
| RELU6 | clamp |
17+
| CONCATENATION | concat |
18+
| ADD | add [+ relu/clamp] |
19+
| SUB | sub [+ relu/clamp] |
20+
| MUL | mul [+ relu/clamp] |
21+
| DIV | div [+ relu/clamp] |
22+
| MAXIMUM | max |
23+
| MINIMUM | min |
24+
| EXP | exp |
25+
| LOGISTIC | sigmoid |
26+
| SQRT | sqrt |
27+
| TANH | tanh |
28+
| FULLY_CONNECTED | matmul [+ add+ relu/clamp] |
29+
| AVERAGE_POOL_2D | averagePool2d [+ relu/clamp] |
30+
| MAX_POOL_2D | maxPool2d [+ relu/clamp] |
31+
| CONV_2D | conv2d [+ add + relu/clamp] |
32+
| DEPTHWISE_CONV_2D | conv2d [+ add + relu/clamp] |
33+
| RELU | relu |
34+
| RESHAPE | reshape |
35+
| SLICE | slice |
36+
| SOFTMAX | softmax |
37+
| SPLIT | split |
38+
| SQUEEZE | squeeze |
39+
| TRANSPOSE | transpose |
40+
41+
* Part II
42+
43+
And there're these following NNAPI operations which could be map to others
44+
WebNN API operations of next waves.
45+
46+
| NNAPI | WebNN API (next waves ops) |
47+
|:------------------------------|:------------------------------|
48+
| ABS | abs |
49+
| FLOOR | floor |
50+
| LOG | log |
51+
| NEG | neg |
52+
| SIN | sin |
53+
| L2_POOL_2D | l2Pool2d [+ relu/clamp] |
54+
55+
* Note:
56+
57+
1. Current WebNN Polyfill API supports Float32 and Int32 two types, so these
58+
NNAPI CTS using Float32 and Int32 types were able to be converted, while those
59+
NNAPI CTS using Uint8 and Int8 types would be converted until WebNN Polyfill API
60+
supports Uint8 and Int8 types.
61+
62+
2. Native NNAPI supports Float16, while there's lack of Float16 in JavaScript
63+
environments, so such NNAPI CTS with Float16 wouldn't be convertted.
64+
65+
### Usage
66+
* Prerequisites
67+
* Python3
68+
* Numpy
69+
* Download and unzip NNAPI CTS Specs tarball files locally by below commands
70+
71+
```shell
72+
cd test_generator
73+
./ready_nnapi_cts_specs.sh
74+
```
75+
* Generate Tests
76+
77+
```shell
78+
npm start
79+
```
80+
81+
Generated tests would be in following three folders
82+
[./tests/V1_0](./test/V1_0)
83+
[./tests/V1_1](./test/V1_1)
84+
[./tests/V1_2](./test/V1_2)
85+
and these tests could also be in all-in-one
86+
[./tests/cts.js](./tests/cts.js) file.
87+
88+
89+
### Accuracy for Generated Tests
90+
The converted tests follow these [reference accuracy](https://android.googlesource.com/platform/frameworks/ml/+/refs/tags/android-cts-10.0_r5/nn/runtime/test/TestGenerated.cpp#117):
91+
```cpp
92+
float fpAtol = 1e-5f;
93+
float fpRtol = 5.0f * 1.1920928955078125e-7f;
94+
```
95+
And for relaxed tests
96+
```cpp
97+
// If in relaxed mode, set the absolute tolerance to be 5ULP of FP16.
98+
fpAtol = 5.0f * 0.0009765625f;
99+
// Set the relative tolerance to be 5ULP of the corresponding FP precision.
100+
fpRtol = 5.0f * 0.0009765625f;
101+
```

‎test/cts/from_nnapi/test_generator/package.json

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"name": "test_generator",
3+
"version": "0.0.1",
4+
"description": "Convert native NNAPI CTS tests to WebNN API tests",
5+
"license": "Apache 2.0",
6+
"main": "./src/main.py",
7+
"directories": {
8+
"src": "src files"
9+
},
10+
"scripts": {
11+
"start": "python3 ./src/main.py ./specs -t ../tests -c ../tests/cts.js"
12+
},
13+
"dependencies": {},
14+
"devDependencies": {},
15+
"authors": [
16+
"Bruce Dai <[email protected]>"
17+
]
18+
}

‎test/cts/from_nnapi/test_generator/ready_nnapi_cts_specs.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/sh
2+
TAG=android-cts-10.0_r5
3+
NNAPI_VERSION="
4+
V1_0
5+
V1_1
6+
V1_2
7+
"
8+
9+
# download and unzip Specs tarball files
10+
for version in ${NNAPI_VERSION}
11+
do
12+
wget https://android.googlesource.com/platform/frameworks/ml/+archive/refs/tags/${TAG}/nn/runtime/test/specs/${version}.tar.gz
13+
mkdir -p specs/${version}
14+
tar -xvzf ${version}.tar.gz -C specs/${version}
15+
done

0 commit comments

Comments
 (0)