Add compatibility tests generated from Android Neural Networks API (NNAPI) Compatibility Test Suite.

[BruceDai]

This PR is on the purpose of adding compatibility tests for WebNN first wave ops by converting existed native Android Neural Networks API tests.

NNAPI Operations mapping to WebNN Operations (first wave ops)

The Android Neural Networks API (NNAPI) is an Android C API designed for running computationally intensive operations for machine learning on Android devices. NNAPI is designed to provide a base layer of functionality for higher-level machine learning frameworks.

Here's a table of NNAPI Operations mapping to WebNN Operations (first wave ops), please review details of converting NNAPI CTS on this link.

The NNAPI ADD/SUB/MUL/DIV/AVERAGE_POOL_2D/MAX_POOL_2D Operations fused activation computation, and CONV_2D/DEPTHWISE_CONV_2D/FULLY_CONNECTED Operations fused both add bias computation and activation computation.

NNAPI	WebNN API (first wave ops)	Converted Test Status (Pass/Total)
RELU1	clamp	4/4
RELU6	clamp	4/4
CONCATENATION	concat	6/6
ADD	add [+ relu/clamp]	2/2
SUB	sub [+ relu/clamp]	12/12
MUL	mul [+ relu/clamp]	4/4
DIV	div [+ relu/clamp]	4/4
MAXIMUM	max	6/6
MINIMUM	min	6/6
EXP	exp	2/2
LOGISTIC	sigmoid	4/4
SQRT	sqrt	2/2
TANH	tanh	2/2
FULLY_CONNECTED	matmul [+ add + relu/clamp]	15/15
AVERAGE_POOL_2D	averagePool2d [+ relu/clamp]	24/24
MAX_POOL_2D	maxPool2d [+ relu/clamp]	18/18
CONV_2D	conv2d [+ add + relu/clamp]	52/52
DEPTHWISE_CONV_2D	conv2d [+ add + relu/clamp]	24/24
RELU	relu	4/4
RESHAPE	reshape	4/4
SLICE	slice	17/17
SOFTMAX	[mul +] softmax	6/6
SPLIT	split	18/18
SQUEEZE	squeeze	4/4
TRANSPOSE	transpose	6/6

Generated WebNN Tests Status

Please have a quick glance at those converted test on this link. Current I leverage and upgrade NNAPI test_generator tools to generate total 250 (Pass: 250) compatibility tests of Int32 and Float32 types which are supported by WebNN Polyfill API, please the detail generated test cases on the above table.
Except WebNN API unsupported Operand types, there're following cases for some NNAPI CTS not being converted for WebNN API:

1. AVERAGE_POOL_2D/MAX_POOL_2D/CONV_2D/DEPTHWISE_CONV_2D tests using implicit padding, current this convert tool doesn't involve padding compute function for 'VALID', 'SAME' implicit padding, this would be an enhancement feature.
2. DEPTHWISE_CONV_2D tests with depth_multiplier not being equal to 1, that is, input_channels not being equal to output_channels
3. softmax Op of WebNN API requires 2D input, such SOFTMAX tests with rank of input is greater than 2, or SOFTMAX tests used 2D input with optional inupt2 axis being not equal to 1 or -1.

And for FULLY_CONNECTED tests, if rank of input is greater than 2, this convert tool would flatten input to a 2-D Tensor of generated tests for WebNN API.

Accuracy for Generated Tests

Current WebNN Test adopts this accuracy:

episilon = 1e-6, 
rtol = 5.0 * 1.1920928955078125e-7

While generated tests follows NNAPI accuracy:

  float fpAtol = 1e-5f;
  float fpRtol = 5.0f * 1.1920928955078125e-7f;

and for relaxed tests using this following accuracy

  // If in relaxed mode, set the absolute tolerance to be 5ULP of FP16.
  fpAtol = 5.0f * 0.0009765625f;
  // Set the relative tolerance to be 5ULP of the corresponding FP precision.
  fpRtol = 5.0f * 0.0009765625f;

@anssiko @huningxin Please take a look. What's your suggestion? Thanks.

[anssiko]

Thanks @BruceDai! This is quite a substantial PR so I'll defer to @huningxin for the detailed review.

Web spec usually reuse https://web-platform-tests.org/ infrastructure. Can these tests be repurposed to use testharness.js with a reasonable effort? It is not a hard requirement to have tests written using testharness.js, but it'd help sharing tests cross-browser and also many (all?) browsers automatically import and run w-p-t tests as part of their CI processes.

[huningxin]

@BruceDai , thanks for your great efforts!

This work would help Conformance testing of ML APIs for the We by reusing/porting Android NNAPI CTS.

@anssiko , I propose this as an agenda item for the next WebML CG call for wide feedbacks.

[huningxin]

I didn't find test cases for grouped_conv_2d?

[BruceDai]

Can these tests be repurposed to use testharness.js with a reasonable effort?

@anssiko Yes, these tests can use w-p-t testharness.js as we're discussing "Reuse mocha tests for web platform tests" #2 .

[anssiko]

@anssiko , I propose this as an agenda item for the next WebML CG call for wide feedbacks.

Good suggestion, added to WebML CG Teleconference – 10 December 2020 - 15:00-16:00 UTC+0.

[huningxin]

These 12 pending tests helped catch the pool issue tensorflow/tfjs#4271 for tensorflow, now the patch tensorflow/tfjs#4282 for this issue landed, these 12 tests will add to test with next release @tensorlfow/tfjs-core.

@BruceDai , tf.js released 2.8.3, could you please verify whether it fixes the issue?

[BruceDai]

@huningxin Thanks for your suggestion. Now I have upgrade tf.js to latest 2.8.3, the 12 skipped pool tests now all pass.

[huningxin]

Now I have upgrade tf.js to latest 2.8.3

Thanks @BruceDai , since you upgraded the tf.js version, could you please confirm there are no regressions for other tests?

[BruceDai]

Thanks @huningxin
Yes, previous I upgraded tf.js as 2.8.2, I caught a tf.js regression issue of split onn tf.js 2.8.2, and I submitted/landed my patch ( tensorflow/tfjs#4478).
With upgraded tf.js 2.8.3, all tests including 175 existed tests plus 250 tests of this pr passed without any regression tested in node.js and in Chrome browser.

[huningxin]

As there were no objects in WebML CG 7 Jan call, I am going to merge this PR. Thanks for your great contributions, @BruceDai !