Applicability to non-browser JS environments

[dontcallmedom]

Both @jasonmayes in his Opportunities and challenges talk and @cynthia in his TAG feedback presentation mention the need to pay attention to the applicability of our browser-targeted work to non-browser JS environments, in particular node.js.

Can anyone (@anssiko ? @huningxin ?) comment on whether any work in that direction has been considered in the Web Machine Learning CG? both for WebNN and Model Loader API?

Complementary questions:

• are all JS-ML frameworks targeting both browsers and node.js - the presentations on tensorflow.js and ONNX.js made it clear it was the case for these two, but not sure how paddle.js, magenta.js, ml5.js
• are there node.js-specific ML frameworks?
• are there well-known obstacles in making the browser-targeted work for ML applicable to node.js?

[jasonmayes]

Adding @pyu10055 @dsmilkov @nsthorat @annxingyuan @tafsiri @lina128 for any thoughts related to this topic too (TF.js team) and for visibility.

[bhack]

Sorry for cross-posting but you could be interested also in this thread iree-org/iree#2863

[anssiko]

Can anyone (@anssiko ? @huningxin ?) comment on whether any work in that direction has been considered in the Web Machine Learning CG? both for WebNN and Model Loader API?

The initial focus has been on the browser as an implementation target, but there is nothing that would prevent non-browser JS environments to implement these APIs. Node.js or similar would need to host the API elsewhere than on the browser-specific Navigator object, of course. There may be other subtleties too.

We have discussed this topic in the group and will try to keep browser-specific abstractions out from the spec as much as we can as to not discourage non-browser implementations.

Related, the community has started work on webnn-polyfill. Ping @huningxin to comment on feasibility of running this polyfill in a non-WebGL environments like Node.js using the CPU backend.

[phoddie]

Please forgive me if this is out of scope, but the topic of "non-browser JS environments" is an area of focus for me.

The reference to Node.js above suggests that this discussion may be about whether to support JavaScript environments outside the browser which are still part of the Web Platform. I chair Ecma TC53, ECMAScript Modules for Embedded Systems, which is defining a JavaScript runtime (environment) that is separate from the Web Platform. The work focuses on microcontrollers with considerably fewer resources than the devices which typically host Web Platform runtimes. Still, the manufacturers of microcontrollers are increasingly integrating ML hardware. The ecosystem would benefit from standard APIs.

Whether it is possible for the same APIs that are optimal for the Web Platform to also be applied, in some form, to this more resource constrained environment is an interesting question with far reaching implications. If it is out of scope for W3C, I understand. If exploring that is of interest, that would be welcome.

[anssiko]

@phoddie, what is being worked on in TC53 would be definitely of interest to the workshop participants. The goal of the workshop is to explore what the overall ecosystem is doing in this space, not limited to W3C efforts. For example, we have a number of ML frameworks sharing perspectives. There are some examples of Web APIs that have their roots in the Node-land, for example https://github.com/w3c/sensors/ started as a contribution from Johnny-Five by @rwaldron. Similarly, some web primitives have made their way to the other direction.

We'd be very interested in hearing more about TC53 plans and the question you laid out above would be a great discussion point.

[huningxin]

Related, the community has started work on webnn-polyfill. Ping @huningxin to comment on feasibility of running this polyfill in a non-WebGL environments like Node.js using the CPU backend.

As the current implementation of webnn-polyfill is based on tf.js, it should run in non-WebGL environment by tf.js CPU backend and WASM backend. @pyu10055 for tf.js clarification.

[WenheLI]

I am currently working on pipcook, a nodejs(tfjs) based front-end ML/DL framework.

This might be interesting to talk about.

[anssiko]

The Extending W3C ML Work to Embedded Systems talk by @phoddie shares the following learning from prior work on JS & embedded systems:

It is better to have a dedicated embedded API and a dedicated web API that share concepts and operations to the extent practical.

This suggests closer coordination with TC53 and possible other efforts from JS (and Wasm) on embedded systems space to ensure proposals and designs are reviewed also outside W3C community and browser vendors.

[phoddie]

I'd like to add one additional note here.

The talk given by Chai Chaoweeraprasit (@wchao1115) of Microsoft on DirectML describes an approach that works well when the goal is to provide high performance access to a diverse set of hardware.

Rather than providing a single API that then needs to be implemented for vastly different hardware, focus on providing efficient access to the hardware resources. Once that is in place, higher level libraries can be developed -- sometimes even in script -- that compose those low level APIs in ways that are useful and, to varying degrees, portable for applications.

That approach is consistent with the Extensible Web Manifesto that helps guide the work of Ecma TC53:

• Free up the standards process (especially in the short-term) to focus on features with security or performance concerns, and features that can only be added at the platform level, such as new hardware.
• Allow web developers and browser-initiated libraries to take the lead in costly explorations.