Building an extensible web platform for ML, one abstraction at a time

[anssiko]

Building ML experiences using the web platform-provided built-in low-level APIs and capabilities (WebGL, WebGPU, Wasm, WebNN) require domain-specific knowledge. JS libraries and frameworks have emerged to fill in the gap providing more approachable programming interfaces that not only increase developer productivity but also unlock ML in new areas such as education, learning, and accessibility.

TensorFlow.js and ML5.js represent two such higher-level abstractions. ML5.js provides the highest-level abstraction built around popular models for image, sound and text. ML5.js sits atop TensorFlow.js that in turn abstracts out low-level web-platform APIs to provide constructs ML practitioners are familiar with.

TensorFlow.js was discussed in the talk by @jasonmayes and talk by @annxingyuan. ML5.js was introduced in the talk by @yining1023.

Web platform has been incubating a few APIs that surface high-level abstractions, namely Accelerated Shape Detection in Images and Web Speech API (discussed by @kdavis-mozilla in his talk). These APIs are agnostic with respect to low-level implementation details and have been stuck in the incubation phase in part due to lack of interoperability and privacy issues.

Questions:

• Are we in agreement that advancing with standardization of low-level capabilities e.g. WebNN API is the pragmatic first step? Should we in addition consider standardizing higher-level abstractions in parallel e.g. Model Loader API, possibly others inspired by learnings from JS libraries and frameworks?
• What are the implications to user's privacy of focusing on the low-level capabilities first? See also the Extensible Web Manifesto.

[wchao1115]

While much of the discussion has been on high-level abstraction in the framework, the hardware ecosystems have also become an active area of ML development in the past many years. Evidently much of this development have been concentrated at the OS layer with more hardware vendors surfacing their underlying architectures and performance novelties for ML through the operating system APIs. Having a web standard facilitating this exposure in a browser and operating system-agnostic way will be a key foundation for ML on the web for years to come. I believe this is the right time for the web community to become a stakeholder in this development and establish this standard with WebNN.

A good news is that most applications of ML today are not tied to high-level format. The first-order problems seem to be model deployment, adaptability, reliability and performance. The web will organically sort out what formats to last based on practicality and usefulness of experience.