Node-API Conformance Test Suite (`node-api-cts`)

[kraenhansen]

As Node-API is gaining popularity as a de-facto standard for ABI-stable access to JS engines and runtimes, I'd like to propose a project, implementing a pure ECMAScript Conformance Test Suite (CTS) for implementors of Node-API to exercise their implementations.

Why?

The Node.js project already maintains tests for Node-API in test/js-native-api and test/node-api. However, implementors of Node-API aren't necessarily implementing Node.js specific runtime APIs / built-ins modules (such as the process global, CommonJS require function or node:fs, node:path, etc.) and running these existing tests require polyfills for the Node.js runtime APIs or outright forking the tests - neither is ideal when when you want to downstream fixes and enhancements to these tests and the Node-API spec in general.

Goals

The Conformance Test Suite should:

1. Be maintained on the Node.js org (ex https://github.com/nodejs/node-api-cts), to affirm legitimacy and increase discoverability
2. Not use Node.js specific runtime APIs, as pure ECMAScript is needed for external implementors to use this effectively.
3. Target all of Node-API (both the engine and runtime specific functions).
4. Offer a way to gradually opt into / allow-list tests, to facilitate a gradual implementation for new implementors.
5. Drive the build of native addons while (optionally) delegating to the implementer to
   1. generate build system configuration files (such a CMakeList.txt for cmake-js)
   2. provide the command needed for building individual the Node-API addons
6. Implement each test as a separate file (like "Test262: ECMAScript Test Suite")
7. Rely on the implementor to provide a harness pre-loaded with helpers in globals, including:
   1. an assert implementation, provided to the harness by the CTS.
   2. a function to use to load an addon, since require and process.dlopen aren't documented as required (nor always possible to provide) for an implementor of Node-API.
8. Publish as a package on NPM with prebuilt addons, to make it trivial for implementors to run these tests as part of their own CI infrastructure.
9. Be have tests implemented
   1. in JavaScript and without use of a module system and instead rely on globals being provided by the harness
   2. using eslint to ensure limited globals are accessed and only known language features are used.
10. Use TypeScript to implement the any build scripts / CLI.

Non-goals

We don't want to:

1. Sync the contents of test/js-native-api in nodejs/node into the CTS tests, since these would require a transformations of Node.js runtime APIs and we expect this to be cumbersome / error-prone with little practical value (as Node-API isn't changed much and most changes to the tests will be capturing regressions or when adding new functions to Node-API).
2. Have the CTS export functions to inject TDD functions (like describe and it), to enable use of test tooling / reporting already known / maintained by the implementor nor do we want to export an object with "suites and tests" that a implementor-specific test harness can import and call as they see fit. Both would entail relying on a module system.

Open questions

1. Should we use TypeScript for the tests as well? This was a part of my initial proposal and I got some push-back on this and suggested using eslint instead, since it's simpler and doesn't force a "build step".
2. Would it be interesting to build out a public dashboard to track the conformance of implementors? Like https://wpt.fyi/ (maintained by The web-platform-tests Project).

Phases

1. Harness and a single test file

I want to propose starting with a harness and a single test file to show off the approach before we get too deep into the implementation of tests.

Before progressing to the next phase, we want to ensure that it harness will be runnable by most (if not all) known implementors.

2. Upstream into Node.js main repository (optional side quest)

A stretch goal of this project would be to upstream the Node-API CTS into the main Node.js repository, by gradually introducing changes to the existing test/js-native-api and test/node-api test directories moving them closer to the ideal outlined in the phase 1.

3. Start implementing the tests

If phase 2 is aborted, we'll need to implement the tests in the CTS from scratch. I imagine many of these files could actually be written by an LLM with the prior art as context.

Prior art:

• nodejs/js-native-api-test repo (see goals in nodejs/js-native-api-test/issues/7).
  Focussed only on the engine-specific part of Node-API and the repo is now archived.
• test/js-native-api in nodejs/node
• test/node-api in nodejs/node
• tests/napi in denoland/deno
• test/napi in oven-sh/bun
• unittests/NodeApi and specifically unittests/NodeApi/node_lite.cpp implemented by @vmoroz for the Hermes process, providing the minimal Node.js runtime polyfill needed for the existing Node.js tests against the Hermes engine.
• packages/node-tests in callstackincubator/react-native-node-api
• packages/test in toyobayashi/emnapi
• Test262: ECMAScript Test Suite could serve as inspiration.
• The WebGPU CTS could serve as inspiration: Tests and harness is written in TypeScript and it expects implementors to be able to interpret TS (using ts-node internally)
• The web-platform-tests Project could serve as inspiration.

[mcollina]

cc @nodejs/tsc

[targos]

@nodejs/node-api @nodejs/node-api-implementer

[NickNaso]

Hi everybody,
it's very interesting to have a unique test suite for Node-API (N-API) in the past the @nodejs/node-api team opened a repo https://github.com/nodejs/js-native-api-test (#7 ) with the purpose to write test for suite for JavaScript Engine specific portion of Node-API.
I think that now the times are more mature to create this kind of test suite.
What's unclear to me is how to manage native code within the test suite. Who should be responsible for building the native code?
Not all runtimes that support add-ons written with Node-API allow you to build them.
Is this something you want to delegate to the implementer, or should it be handled by the test suite?

[toyobayashi]

Wow nice to hear this project. BTW emnapi has almost forked all of the node-api test code (packages/test in toyobayashi/emnapi, even includes node-addon-api), and compiles for the following targets:

• wasm32-emscripten
• wasm64-emscripten
• wasm32-unknown-unknown
• wasm32-wasip1
• wasm32-wasip1-threads

And to some extent, the native code needs to be fine-tuned for each specific wasm target to make it work properly. Therefore, maybe I prefer to let the implementer build the native code.

[kraenhansen]

@NickNaso

in the past the @nodejs/node-api team opened a repo https://github.com/nodejs/js-native-api-test

I meant to reference this in the "prior art" section 🙈 I've now rectified that 👍

One of their original goals was to "sync the contents of nodejs/node's test/js-native-api regularly to the tests directory in" their new repository. I generally liked that idea initially when working on adding tests to the React Native implementation, but I have since stepped back from approach.

The API is stable and it would therefore mainly be new regression tests and tests for new functionality that would realistically benefit from a "sync" approach.
I'm currently in the belief that we could get a better result if we use an LLM to generate new pure-ECMAScript tests from the existing tests as context.

Not all runtimes that support add-ons written with Node-API allow you to build them.
Is this something you want to delegate to the implementer, or should it be handled by the test suite?

My immediate thought would be for the CTS package to drive the native builds, by delegating to the implementer.

The CTS could provide hooks for the implementer to generating build system configuration files (such CMakeList.txt for cmake-js) and it might be good to have some default implementations which could be shared across implementors, which support loading binaries built for Node.js.

I've added a goal for the CTS to support this. Let me know what you think 👍

[NickNaso]

@kraenhansen
I like the idea to give the implementer all the necessary things to build the addon. In this way a implementer can choose the build tool he prefer.
The initial approach proposed with synchronization with the test/js-native-api folder dates back to 2021; today, perhaps it would be better to write tests based on those already present. These tests could then be used in Node.js as in any other implementer.

[legendecas]

Thanks for the initiative, and I like the idea on a conformance test suite that helps the node-api implementers to unify the behavior.

To kick off the step 1, I opened nodejs/admin#1013.